API tools faq
paste
Guest User

Untitled

a guest
May 31st, 2015
269
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 88.36 KB | None | 0 0
raw download clone embed print report
  1. 2015-04-22T13:22:06.092256+00:00 app[web.1]: task_id=1051 fn=all_jobs_finished? finished=true
  2. 2015-04-22T13:22:13.066113+00:00 app[web.2]: request_id=caea9620-5ac2-4404-8e6d-8f6dfbe96b31 ns=api at=start
  3. 2015-04-22T13:22:13.086345+00:00 app[web.2]: request_id=caea9620-5ac2-4404-8e6d-8f6dfbe96b31 app_id=app-one api=/push task_id=1052 fn=fx_target selector='{"ptype"=>{:$in=>[:apns]}, "tags"=>{:$in=>["fake_5kk"]}}' indicator=1
  4. 2015-04-22T13:22:13.070723+00:00 app[web.2]: request_id=caea9620-5ac2-4404-8e6d-8f6dfbe96b31 app_id=app-one api=/push fn=request path=/push body='{"target":{"platform":"ios","tags":["fake_5kk"]}, "payload":{"alert":"load test"}}'
  5. 2015-04-22T13:22:13.088983+00:00 app[web.2]: request_id=caea9620-5ac2-4404-8e6d-8f6dfbe96b31 app_id=app-one api=/push task_id=1052 fn=response status=201 length=18 body=["{\"task_id\":\"1052\"}"]
  6. 2015-04-22T13:22:13.089053+00:00 app[web.2]: request_id=caea9620-5ac2-4404-8e6d-8f6dfbe96b31 app_id=app-one api=/push task_id=1052 ns=api at=finish elapsed=23.026
  7. 2015-04-22T13:22:13.089354+00:00 app[web.2]: [3:INFO] 2015-04-22 13:22:13 :: Status: 201, Content-Length: 18, Response Time: 25.22ms
  8. 2015-04-22T13:22:13.095896+00:00 heroku[router]: at=info method=POST path="/push" host=mocapi-load.herokuapp.com request_id=caea9620-5ac2-4404-8e6d-8f6dfbe96b31 fwd="178.62.172.184" dyno=web.2 connect=1ms service=28ms status=201 bytes=148
  9. 2015-04-22T13:37:45.186484+00:00 app[web.2]: fn=submit_job job_id=1052:apns:0 app_id=app-one devices=1000
  10. 2015-04-22T13:37:45.282859+00:00 app[web.2]: fn=submit_job job_id=1052:apns:2 app_id=app-one devices=1000
  11. 2015-04-22T13:37:45.376513+00:00 app[web.2]: fn=submit_job job_id=1052:apns:4 app_id=app-one devices=1000
  12. 2015-04-22T13:37:45.423999+00:00 app[web.2]: fn=submit_job job_id=1052:apns:6 app_id=app-one devices=1000
  13. 2015-04-22T13:37:45.211415+00:00 app[web.2]: fn=submit_job job_id=1052:apns:1 app_id=app-one devices=1000
  14. 2015-04-22T13:37:45.353729+00:00 app[web.2]: fn=submit_job job_id=1052:apns:3 app_id=app-one devices=1000
  15. 2015-04-22T13:37:45.399992+00:00 app[web.2]: fn=submit_job job_id=1052:apns:5 app_id=app-one devices=1000
  16. 2015-04-22T13:37:45.468087+00:00 app[web.2]: fn=submit_job job_id=1052:apns:7 app_id=app-one devices=1000
  17. 2015-04-22T13:37:45.514694+00:00 app[web.2]: fn=submit_job job_id=1052:apns:9 app_id=app-one devices=1000
  18. 2015-04-22T13:37:45.490447+00:00 app[web.2]: fn=submit_job job_id=1052:apns:8 app_id=app-one devices=1000
  19. 2015-04-22T13:37:46.157101+00:00 app[web.2]: fn=submit_job job_id=1052:apns:10 app_id=app-one devices=1000
  20. 2015-04-22T13:37:46.246479+00:00 app[web.2]: fn=submit_job job_id=1052:apns:12 app_id=app-one devices=1000
  21. 2015-04-22T13:37:46.287502+00:00 app[web.2]: fn=submit_job job_id=1052:apns:14 app_id=app-one devices=1000
  22. 2015-04-22T13:37:46.331255+00:00 app[web.2]: fn=submit_job job_id=1052:apns:16 app_id=app-one devices=1000
  23. 2015-04-22T13:37:46.366862+00:00 app[web.2]: fn=submit_job job_id=1052:apns:18 app_id=app-one devices=1000
  24. 2015-04-22T13:37:46.188560+00:00 app[web.2]: fn=submit_job job_id=1052:apns:11 app_id=app-one devices=1000
  25. 2015-04-22T13:37:46.268473+00:00 app[web.2]: fn=submit_job job_id=1052:apns:13 app_id=app-one devices=1000
  26. 2015-04-22T13:37:46.308885+00:00 app[web.2]: fn=submit_job job_id=1052:apns:15 app_id=app-one devices=1000
  27. 2015-04-22T13:37:46.348970+00:00 app[web.2]: fn=submit_job job_id=1052:apns:17 app_id=app-one devices=1000
  28. 2015-04-22T13:37:46.385347+00:00 app[web.2]: fn=submit_job job_id=1052:apns:19 app_id=app-one devices=1000
  29. 2015-04-22T13:37:49.876710+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=d9689228-5efc-4b8f-aa60-97efc0350af3 fwd="54.159.16.8" dyno=web.2 connect=2ms service=7ms status=200 bytes=143
  30. 2015-04-22T13:37:49.868489+00:00 app[web.2]: request_id=d9689228-5efc-4b8f-aa60-97efc0350af3 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  31. 2015-04-22T13:37:49.884007+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=10
  32. 2015-04-22T13:37:49.866840+00:00 app[web.2]: request_id=d9689228-5efc-4b8f-aa60-97efc0350af3 ns=api at=start
  33. 2015-04-22T13:37:49.869313+00:00 app[web.2]: request_id=d9689228-5efc-4b8f-aa60-97efc0350af3 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  34. 2015-04-22T13:37:49.869396+00:00 app[web.2]: request_id=d9689228-5efc-4b8f-aa60-97efc0350af3 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.559
  35. 2015-04-22T13:37:49.869713+00:00 app[web.2]: [3:INFO] 2015-04-22 13:37:49 :: Status: 200, Content-Length: 18, Response Time: 3.89ms
  36. 2015-04-22T13:37:50.096231+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  37. 2015-04-22T13:37:54.889084+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=c0e2e64b-7422-4d1d-bebb-e9aaf6ea59c0 fwd="54.159.16.8" dyno=web.2 connect=1ms service=6ms status=200 bytes=143
  38. 2015-04-22T13:37:54.880376+00:00 app[web.2]: request_id=c0e2e64b-7422-4d1d-bebb-e9aaf6ea59c0 ns=api at=start
  39. 2015-04-22T13:37:54.882808+00:00 app[web.2]: request_id=c0e2e64b-7422-4d1d-bebb-e9aaf6ea59c0 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  40. 2015-04-22T13:37:54.882896+00:00 app[web.2]: request_id=c0e2e64b-7422-4d1d-bebb-e9aaf6ea59c0 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.562
  41. 2015-04-22T13:37:54.883501+00:00 app[web.2]: [3:INFO] 2015-04-22 13:37:54 :: Status: 200, Content-Length: 18, Response Time: 3.64ms
  42. 2015-04-22T13:37:55.105017+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  43. 2015-04-22T13:37:54.882000+00:00 app[web.2]: request_id=c0e2e64b-7422-4d1d-bebb-e9aaf6ea59c0 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  44. 2015-04-22T13:37:54.896612+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=10
  45. 2015-04-22T13:53:17.746116+00:00 app[web.2]: try=1 device_count=20000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154569942340
  46. 2015-04-22T13:53:17.746175+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  47. 2015-04-22T13:53:17.746252+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  48. 2015-04-22T13:53:17.746339+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  49. 2015-04-22T13:53:17.746414+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  50. 2015-04-22T13:53:17.746556+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  51. 2015-04-22T13:53:17.746725+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  52. 2015-04-22T13:53:17.746798+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  53. 2015-04-22T13:53:17.746874+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  54. 2015-04-22T13:53:17.746952+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  55. 2015-04-22T13:53:17.747022+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  56. 2015-04-22T13:53:17.747137+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  57. 2015-04-22T13:53:17.747220+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  58. 2015-04-22T13:53:17.747284+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  59. 2015-04-22T13:53:17.747356+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  60. 2015-04-22T13:53:17.747432+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  61. 2015-04-22T13:53:17.747537+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  62. 2015-04-22T13:53:17.747612+00:00 app[web.2]: try=1 device_count=20000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569942340 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  63. 2015-04-22T13:53:18.538443+00:00 app[web.2]: fn=submit_job job_id=1052:apns:20 app_id=app-one devices=1000
  64. 2015-04-22T13:53:18.614205+00:00 app[web.2]: fn=submit_job job_id=1052:apns:22 app_id=app-one devices=1000
  65. 2015-04-22T13:53:18.690155+00:00 app[web.2]: fn=submit_job job_id=1052:apns:24 app_id=app-one devices=1000
  66. 2015-04-22T13:53:18.581088+00:00 app[web.2]: fn=submit_job job_id=1052:apns:21 app_id=app-one devices=1000
  67. 2015-04-22T13:53:18.657804+00:00 app[web.2]: fn=submit_job job_id=1052:apns:23 app_id=app-one devices=1000
  68. 2015-04-22T13:53:18.762545+00:00 app[web.2]: fn=submit_job job_id=1052:apns:25 app_id=app-one devices=1000
  69. 2015-04-22T13:53:18.805952+00:00 app[web.2]: fn=submit_job job_id=1052:apns:26 app_id=app-one devices=1000
  70. 2015-04-22T13:53:18.826070+00:00 app[web.2]: fn=submit_job job_id=1052:apns:27 app_id=app-one devices=1000
  71. 2015-04-22T13:53:18.874268+00:00 app[web.2]: fn=submit_job job_id=1052:apns:28 app_id=app-one devices=1000
  72. 2015-04-22T13:53:18.919249+00:00 app[web.2]: fn=submit_job job_id=1052:apns:29 app_id=app-one devices=1000
  73. 2015-04-22T13:53:23.050535+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=e24c5aae-4c7c-40bf-b6f6-99e392a1f765 fwd="54.159.16.8" dyno=web.2 connect=1ms service=8ms status=200 bytes=143
  74. 2015-04-22T13:53:23.041776+00:00 app[web.2]: request_id=e24c5aae-4c7c-40bf-b6f6-99e392a1f765 ns=api at=start
  75. 2015-04-22T13:53:23.044751+00:00 app[web.2]: request_id=e24c5aae-4c7c-40bf-b6f6-99e392a1f765 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  76. 2015-04-22T13:53:23.044920+00:00 app[web.2]: request_id=e24c5aae-4c7c-40bf-b6f6-99e392a1f765 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=3.287
  77. 2015-04-22T13:53:23.045338+00:00 app[web.2]: [3:INFO] 2015-04-22 13:53:23 :: Status: 200, Content-Length: 18, Response Time: 4.63ms
  78. 2015-04-22T13:53:23.043724+00:00 app[web.2]: request_id=e24c5aae-4c7c-40bf-b6f6-99e392a1f765 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  79. 2015-04-22T13:53:23.069430+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=10
  80. 2015-04-22T13:53:25.023814+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  81. 2015-04-22T14:08:50.098194+00:00 app[web.2]: try=1 device_count=30000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154569702200
  82. 2015-04-22T14:08:50.098249+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  83. 2015-04-22T14:08:50.098329+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  84. 2015-04-22T14:08:50.098421+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  85. 2015-04-22T14:08:50.098539+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  86. 2015-04-22T14:08:50.098608+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  87. 2015-04-22T14:08:50.098687+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  88. 2015-04-22T14:08:50.098769+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  89. 2015-04-22T14:08:50.098840+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  90. 2015-04-22T14:08:50.098960+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  91. 2015-04-22T14:08:50.099035+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  92. 2015-04-22T14:08:50.099119+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  93. 2015-04-22T14:08:50.099211+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  94. 2015-04-22T14:08:50.099295+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  95. 2015-04-22T14:08:50.099366+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  96. 2015-04-22T14:08:50.099503+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  97. 2015-04-22T14:08:50.113792+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  98. 2015-04-22T14:08:50.113867+00:00 app[web.2]: try=1 device_count=30000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569702200 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  99. 2015-04-22T14:08:51.830072+00:00 app[web.2]: fn=submit_job job_id=1052:apns:30 app_id=app-one devices=1000
  100. 2015-04-22T14:08:51.878233+00:00 app[web.2]: fn=submit_job job_id=1052:apns:31 app_id=app-one devices=1000
  101. 2015-04-22T14:08:52.106056+00:00 app[web.2]: fn=submit_job job_id=1052:apns:32 app_id=app-one devices=1000
  102. 2015-04-22T14:08:52.146156+00:00 app[web.2]: fn=submit_job job_id=1052:apns:33 app_id=app-one devices=1000
  103. 2015-04-22T14:08:52.208384+00:00 app[web.2]: fn=submit_job job_id=1052:apns:35 app_id=app-one devices=1000
  104. 2015-04-22T14:08:52.256925+00:00 app[web.2]: fn=submit_job job_id=1052:apns:37 app_id=app-one devices=1000
  105. 2015-04-22T14:08:52.346041+00:00 app[web.2]: fn=submit_job job_id=1052:apns:39 app_id=app-one devices=1000
  106. 2015-04-22T14:08:52.172081+00:00 app[web.2]: fn=submit_job job_id=1052:apns:34 app_id=app-one devices=1000
  107. 2015-04-22T14:08:52.236218+00:00 app[web.2]: fn=submit_job job_id=1052:apns:36 app_id=app-one devices=1000
  108. 2015-04-22T14:08:52.305368+00:00 app[web.2]: fn=submit_job job_id=1052:apns:38 app_id=app-one devices=1000
  109. 2015-04-22T14:08:57.172274+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=d379894e-b876-48ec-abb9-4eaef4ea4226 fwd="54.159.16.8" dyno=web.2 connect=1ms service=8ms status=200 bytes=143
  110. 2015-04-22T14:08:57.168597+00:00 app[web.2]: request_id=d379894e-b876-48ec-abb9-4eaef4ea4226 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  111. 2015-04-22T14:08:57.186203+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=10
  112. 2015-04-22T14:08:57.166971+00:00 app[web.2]: request_id=d379894e-b876-48ec-abb9-4eaef4ea4226 ns=api at=start
  113. 2015-04-22T14:08:57.169605+00:00 app[web.2]: request_id=d379894e-b876-48ec-abb9-4eaef4ea4226 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  114. 2015-04-22T14:08:57.169776+00:00 app[web.2]: request_id=d379894e-b876-48ec-abb9-4eaef4ea4226 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.772
  115. 2015-04-22T14:08:57.170174+00:00 app[web.2]: [3:INFO] 2015-04-22 14:08:57 :: Status: 200, Content-Length: 18, Response Time: 4.06ms
  116. 2015-04-22T14:08:57.363204+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  117. 2015-04-22T14:24:23.474066+00:00 app[web.2]: try=1 device_count=40000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154569636580
  118. 2015-04-22T14:24:23.474120+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  119. 2015-04-22T14:24:23.474260+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  120. 2015-04-22T14:24:23.474337+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  121. 2015-04-22T14:24:23.474417+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  122. 2015-04-22T14:24:23.474546+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  123. 2015-04-22T14:24:23.474640+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  124. 2015-04-22T14:24:23.474723+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  125. 2015-04-22T14:24:23.474797+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  126. 2015-04-22T14:24:23.474881+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  127. 2015-04-22T14:24:23.474978+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  128. 2015-04-22T14:24:23.475049+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  129. 2015-04-22T14:24:23.475129+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  130. 2015-04-22T14:24:23.475291+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  131. 2015-04-22T14:24:23.475361+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  132. 2015-04-22T14:24:23.475509+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  133. 2015-04-22T14:24:23.475591+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  134. 2015-04-22T14:24:23.475664+00:00 app[web.2]: try=1 device_count=40000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569636580 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  135. 2015-04-22T14:24:24.282147+00:00 app[web.2]: fn=submit_job job_id=1052:apns:40 app_id=app-one devices=1000
  136. 2015-04-22T14:24:24.322569+00:00 app[web.2]: fn=submit_job job_id=1052:apns:42 app_id=app-one devices=1000
  137. 2015-04-22T14:24:24.360123+00:00 app[web.2]: fn=submit_job job_id=1052:apns:44 app_id=app-one devices=1000
  138. 2015-04-22T14:24:24.397322+00:00 app[web.2]: fn=submit_job job_id=1052:apns:46 app_id=app-one devices=1000
  139. 2015-04-22T14:24:24.442712+00:00 app[web.2]: fn=submit_job job_id=1052:apns:48 app_id=app-one devices=1000
  140. 2015-04-22T14:24:24.303258+00:00 app[web.2]: fn=submit_job job_id=1052:apns:41 app_id=app-one devices=1000
  141. 2015-04-22T14:24:24.341510+00:00 app[web.2]: fn=submit_job job_id=1052:apns:43 app_id=app-one devices=1000
  142. 2015-04-22T14:24:24.379476+00:00 app[web.2]: fn=submit_job job_id=1052:apns:45 app_id=app-one devices=1000
  143. 2015-04-22T14:24:24.416924+00:00 app[web.2]: fn=submit_job job_id=1052:apns:47 app_id=app-one devices=1000
  144. 2015-04-22T14:24:24.460792+00:00 app[web.2]: fn=submit_job job_id=1052:apns:49 app_id=app-one devices=1000
  145. 2015-04-22T14:24:26.937430+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=228d58ac-9374-4f3b-8803-2f227ebe217e fwd="54.159.16.8" dyno=web.2 connect=18ms service=18ms status=200 bytes=143
  146. 2015-04-22T14:24:26.922841+00:00 app[web.2]: request_id=228d58ac-9374-4f3b-8803-2f227ebe217e app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  147. 2015-04-22T14:24:26.931707+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=5
  148. 2015-04-22T14:24:26.921272+00:00 app[web.2]: request_id=228d58ac-9374-4f3b-8803-2f227ebe217e ns=api at=start
  149. 2015-04-22T14:24:26.923643+00:00 app[web.2]: request_id=228d58ac-9374-4f3b-8803-2f227ebe217e app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  150. 2015-04-22T14:24:26.923729+00:00 app[web.2]: request_id=228d58ac-9374-4f3b-8803-2f227ebe217e app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.475
  151. 2015-04-22T14:24:26.923977+00:00 app[web.2]: [3:INFO] 2015-04-22 14:24:26 :: Status: 200, Content-Length: 18, Response Time: 3.54ms
  152. 2015-04-22T14:24:26.992539+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  153. 2015-04-22T14:24:31.905695+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=61463f5d-fba4-44d5-ba8d-fe9f8879c896 fwd="54.159.16.8" dyno=web.2 connect=1ms service=6ms status=200 bytes=143
  154. 2015-04-22T14:24:31.894246+00:00 app[web.2]: request_id=61463f5d-fba4-44d5-ba8d-fe9f8879c896 ns=api at=start
  155. 2015-04-22T14:24:31.896553+00:00 app[web.2]: request_id=61463f5d-fba4-44d5-ba8d-fe9f8879c896 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  156. 2015-04-22T14:24:31.896898+00:00 app[web.2]: [3:INFO] 2015-04-22 14:24:31 :: Status: 200, Content-Length: 18, Response Time: 3.53ms
  157. 2015-04-22T14:24:31.995946+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  158. 2015-04-22T14:24:31.895764+00:00 app[web.2]: request_id=61463f5d-fba4-44d5-ba8d-fe9f8879c896 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  159. 2015-04-22T14:24:31.896668+00:00 app[web.2]: request_id=61463f5d-fba4-44d5-ba8d-fe9f8879c896 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.411
  160. 2015-04-22T14:24:31.903946+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=5
  161. 2015-04-22T14:39:55.826024+00:00 app[web.2]: try=1 device_count=50000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154571598160
  162. 2015-04-22T14:39:55.845197+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  163. 2015-04-22T14:39:55.845279+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  164. 2015-04-22T14:39:55.845365+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  165. 2015-04-22T14:39:55.845463+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  166. 2015-04-22T14:39:55.845553+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  167. 2015-04-22T14:39:55.845676+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  168. 2015-04-22T14:39:55.845765+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  169. 2015-04-22T14:39:55.846226+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  170. 2015-04-22T14:39:55.837079+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  171. 2015-04-22T14:39:55.837149+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  172. 2015-04-22T14:39:55.837243+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  173. 2015-04-22T14:39:55.837332+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  174. 2015-04-22T14:39:55.845053+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  175. 2015-04-22T14:39:55.846401+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  176. 2015-04-22T14:39:55.846542+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  177. 2015-04-22T14:39:55.846656+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  178. 2015-04-22T14:39:55.846817+00:00 app[web.2]: try=1 device_count=50000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571598160 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  179. 2015-04-22T14:39:57.225716+00:00 app[web.2]: fn=submit_job job_id=1052:apns:50 app_id=app-one devices=1000
  180. 2015-04-22T14:39:57.303327+00:00 app[web.2]: fn=submit_job job_id=1052:apns:52 app_id=app-one devices=1000
  181. 2015-04-22T14:39:57.348721+00:00 app[web.2]: fn=submit_job job_id=1052:apns:54 app_id=app-one devices=1000
  182. 2015-04-22T14:39:57.387785+00:00 app[web.2]: fn=submit_job job_id=1052:apns:56 app_id=app-one devices=1000
  183. 2015-04-22T14:39:57.424974+00:00 app[web.2]: fn=submit_job job_id=1052:apns:58 app_id=app-one devices=1000
  184. 2015-04-22T14:39:57.283348+00:00 app[web.2]: fn=submit_job job_id=1052:apns:51 app_id=app-one devices=1000
  185. 2015-04-22T14:39:57.328784+00:00 app[web.2]: fn=submit_job job_id=1052:apns:53 app_id=app-one devices=1000
  186. 2015-04-22T14:39:57.369429+00:00 app[web.2]: fn=submit_job job_id=1052:apns:55 app_id=app-one devices=1000
  187. 2015-04-22T14:39:57.404888+00:00 app[web.2]: fn=submit_job job_id=1052:apns:57 app_id=app-one devices=1000
  188. 2015-04-22T14:39:57.448366+00:00 app[web.2]: fn=submit_job job_id=1052:apns:59 app_id=app-one devices=1000
  189. 2015-04-22T14:40:01.045672+00:00 app[web.1]: request_id=ce5b754c-2470-4b74-8f80-1d0ac635fc95 ns=api at=start
  190. 2015-04-22T14:40:01.052051+00:00 app[web.1]: request_id=ce5b754c-2470-4b74-8f80-1d0ac635fc95 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  191. 2015-04-22T14:40:01.052261+00:00 app[web.1]: request_id=ce5b754c-2470-4b74-8f80-1d0ac635fc95 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=18.591
  192. 2015-04-22T14:40:01.052672+00:00 app[web.1]: [3:INFO] 2015-04-22 14:40:01 :: Status: 200, Content-Length: 18, Response Time: 44.61ms
  193. 2015-04-22T14:40:01.051052+00:00 app[web.1]: request_id=ce5b754c-2470-4b74-8f80-1d0ac635fc95 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  194. 2015-04-22T14:40:01.095250+00:00 app[web.1]: task_id=1052 fn=finished_jobs jobs=10
  195. 2015-04-22T14:40:01.060027+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=ce5b754c-2470-4b74-8f80-1d0ac635fc95 fwd="54.159.16.8" dyno=web.1 connect=3ms service=48ms status=200 bytes=143
  196. 2015-04-22T14:40:01.522589+00:00 app[web.1]: task_id=1052 fn=all_jobs_finished? finished=false
  197. 2015-04-22T14:55:28.700406+00:00 app[web.2]: try=1 device_count=60000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154839116160
  198. 2015-04-22T14:55:28.700523+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  199. 2015-04-22T14:55:28.700682+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  200. 2015-04-22T14:55:28.700756+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  201. 2015-04-22T14:55:28.700848+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  202. 2015-04-22T14:55:28.700942+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  203. 2015-04-22T14:55:28.701039+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  204. 2015-04-22T14:55:28.701138+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  205. 2015-04-22T14:55:28.709775+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  206. 2015-04-22T14:55:28.709854+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  207. 2015-04-22T14:55:28.709952+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  208. 2015-04-22T14:55:28.710041+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  209. 2015-04-22T14:55:28.710139+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  210. 2015-04-22T14:55:28.710224+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  211. 2015-04-22T14:55:28.710321+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  212. 2015-04-22T14:55:28.710409+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  213. 2015-04-22T14:55:28.710497+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  214. 2015-04-22T14:55:28.710589+00:00 app[web.2]: try=1 device_count=60000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839116160 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  215. 2015-04-22T14:55:30.992956+00:00 app[web.2]: fn=submit_job job_id=1052:apns:60 app_id=app-one devices=1000
  216. 2015-04-22T14:55:31.036913+00:00 app[web.2]: fn=submit_job job_id=1052:apns:62 app_id=app-one devices=1000
  217. 2015-04-22T14:55:31.129459+00:00 app[web.2]: fn=submit_job job_id=1052:apns:64 app_id=app-one devices=1000
  218. 2015-04-22T14:55:31.169323+00:00 app[web.2]: fn=submit_job job_id=1052:apns:66 app_id=app-one devices=1000
  219. 2015-04-22T14:55:31.217978+00:00 app[web.2]: fn=submit_job job_id=1052:apns:68 app_id=app-one devices=1000
  220. 2015-04-22T14:55:31.014071+00:00 app[web.2]: fn=submit_job job_id=1052:apns:61 app_id=app-one devices=1000
  221. 2015-04-22T14:55:31.104509+00:00 app[web.2]: fn=submit_job job_id=1052:apns:63 app_id=app-one devices=1000
  222. 2015-04-22T14:55:31.149438+00:00 app[web.2]: fn=submit_job job_id=1052:apns:65 app_id=app-one devices=1000
  223. 2015-04-22T14:55:31.192647+00:00 app[web.2]: fn=submit_job job_id=1052:apns:67 app_id=app-one devices=1000
  224. 2015-04-22T14:55:31.264205+00:00 app[web.2]: fn=submit_job job_id=1052:apns:69 app_id=app-one devices=1000
  225. 2015-04-22T14:55:34.812826+00:00 app[web.2]: request_id=444011ce-149b-4837-a5ff-ebd7f2f5d97a app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  226. 2015-04-22T14:55:34.827992+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=10
  227. 2015-04-22T14:55:34.810613+00:00 app[web.2]: request_id=444011ce-149b-4837-a5ff-ebd7f2f5d97a ns=api at=start
  228. 2015-04-22T14:55:34.813693+00:00 app[web.2]: request_id=444011ce-149b-4837-a5ff-ebd7f2f5d97a app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  229. 2015-04-22T14:55:34.813825+00:00 app[web.2]: request_id=444011ce-149b-4837-a5ff-ebd7f2f5d97a app_id=api api=/control task_id=1052 ns=api at=finish elapsed=3.234
  230. 2015-04-22T14:55:34.814083+00:00 app[web.2]: [3:INFO] 2015-04-22 14:55:34 :: Status: 200, Content-Length: 18, Response Time: 4.36ms
  231. 2015-04-22T14:55:34.955550+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  232. 2015-04-22T14:55:34.815261+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=444011ce-149b-4837-a5ff-ebd7f2f5d97a fwd="54.159.16.8" dyno=web.2 connect=0ms service=6ms status=200 bytes=143
  233. 2015-04-22T15:11:06.682346+00:00 app[web.2]: try=1 device_count=70000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154837777640
  234. 2015-04-22T15:11:06.682428+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  235. 2015-04-22T15:11:06.682522+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  236. 2015-04-22T15:11:06.682620+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  237. 2015-04-22T15:11:06.682709+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  238. 2015-04-22T15:11:06.682818+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  239. 2015-04-22T15:11:06.687828+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  240. 2015-04-22T15:11:06.687911+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  241. 2015-04-22T15:11:06.688008+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  242. 2015-04-22T15:11:06.688098+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  243. 2015-04-22T15:11:06.693047+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  244. 2015-04-22T15:11:06.693128+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  245. 2015-04-22T15:11:06.693527+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  246. 2015-04-22T15:11:06.693611+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  247. 2015-04-22T15:11:06.693705+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  248. 2015-04-22T15:11:06.693791+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  249. 2015-04-22T15:11:06.694147+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  250. 2015-04-22T15:11:06.694228+00:00 app[web.2]: try=1 device_count=70000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154837777640 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  251. 2015-04-22T15:11:07.706058+00:00 app[web.2]: fn=submit_job job_id=1052:apns:70 app_id=app-one devices=1000
  252. 2015-04-22T15:11:07.782998+00:00 app[web.2]: fn=submit_job job_id=1052:apns:72 app_id=app-one devices=1000
  253. 2015-04-22T15:11:07.825430+00:00 app[web.2]: fn=submit_job job_id=1052:apns:74 app_id=app-one devices=1000
  254. 2015-04-22T15:11:07.862594+00:00 app[web.2]: fn=submit_job job_id=1052:apns:76 app_id=app-one devices=1000
  255. 2015-04-22T15:11:07.908932+00:00 app[web.2]: fn=submit_job job_id=1052:apns:78 app_id=app-one devices=1000
  256. 2015-04-22T15:11:07.729772+00:00 app[web.2]: fn=submit_job job_id=1052:apns:71 app_id=app-one devices=1000
  257. 2015-04-22T15:11:07.807214+00:00 app[web.2]: fn=submit_job job_id=1052:apns:73 app_id=app-one devices=1000
  258. 2015-04-22T15:11:07.844347+00:00 app[web.2]: fn=submit_job job_id=1052:apns:75 app_id=app-one devices=1000
  259. 2015-04-22T15:11:07.882674+00:00 app[web.2]: fn=submit_job job_id=1052:apns:77 app_id=app-one devices=1000
  260. 2015-04-22T15:11:07.928271+00:00 app[web.2]: fn=submit_job job_id=1052:apns:79 app_id=app-one devices=1000
  261. 2015-04-22T15:11:14.128204+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=f63a8a4d-9f3f-4dd2-a190-e204832ca7eb fwd="54.159.16.8" dyno=web.1 connect=16ms service=10ms status=200 bytes=143
  262. 2015-04-22T15:11:14.123373+00:00 app[web.1]: request_id=f63a8a4d-9f3f-4dd2-a190-e204832ca7eb app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  263. 2015-04-22T15:11:14.141341+00:00 app[web.1]: task_id=1052 fn=finished_jobs jobs=10
  264. 2015-04-22T15:11:14.121707+00:00 app[web.1]: request_id=f63a8a4d-9f3f-4dd2-a190-e204832ca7eb ns=api at=start
  265. 2015-04-22T15:11:14.124167+00:00 app[web.1]: request_id=f63a8a4d-9f3f-4dd2-a190-e204832ca7eb app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  266. 2015-04-22T15:11:14.124251+00:00 app[web.1]: request_id=f63a8a4d-9f3f-4dd2-a190-e204832ca7eb app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.568
  267. 2015-04-22T15:11:14.124646+00:00 app[web.1]: [3:INFO] 2015-04-22 15:11:14 :: Status: 200, Content-Length: 18, Response Time: 4.67ms
  268. 2015-04-22T15:11:14.616640+00:00 app[web.1]: task_id=1052 fn=all_jobs_finished? finished=false
  269. 2015-04-22T15:16:34.499970+00:00 heroku[web.1]: State changed from up to down
  270. 2015-04-22T15:16:34.503272+00:00 heroku[web.1]: State changed from down to starting
  271. 2015-04-22T15:16:38.323865+00:00 heroku[web.1]: Starting process with command `bundle exec ruby pserver.rb -sv -e prod -p 44223`
  272. 2015-04-22T15:16:42.126938+00:00 app[web.1]: [3:INFO] 2015-04-22 15:16:42 :: Starting server on 0.0.0.0:44223 in production mode. Watch out for stones.
  273. 2015-04-22T15:16:42.514549+00:00 heroku[web.1]: State changed from starting to up
  274. 2015-04-22T15:26:43.122132+00:00 app[web.2]: try=1 device_count=80000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154571486160
  275. 2015-04-22T15:26:43.122211+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  276. 2015-04-22T15:26:43.133164+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  277. 2015-04-22T15:26:43.133276+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  278. 2015-04-22T15:26:43.133366+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  279. 2015-04-22T15:26:43.133800+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  280. 2015-04-22T15:26:43.133884+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  281. 2015-04-22T15:26:43.133972+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  282. 2015-04-22T15:26:43.134071+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  283. 2015-04-22T15:26:43.134157+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  284. 2015-04-22T15:26:43.134254+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  285. 2015-04-22T15:26:43.134342+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  286. 2015-04-22T15:26:43.134443+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  287. 2015-04-22T15:26:43.134531+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  288. 2015-04-22T15:26:43.134649+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  289. 2015-04-22T15:26:43.135064+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  290. 2015-04-22T15:26:43.135156+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  291. 2015-04-22T15:26:43.135258+00:00 app[web.2]: try=1 device_count=80000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154571486160 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  292. 2015-04-22T15:26:45.228222+00:00 app[web.2]: fn=submit_job job_id=1052:apns:81 app_id=app-one devices=1000
  293. 2015-04-22T15:26:45.282953+00:00 app[web.2]: fn=submit_job job_id=1052:apns:83 app_id=app-one devices=1000
  294. 2015-04-22T15:26:45.326027+00:00 app[web.2]: fn=submit_job job_id=1052:apns:85 app_id=app-one devices=1000
  295. 2015-04-22T15:26:45.362714+00:00 app[web.2]: fn=submit_job job_id=1052:apns:87 app_id=app-one devices=1000
  296. 2015-04-22T15:26:45.401765+00:00 app[web.2]: fn=submit_job job_id=1052:apns:89 app_id=app-one devices=1000
  297. 2015-04-22T15:26:45.202739+00:00 app[web.2]: fn=submit_job job_id=1052:apns:80 app_id=app-one devices=1000
  298. 2015-04-22T15:26:45.260615+00:00 app[web.2]: fn=submit_job job_id=1052:apns:82 app_id=app-one devices=1000
  299. 2015-04-22T15:26:45.304909+00:00 app[web.2]: fn=submit_job job_id=1052:apns:84 app_id=app-one devices=1000
  300. 2015-04-22T15:26:45.345396+00:00 app[web.2]: fn=submit_job job_id=1052:apns:86 app_id=app-one devices=1000
  301. 2015-04-22T15:26:45.383078+00:00 app[web.2]: fn=submit_job job_id=1052:apns:88 app_id=app-one devices=1000
  302. 2015-04-22T15:26:47.288819+00:00 app[web.2]: fn=submit_job job_id=1052:apns:90 app_id=app-one devices=1000
  303. 2015-04-22T15:26:47.331735+00:00 app[web.2]: fn=submit_job job_id=1052:apns:92 app_id=app-one devices=1000
  304. 2015-04-22T15:26:47.371515+00:00 app[web.2]: fn=submit_job job_id=1052:apns:94 app_id=app-one devices=1000
  305. 2015-04-22T15:26:47.446075+00:00 app[web.2]: fn=submit_job job_id=1052:apns:96 app_id=app-one devices=1000
  306. 2015-04-22T15:26:47.487029+00:00 app[web.2]: fn=submit_job job_id=1052:apns:98 app_id=app-one devices=1000
  307. 2015-04-22T15:26:47.310917+00:00 app[web.2]: fn=submit_job job_id=1052:apns:91 app_id=app-one devices=1000
  308. 2015-04-22T15:26:47.350949+00:00 app[web.2]: fn=submit_job job_id=1052:apns:93 app_id=app-one devices=1000
  309. 2015-04-22T15:26:47.394114+00:00 app[web.2]: fn=submit_job job_id=1052:apns:95 app_id=app-one devices=1000
  310. 2015-04-22T15:26:47.465100+00:00 app[web.2]: fn=submit_job job_id=1052:apns:97 app_id=app-one devices=1000
  311. 2015-04-22T15:26:47.512202+00:00 app[web.2]: fn=submit_job job_id=1052:apns:99 app_id=app-one devices=1000
  312. 2015-04-22T15:26:51.034165+00:00 app[web.2]: fn=submit_job job_id=1052:apns:100 app_id=app-one devices=1000
  313. 2015-04-22T15:26:51.089407+00:00 app[web.2]: fn=submit_job job_id=1052:apns:102 app_id=app-one devices=1000
  314. 2015-04-22T15:26:51.168268+00:00 app[web.2]: fn=submit_job job_id=1052:apns:104 app_id=app-one devices=1000
  315. 2015-04-22T15:26:51.223236+00:00 app[web.2]: fn=submit_job job_id=1052:apns:106 app_id=app-one devices=1000
  316. 2015-04-22T15:26:51.062040+00:00 app[web.2]: fn=submit_job job_id=1052:apns:101 app_id=app-one devices=1000
  317. 2015-04-22T15:26:51.112521+00:00 app[web.2]: fn=submit_job job_id=1052:apns:103 app_id=app-one devices=1000
  318. 2015-04-22T15:26:51.201628+00:00 app[web.2]: fn=submit_job job_id=1052:apns:105 app_id=app-one devices=1000
  319. 2015-04-22T15:26:51.249208+00:00 app[web.2]: fn=submit_job job_id=1052:apns:107 app_id=app-one devices=1000
  320. 2015-04-22T15:26:51.303391+00:00 app[web.2]: fn=submit_job job_id=1052:apns:108 app_id=app-one devices=1000
  321. 2015-04-22T15:26:51.370112+00:00 app[web.2]: fn=submit_job job_id=1052:apns:109 app_id=app-one devices=1000
  322. 2015-04-22T15:26:52.419926+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=bcf56cf3-cbab-488d-ae73-ceb29b3df365 fwd="54.159.16.8" dyno=web.2 connect=1ms service=5ms status=200 bytes=143
  323. 2015-04-22T15:26:52.422047+00:00 app[web.2]: request_id=bcf56cf3-cbab-488d-ae73-ceb29b3df365 ns=api at=start
  324. 2015-04-22T15:26:52.424455+00:00 app[web.2]: request_id=bcf56cf3-cbab-488d-ae73-ceb29b3df365 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  325. 2015-04-22T15:26:52.424542+00:00 app[web.2]: request_id=bcf56cf3-cbab-488d-ae73-ceb29b3df365 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.527
  326. 2015-04-22T15:26:52.424779+00:00 app[web.2]: [3:INFO] 2015-04-22 15:26:52 :: Status: 200, Content-Length: 18, Response Time: 3.72ms
  327. 2015-04-22T15:26:52.423625+00:00 app[web.2]: request_id=bcf56cf3-cbab-488d-ae73-ceb29b3df365 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  328. 2015-04-22T15:26:52.446924+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=16
  329. 2015-04-22T15:26:52.703500+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  330. 2015-04-22T15:26:57.471819+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=8fae2f7c-fcc9-4139-b1fd-40254a3d8ec6 fwd="54.159.16.8" dyno=web.2 connect=3ms service=8ms status=200 bytes=143
  331. 2015-04-22T15:26:57.466536+00:00 app[web.2]: request_id=8fae2f7c-fcc9-4139-b1fd-40254a3d8ec6 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  332. 2015-04-22T15:26:57.484379+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=14
  333. 2015-04-22T15:26:57.464601+00:00 app[web.2]: request_id=8fae2f7c-fcc9-4139-b1fd-40254a3d8ec6 ns=api at=start
  334. 2015-04-22T15:26:57.467397+00:00 app[web.2]: request_id=8fae2f7c-fcc9-4139-b1fd-40254a3d8ec6 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  335. 2015-04-22T15:26:57.467482+00:00 app[web.2]: request_id=8fae2f7c-fcc9-4139-b1fd-40254a3d8ec6 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.948
  336. 2015-04-22T15:26:57.467713+00:00 app[web.2]: [3:INFO] 2015-04-22 15:26:57 :: Status: 200, Content-Length: 18, Response Time: 4.06ms
  337. 2015-04-22T15:26:57.653833+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  338. 2015-04-22T15:42:22.642062+00:00 app[web.2]: try=1 device_count=110000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154839091220
  339. 2015-04-22T15:42:22.642155+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  340. 2015-04-22T15:42:22.642231+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  341. 2015-04-22T15:42:22.642322+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  342. 2015-04-22T15:42:22.642407+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  343. 2015-04-22T15:42:22.642532+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  344. 2015-04-22T15:42:22.642618+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  345. 2015-04-22T15:42:22.642705+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  346. 2015-04-22T15:42:22.642790+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  347. 2015-04-22T15:42:22.642915+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  348. 2015-04-22T15:42:22.643002+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  349. 2015-04-22T15:42:22.643089+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  350. 2015-04-22T15:42:22.643177+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  351. 2015-04-22T15:42:22.643352+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  352. 2015-04-22T15:42:22.643485+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  353. 2015-04-22T15:42:22.643572+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  354. 2015-04-22T15:42:22.643657+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  355. 2015-04-22T15:42:22.643742+00:00 app[web.2]: try=1 device_count=110000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154839091220 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  356. 2015-04-22T15:42:23.643080+00:00 app[web.2]: fn=submit_job job_id=1052:apns:110 app_id=app-one devices=1000
  357. 2015-04-22T15:42:23.685793+00:00 app[web.2]: fn=submit_job job_id=1052:apns:112 app_id=app-one devices=1000
  358. 2015-04-22T15:42:23.786770+00:00 app[web.2]: fn=submit_job job_id=1052:apns:114 app_id=app-one devices=1000
  359. 2015-04-22T15:42:23.826496+00:00 app[web.2]: fn=submit_job job_id=1052:apns:116 app_id=app-one devices=1000
  360. 2015-04-22T15:42:23.868254+00:00 app[web.2]: fn=submit_job job_id=1052:apns:118 app_id=app-one devices=1000
  361. 2015-04-22T15:42:23.665425+00:00 app[web.2]: fn=submit_job job_id=1052:apns:111 app_id=app-one devices=1000
  362. 2015-04-22T15:42:23.709394+00:00 app[web.2]: fn=submit_job job_id=1052:apns:113 app_id=app-one devices=1000
  363. 2015-04-22T15:42:23.806803+00:00 app[web.2]: fn=submit_job job_id=1052:apns:115 app_id=app-one devices=1000
  364. 2015-04-22T15:42:23.848094+00:00 app[web.2]: fn=submit_job job_id=1052:apns:117 app_id=app-one devices=1000
  365. 2015-04-22T15:42:23.888974+00:00 app[web.2]: fn=submit_job job_id=1052:apns:119 app_id=app-one devices=1000
  366. 2015-04-22T15:42:25.342483+00:00 app[web.2]: fn=submit_job job_id=1052:apns:120 app_id=app-one devices=1000
  367. 2015-04-22T15:42:25.386574+00:00 app[web.2]: fn=submit_job job_id=1052:apns:122 app_id=app-one devices=1000
  368. 2015-04-22T15:42:25.428472+00:00 app[web.2]: fn=submit_job job_id=1052:apns:124 app_id=app-one devices=1000
  369. 2015-04-22T15:42:25.486734+00:00 app[web.2]: fn=submit_job job_id=1052:apns:126 app_id=app-one devices=1000
  370. 2015-04-22T15:42:25.527214+00:00 app[web.2]: fn=submit_job job_id=1052:apns:128 app_id=app-one devices=1000
  371. 2015-04-22T15:42:25.365619+00:00 app[web.2]: fn=submit_job job_id=1052:apns:121 app_id=app-one devices=1000
  372. 2015-04-22T15:42:25.406372+00:00 app[web.2]: fn=submit_job job_id=1052:apns:123 app_id=app-one devices=1000
  373. 2015-04-22T15:42:25.452955+00:00 app[web.2]: fn=submit_job job_id=1052:apns:125 app_id=app-one devices=1000
  374. 2015-04-22T15:42:25.505693+00:00 app[web.2]: fn=submit_job job_id=1052:apns:127 app_id=app-one devices=1000
  375. 2015-04-22T15:42:25.547524+00:00 app[web.2]: fn=submit_job job_id=1052:apns:129 app_id=app-one devices=1000
  376. 2015-04-22T15:42:31.272447+00:00 app[web.2]: request_id=10b7892f-a2ef-4cdc-af26-a7c89a26b051 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  377. 2015-04-22T15:42:31.309320+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=14
  378. 2015-04-22T15:42:31.270729+00:00 app[web.2]: request_id=10b7892f-a2ef-4cdc-af26-a7c89a26b051 ns=api at=start
  379. 2015-04-22T15:42:31.273307+00:00 app[web.2]: request_id=10b7892f-a2ef-4cdc-af26-a7c89a26b051 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  380. 2015-04-22T15:42:31.273422+00:00 app[web.2]: request_id=10b7892f-a2ef-4cdc-af26-a7c89a26b051 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.681
  381. 2015-04-22T15:42:31.273846+00:00 app[web.2]: [3:INFO] 2015-04-22 15:42:31 :: Status: 200, Content-Length: 18, Response Time: 4.00ms
  382. 2015-04-22T15:42:31.493997+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  383. 2015-04-22T15:42:36.388673+00:00 app[web.1]: request_id=6ef70924-308f-4526-9b6a-9387673ee1f2 ns=api at=start
  384. 2015-04-22T15:42:36.898192+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=6ef70924-308f-4526-9b6a-9387673ee1f2 fwd="54.159.16.8" dyno=web.1 connect=4ms service=667ms status=200 bytes=143
  385. 2015-04-22T15:42:36.880121+00:00 app[web.1]: request_id=6ef70924-308f-4526-9b6a-9387673ee1f2 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  386. 2015-04-22T15:42:36.942151+00:00 app[web.1]: task_id=1052 fn=finished_jobs jobs=4
  387. 2015-04-22T15:42:36.890749+00:00 app[web.1]: request_id=6ef70924-308f-4526-9b6a-9387673ee1f2 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  388. 2015-04-22T15:42:36.890939+00:00 app[web.1]: request_id=6ef70924-308f-4526-9b6a-9387673ee1f2 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=502.265
  389. 2015-04-22T15:42:36.891520+00:00 app[web.1]: [3:INFO] 2015-04-22 15:42:36 :: Status: 200, Content-Length: 18, Response Time: 661.69ms
  390. 2015-04-22T15:42:37.565480+00:00 app[web.1]: task_id=1052 fn=all_jobs_finished? finished=false
  391. 2015-04-22T15:42:41.300981+00:00 app[web.2]: request_id=9fb318b4-f861-46fa-a6b7-f69c1c5d2538 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  392. 2015-04-22T15:42:41.307071+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=2
  393. 2015-04-22T15:42:41.296880+00:00 app[web.2]: request_id=9fb318b4-f861-46fa-a6b7-f69c1c5d2538 ns=api at=start
  394. 2015-04-22T15:42:41.302075+00:00 app[web.2]: request_id=9fb318b4-f861-46fa-a6b7-f69c1c5d2538 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  395. 2015-04-22T15:42:41.302118+00:00 app[web.2]: request_id=9fb318b4-f861-46fa-a6b7-f69c1c5d2538 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=5.242
  396. 2015-04-22T15:42:41.302486+00:00 app[web.2]: [3:INFO] 2015-04-22 15:42:41 :: Status: 200, Content-Length: 18, Response Time: 6.97ms
  397. 2015-04-22T15:42:41.337389+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  398. 2015-04-22T15:42:41.300982+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=9fb318b4-f861-46fa-a6b7-f69c1c5d2538 fwd="54.159.16.8" dyno=web.2 connect=0ms service=9ms status=200 bytes=143
  399. 2015-04-22T15:58:00.626189+00:00 app[web.2]: try=1 device_count=130000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154570610320
  400. 2015-04-22T15:58:00.626273+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  401. 2015-04-22T15:58:00.626353+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  402. 2015-04-22T15:58:00.626478+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  403. 2015-04-22T15:58:00.626572+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  404. 2015-04-22T15:58:00.626630+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  405. 2015-04-22T15:58:00.626729+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  406. 2015-04-22T15:58:00.626819+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  407. 2015-04-22T15:58:00.626944+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  408. 2015-04-22T15:58:00.627124+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  409. 2015-04-22T15:58:00.627126+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  410. 2015-04-22T15:58:00.627198+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  411. 2015-04-22T15:58:00.627241+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  412. 2015-04-22T15:58:00.627320+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  413. 2015-04-22T15:58:00.627449+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  414. 2015-04-22T15:58:00.627525+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  415. 2015-04-22T15:58:00.627597+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  416. 2015-04-22T15:58:00.627680+00:00 app[web.2]: try=1 device_count=130000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154570610320 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  417. 2015-04-22T15:58:02.664892+00:00 app[web.2]: fn=submit_job job_id=1052:apns:130 app_id=app-one devices=1000
  418. 2015-04-22T15:58:02.855951+00:00 app[web.2]: fn=submit_job job_id=1052:apns:132 app_id=app-one devices=1000
  419. 2015-04-22T15:58:02.821180+00:00 app[web.2]: fn=submit_job job_id=1052:apns:131 app_id=app-one devices=1000
  420. 2015-04-22T15:58:02.930003+00:00 app[web.2]: fn=submit_job job_id=1052:apns:133 app_id=app-one devices=1000
  421. 2015-04-22T15:58:03.043630+00:00 app[web.2]: fn=submit_job job_id=1052:apns:135 app_id=app-one devices=1000
  422. 2015-04-22T15:58:02.970151+00:00 app[web.2]: fn=submit_job job_id=1052:apns:134 app_id=app-one devices=1000
  423. 2015-04-22T15:58:03.426588+00:00 app[web.2]: fn=submit_job job_id=1052:apns:136 app_id=app-one devices=1000
  424. 2015-04-22T15:58:03.573458+00:00 app[web.2]: fn=submit_job job_id=1052:apns:138 app_id=app-one devices=1000
  425. 2015-04-22T15:58:03.458468+00:00 app[web.2]: fn=submit_job job_id=1052:apns:137 app_id=app-one devices=1000
  426. 2015-04-22T15:58:03.656450+00:00 app[web.2]: fn=submit_job job_id=1052:apns:139 app_id=app-one devices=1000
  427. 2015-04-22T15:58:06.585476+00:00 app[web.2]: fn=submit_job job_id=1052:apns:140 app_id=app-one devices=1000
  428. 2015-04-22T15:58:06.745655+00:00 app[web.2]: fn=submit_job job_id=1052:apns:142 app_id=app-one devices=1000
  429. 2015-04-22T15:58:06.693339+00:00 app[web.2]: fn=submit_job job_id=1052:apns:141 app_id=app-one devices=1000
  430. 2015-04-22T15:58:06.795642+00:00 app[web.2]: fn=submit_job job_id=1052:apns:143 app_id=app-one devices=1000
  431. 2015-04-22T15:58:06.854319+00:00 app[web.2]: fn=submit_job job_id=1052:apns:144 app_id=app-one devices=1000
  432. 2015-04-22T15:58:06.923776+00:00 app[web.2]: fn=submit_job job_id=1052:apns:146 app_id=app-one devices=1000
  433. 2015-04-22T15:58:06.875685+00:00 app[web.2]: fn=submit_job job_id=1052:apns:145 app_id=app-one devices=1000
  434. 2015-04-22T15:58:06.990432+00:00 app[web.2]: fn=submit_job job_id=1052:apns:147 app_id=app-one devices=1000
  435. 2015-04-22T15:58:07.070877+00:00 app[web.2]: fn=submit_job job_id=1052:apns:149 app_id=app-one devices=1000
  436. 2015-04-22T15:58:07.040768+00:00 app[web.2]: fn=submit_job job_id=1052:apns:148 app_id=app-one devices=1000
  437. 2015-04-22T15:58:09.950904+00:00 app[web.1]: request_id=1eb8432d-b587-4918-8ee5-d2a225ccbfd2 ns=api at=start
  438. 2015-04-22T15:58:09.953492+00:00 app[web.1]: request_id=1eb8432d-b587-4918-8ee5-d2a225ccbfd2 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  439. 2015-04-22T15:58:09.953577+00:00 app[web.1]: request_id=1eb8432d-b587-4918-8ee5-d2a225ccbfd2 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=2.708
  440. 2015-04-22T15:58:09.953868+00:00 app[web.1]: [3:INFO] 2015-04-22 15:58:09 :: Status: 200, Content-Length: 18, Response Time: 4.27ms
  441. 2015-04-22T15:58:10.160379+00:00 app[web.1]: task_id=1052 fn=all_jobs_finished? finished=false
  442. 2015-04-22T15:58:09.952860+00:00 app[web.1]: request_id=1eb8432d-b587-4918-8ee5-d2a225ccbfd2 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  443. 2015-04-22T15:58:09.980217+00:00 app[web.1]: task_id=1052 fn=finished_jobs jobs=15
  444. 2015-04-22T15:58:09.956198+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=1eb8432d-b587-4918-8ee5-d2a225ccbfd2 fwd="54.159.16.8" dyno=web.1 connect=1ms service=6ms status=200 bytes=143
  445. 2015-04-22T15:58:14.990513+00:00 app[web.2]: request_id=cb49dcfd-9c26-4b4a-be33-ceb578059738 ns=api at=start
  446. 2015-04-22T15:58:14.997052+00:00 app[web.2]: request_id=cb49dcfd-9c26-4b4a-be33-ceb578059738 app_id=api api=/control fn=request path=/control/push/finished/1052 body=
  447. 2015-04-22T15:58:14.997055+00:00 app[web.2]: request_id=cb49dcfd-9c26-4b4a-be33-ceb578059738 app_id=api api=/control task_id=1052 fn=response status=200 length=18 body=["{\"task_id\":\"1052\"}"]
  448. 2015-04-22T15:58:14.997057+00:00 app[web.2]: request_id=cb49dcfd-9c26-4b4a-be33-ceb578059738 app_id=api api=/control task_id=1052 ns=api at=finish elapsed=6.273
  449. 2015-04-22T15:58:14.997059+00:00 app[web.2]: [3:INFO] 2015-04-22 15:58:14 :: Status: 200, Content-Length: 18, Response Time: 7.41ms
  450. 2015-04-22T15:58:15.012352+00:00 app[web.2]: task_id=1052 fn=finished_jobs jobs=5
  451. 2015-04-22T15:58:14.996878+00:00 heroku[router]: at=info method=PUT path="/control/push/finished/1052" host=mocapi-load.herokuapp.com request_id=cb49dcfd-9c26-4b4a-be33-ceb578059738 fwd="54.159.16.8" dyno=web.2 connect=2ms service=10ms status=200 bytes=143
  452. 2015-04-22T15:58:15.294473+00:00 app[web.2]: task_id=1052 fn=all_jobs_finished? finished=false
  453. 2015-04-22T16:12:32.926042+00:00 heroku[web.2]: State changed from up to down
  454. 2015-04-22T16:12:32.927465+00:00 heroku[web.2]: State changed from down to starting
  455. 2015-04-22T16:12:36.463032+00:00 heroku[web.2]: Starting process with command `bundle exec ruby pserver.rb -sv -e prod -p 16095`
  456. 2015-04-22T16:12:40.234866+00:00 app[web.2]: [3:INFO] 2015-04-22 16:12:40 :: Starting server on 0.0.0.0:16095 in production mode. Watch out for stones.
  457. 2015-04-22T16:12:40.782362+00:00 heroku[web.2]: State changed from starting to up
  458. 2015-04-22T16:13:41.623379+00:00 app[web.2]: try=1 device_count=150000 at=exception class=Mongo::ConnectionFailure message="Operation failed with the following exception: Connection timed out" exception_id=70154569457960
  459. 2015-04-22T16:13:42.493594+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:336:in rescue in receive_message_on_socket"
  460. 2015-04-22T16:13:43.309410+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:328:in receive_message_on_socket"
  461. 2015-04-22T16:13:44.433110+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:189:in receive_header"
  462. 2015-04-22T16:13:45.760628+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:180:in receive"
  463. 2015-04-22T16:13:46.377173+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/networking.rb:144:in receive_message"
  464. 2015-04-22T16:13:46.892522+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:601:in send_get_more"
  465. 2015-04-22T16:13:47.618719+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:46:in block in send_get_more_with_newrelic_trace"
  466. 2015-04-22T16:13:49.356092+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:69:in block in trace_execution_scoped"
  467. 2015-04-22T16:13:50.104996+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer_helpers.rb:82:in trace_execution_scoped"
  468. 2015-04-22T16:13:50.875702+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/method_tracer.rb:67:in trace_execution_scoped"
  469. 2015-04-22T16:13:52.146664+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/newrelic_mongo-0.0.4/lib/newrelic_mongo/instrumentation/mongo.rb:45:in send_get_more_with_newrelic_trace"
  470. 2015-04-22T16:13:53.766028+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:518:in num_remaining"
  471. 2015-04-22T16:13:53.197547+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:531:in refresh"
  472. 2015-04-22T16:13:55.333536+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/mongo-1.11.1/lib/mongo/cursor.rb:187:in has_next?"
  473. 2015-04-22T16:13:56.824101+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/api/job_dispatcher.rb:154:in block in db_devices_has_next?"
  474. 2015-04-22T16:13:58.135561+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in call"
  475. 2015-04-22T16:13:59.190022+00:00 app[web.2]: try=1 device_count=150000 at=exception class="Operation failed with the following exception: Connection timed out" exception_id=70154569457960 site="/app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool"
  476. Not found
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!