API tools faq
paste
Guest User

driver before crash

a guest
Oct 29th, 2015
102
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 42.92 KB | None | 0 0
raw download clone embed print report
  1. 2015-10-18 12:17:18,838 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170120000 ms
  2. 2015-10-18 12:17:18,838 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170120000 ms
  3. 2015-10-18 12:17:18,839 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170120000 ms
  4. 2015-10-18 12:17:18,844 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170120000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170120000'
  5. 2015-10-18 12:17:25,892 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170120000
  6. 2015-10-18 12:17:25,894 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170120000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170120000', took 33144 bytes and 7050 ms
  7. 2015-10-18 12:17:34,050 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283509 for time 1445170640000 ms for checkpointing
  8. 2015-10-18 12:17:34,059 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170120000 ms
  9. 2015-10-18 12:17:34,063 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282599' for time 1445170080000 ms
  10. 2015-10-18 12:17:34,063 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170120000 ms
  11. 2015-10-18 12:17:34,064 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170640000 ms
  12. 2015-10-18 12:17:34,064 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170640000 ms
  13. 2015-10-18 12:17:34,065 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170640000 ms
  14. 2015-10-18 12:17:34,068 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170640000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170640000'
  15. 2015-10-18 12:17:34,090 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170440000
  16. 2015-10-18 12:17:34,091 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170640000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170640000', took 33568 bytes and 23 ms
  17.  
  18. ...
  19.  
  20. 2015-10-18 12:17:35,881 [yarn-scheduler-ask-am-thread-pool-2] WARN org.apache.spark.rpc.akka.AkkaRpcEndpointRef - Error sending message [message = KillExecutors(List(3))] in 3 attempts
  21. org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
  22. at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
  23. at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
  24. at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
  25. at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
  26. at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:242)
  27. at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
  28. at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
  29. at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$2.apply$mcV$sp(YarnSchedulerBackend.scala:136)
  30. at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$2.apply(YarnSchedulerBackend.scala:136)
  31. at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$2.apply(YarnSchedulerBackend.scala:136)
  32. at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
  33. at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
  34. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  35. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  36. at java.lang.Thread.run(Thread.java:745)
  37. Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
  38. at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
  39. at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
  40. at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
  41. at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  42. at scala.concurrent.Await$.result(package.scala:190)
  43. at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:241)
  44. ... 10 more
  45. 2015-10-18 12:17:35,882 [yarn-scheduler-ask-am-thread-pool-5] ERROR org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint - Sending KillExecutors(List(3)) to AM was unsuccessful
  46. org.apache.spark.SparkException: Error sending message [message = KillExecutors(List(3))]
  47. at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:118)
  48. at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
  49. at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$2.apply$mcV$sp(YarnSchedulerBackend.scala:136)
  50. at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$2.apply(YarnSchedulerBackend.scala:136)
  51. at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$2.apply(YarnSchedulerBackend.scala:136)
  52. at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
  53. at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
  54. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  55. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  56. at java.lang.Thread.run(Thread.java:745)
  57. Caused by: org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
  58. at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
  59. at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
  60. at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
  61. at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
  62. at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:242)
  63. at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
  64. ... 9 more
  65. Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
  66. at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
  67. at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
  68. at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
  69. at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  70. at scala.concurrent.Await$.result(package.scala:190)
  71. at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:241)
  72. ... 10 more
  73.  
  74.  
  75. ...
  76.  
  77.  
  78. 2015-10-18 12:17:37,420 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282695 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282695, new parent is RDD 283526
  79. 2015-10-18 12:17:38,762 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170140000 ms
  80. 2015-10-18 12:17:38,762 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170140000 ms
  81. 2015-10-18 12:17:38,762 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170140000 ms
  82. 2015-10-18 12:17:38,766 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170140000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170140000'
  83. 2015-10-18 12:17:38,799 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170140000
  84. 2015-10-18 12:17:38,801 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170140000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170140000', took 33127 bytes and 35 ms
  85. 2015-10-18 12:17:38,801 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170140000 ms
  86. 2015-10-18 12:17:38,805 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282631' for time 1445170100000 ms
  87. 2015-10-18 12:17:38,806 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170140000 ms
  88. 2015-10-18 12:17:40,017 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283533 for time 1445170660000 ms for checkpointing
  89. 2015-10-18 12:17:40,024 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283542 for time 1445170660000 ms for checkpointing
  90. 2015-10-18 12:17:40,034 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170660000 ms
  91. 2015-10-18 12:17:40,034 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170660000 ms
  92. 2015-10-18 12:17:40,034 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170660000 ms
  93. 2015-10-18 12:17:40,038 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170660000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170660000'
  94. 2015-10-18 12:17:47,093 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170460000
  95. 2015-10-18 12:17:47,095 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170660000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170660000', took 33559 bytes and 7057 ms
  96. 2015-10-18 12:17:57,095 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282718 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282718, new parent is RDD 283559
  97. 2015-10-18 12:18:00,029 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283575 for time 1445170680000 ms for checkpointing
  98. 2015-10-18 12:18:00,038 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170680000 ms
  99. 2015-10-18 12:18:00,038 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170680000 ms
  100. 2015-10-18 12:18:00,038 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170680000 ms
  101. 2015-10-18 12:18:00,042 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170680000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170680000'
  102. 2015-10-18 12:18:00,063 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170480000
  103. 2015-10-18 12:18:00,066 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170680000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170680000', took 33963 bytes and 24 ms
  104. 2015-10-18 12:18:06,106 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282727 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282727, new parent is RDD 283592
  105. 2015-10-18 12:18:07,474 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170160000 ms
  106. 2015-10-18 12:18:07,474 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170160000 ms
  107. 2015-10-18 12:18:07,475 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170160000 ms
  108. 2015-10-18 12:18:07,480 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170160000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170160000'
  109. 2015-10-18 12:18:07,503 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170160000
  110. 2015-10-18 12:18:07,505 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170160000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170160000', took 33484 bytes and 25 ms
  111. 2015-10-18 12:18:07,505 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170160000 ms
  112. 2015-10-18 12:18:07,509 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282398' for time 1445169960000 ms
  113. 2015-10-18 12:18:07,512 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282663' for time 1445170120000 ms
  114. 2015-10-18 12:18:07,512 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170160000 ms
  115. 2015-10-18 12:18:13,052 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282759 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282759, new parent is RDD 283593
  116. 2015-10-18 12:18:14,670 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170180000 ms
  117. 2015-10-18 12:18:14,670 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170180000 ms
  118. 2015-10-18 12:18:14,671 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170180000 ms
  119. 2015-10-18 12:18:14,674 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170180000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170180000'
  120. 2015-10-18 12:18:14,698 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170180000
  121. 2015-10-18 12:18:14,700 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170180000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170180000', took 33051 bytes and 26 ms
  122. 2015-10-18 12:18:14,700 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170180000 ms
  123. 2015-10-18 12:18:14,704 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282695' for time 1445170140000 ms
  124. 2015-10-18 12:18:14,704 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170180000 ms
  125. 2015-10-18 12:18:20,022 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283609 for time 1445170700000 ms for checkpointing
  126. 2015-10-18 12:18:20,031 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170700000 ms
  127. 2015-10-18 12:18:20,031 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170700000 ms
  128. 2015-10-18 12:18:20,032 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170700000 ms
  129. 2015-10-18 12:18:20,035 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170700000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170700000'
  130. 2015-10-18 12:18:20,062 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170500000
  131. 2015-10-18 12:18:20,064 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170700000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170700000', took 33434 bytes and 29 ms
  132. 2015-10-18 12:18:20,423 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282791 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282791, new parent is RDD 283626
  133. 2015-10-18 12:18:29,160 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170200000 ms
  134. 2015-10-18 12:18:29,160 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170200000 ms
  135. 2015-10-18 12:18:29,160 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170200000 ms
  136. 2015-10-18 12:18:29,164 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170200000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170200000'
  137. 2015-10-18 12:18:29,189 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170200000
  138. 2015-10-18 12:18:29,191 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170200000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170200000', took 33030 bytes and 27 ms
  139. 2015-10-18 12:18:29,192 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170200000 ms
  140. 2015-10-18 12:18:29,195 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282727' for time 1445170160000 ms
  141. 2015-10-18 12:18:29,195 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170200000 ms
  142. 2015-10-18 12:18:34,565 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282823 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282823, new parent is RDD 283627
  143. 2015-10-18 12:18:36,184 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170220000 ms
  144. 2015-10-18 12:18:36,185 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170220000 ms
  145. 2015-10-18 12:18:36,186 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170220000 ms
  146. 2015-10-18 12:18:36,189 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170220000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170220000'
  147. 2015-10-18 12:18:36,235 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170220000
  148. 2015-10-18 12:18:36,239 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170220000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170220000', took 33077 bytes and 50 ms
  149. 2015-10-18 12:18:36,239 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170220000 ms
  150. 2015-10-18 12:18:36,244 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282759' for time 1445170180000 ms
  151. 2015-10-18 12:18:36,244 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170220000 ms
  152. 2015-10-18 12:18:38,692 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282855 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282855, new parent is RDD 283628
  153. 2015-10-18 12:18:41,239 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283644 for time 1445170720000 ms for checkpointing
  154. 2015-10-18 12:18:41,250 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170720000 ms
  155. 2015-10-18 12:18:41,250 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170720000 ms
  156. 2015-10-18 12:18:41,251 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170720000 ms
  157. 2015-10-18 12:18:41,255 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170720000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170720000'
  158. 2015-10-18 12:18:41,302 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170520000
  159. 2015-10-18 12:18:41,305 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170720000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170720000', took 33218 bytes and 50 ms
  160. 2015-10-18 12:18:49,947 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170240000 ms
  161. 2015-10-18 12:18:49,947 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170240000 ms
  162. 2015-10-18 12:18:49,948 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170240000 ms
  163. 2015-10-18 12:18:49,951 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170240000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170240000'
  164. 2015-10-18 12:18:49,984 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170240000
  165. 2015-10-18 12:18:49,992 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170240000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170240000', took 33066 bytes and 41 ms
  166. 2015-10-18 12:18:49,993 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170240000 ms
  167. 2015-10-18 12:18:49,999 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282791' for time 1445170200000 ms
  168. 2015-10-18 12:18:50,000 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170240000 ms
  169. 2015-10-18 12:18:52,310 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282878 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282878, new parent is RDD 283661
  170. 2015-10-18 12:18:55,408 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282887 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282887, new parent is RDD 283662
  171. 2015-10-18 12:19:00,050 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283678 for time 1445170740000 ms for checkpointing
  172. 2015-10-18 12:19:00,059 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170740000 ms
  173. 2015-10-18 12:19:00,059 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170740000 ms
  174. 2015-10-18 12:19:00,060 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170740000 ms
  175. 2015-10-18 12:19:00,063 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170740000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170740000'
  176. 2015-10-18 12:19:00,092 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170540000
  177. 2015-10-18 12:19:00,095 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170740000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170740000', took 33344 bytes and 32 ms
  178. 2015-10-18 12:19:03,921 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170260000 ms
  179. 2015-10-18 12:19:03,921 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170260000 ms
  180. 2015-10-18 12:19:03,921 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170260000 ms
  181. 2015-10-18 12:19:03,925 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170260000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170260000'
  182. 2015-10-18 12:19:03,949 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170260000
  183. 2015-10-18 12:19:03,952 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170260000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170260000', took 33153 bytes and 27 ms
  184. 2015-10-18 12:19:03,952 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170260000 ms
  185. 2015-10-18 12:19:03,957 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282558' for time 1445170060000 ms
  186. 2015-10-18 12:19:03,961 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282823' for time 1445170220000 ms
  187. 2015-10-18 12:19:03,961 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170260000 ms
  188. 2015-10-18 12:19:06,489 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282919 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282919, new parent is RDD 283695
  189. 2015-10-18 12:19:10,650 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170280000 ms
  190. 2015-10-18 12:19:10,650 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170280000 ms
  191. 2015-10-18 12:19:10,651 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170280000 ms
  192. 2015-10-18 12:19:10,657 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170280000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170280000'
  193. 2015-10-18 12:19:10,678 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170280000
  194. 2015-10-18 12:19:10,680 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170280000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170280000', took 32517 bytes and 23 ms
  195. 2015-10-18 12:19:10,680 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170280000 ms
  196. 2015-10-18 12:19:10,683 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282855' for time 1445170240000 ms
  197. 2015-10-18 12:19:10,683 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170280000 ms
  198. 2015-10-18 12:19:13,530 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282951 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282951, new parent is RDD 283696
  199. 2015-10-18 12:19:17,630 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170300000 ms
  200. 2015-10-18 12:19:17,630 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170300000 ms
  201. 2015-10-18 12:19:17,631 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170300000 ms
  202. 2015-10-18 12:19:17,635 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170300000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170300000'
  203. 2015-10-18 12:19:17,671 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170300000
  204. 2015-10-18 12:19:17,673 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170300000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170300000', took 31929 bytes and 39 ms
  205. 2015-10-18 12:19:17,674 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170300000 ms
  206. 2015-10-18 12:19:17,678 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282887' for time 1445170260000 ms
  207. 2015-10-18 12:19:17,678 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170300000 ms
  208. 2015-10-18 12:19:20,015 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283703 for time 1445170760000 ms for checkpointing
  209. 2015-10-18 12:19:20,023 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283712 for time 1445170760000 ms for checkpointing
  210. 2015-10-18 12:19:20,032 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170760000 ms
  211. 2015-10-18 12:19:20,032 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170760000 ms
  212. 2015-10-18 12:19:20,032 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170760000 ms
  213. 2015-10-18 12:19:20,035 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170760000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170760000'
  214. 2015-10-18 12:19:20,079 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170560000
  215. 2015-10-18 12:19:20,083 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170760000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170760000', took 32489 bytes and 48 ms
  216. 2015-10-18 12:19:20,481 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 282983 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282983, new parent is RDD 283729
  217. 2015-10-18 12:19:28,970 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170320000 ms
  218. 2015-10-18 12:19:28,970 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170320000 ms
  219. 2015-10-18 12:19:28,971 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170320000 ms
  220. 2015-10-18 12:19:28,979 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170320000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170320000'
  221. 2015-10-18 12:19:29,014 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170320000
  222. 2015-10-18 12:19:29,018 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170320000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170320000', took 31852 bytes and 39 ms
  223. 2015-10-18 12:19:29,018 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170320000 ms
  224. 2015-10-18 12:19:29,022 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282919' for time 1445170280000 ms
  225. 2015-10-18 12:19:29,022 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170320000 ms
  226. 2015-10-18 12:19:32,376 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 283015 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283015, new parent is RDD 283730
  227. 2015-10-18 12:19:40,794 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283746 for time 1445170780000 ms for checkpointing
  228. 2015-10-18 12:19:40,803 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170780000 ms
  229. 2015-10-18 12:19:40,803 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170780000 ms
  230. 2015-10-18 12:19:40,803 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170780000 ms
  231. 2015-10-18 12:19:40,807 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170780000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170780000'
  232. 2015-10-18 12:19:40,826 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170580000
  233. 2015-10-18 12:19:40,829 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170780000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170780000', took 32315 bytes and 22 ms
  234. 2015-10-18 12:19:47,911 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170340000 ms
  235. 2015-10-18 12:19:47,911 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170340000 ms
  236. 2015-10-18 12:19:47,912 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170340000 ms
  237. 2015-10-18 12:19:47,916 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170340000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170340000'
  238. 2015-10-18 12:19:47,938 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170340000
  239. 2015-10-18 12:19:47,940 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170340000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170340000', took 31765 bytes and 24 ms
  240. 2015-10-18 12:19:47,940 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170340000 ms
  241. 2015-10-18 12:19:47,943 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282951' for time 1445170300000 ms
  242. 2015-10-18 12:19:47,943 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170340000 ms
  243. 2015-10-18 12:19:51,474 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 283039 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283039, new parent is RDD 283763
  244. 2015-10-18 12:19:52,488 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 283048 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283048, new parent is RDD 283764
  245. 2015-10-18 12:19:57,409 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170360000 ms
  246. 2015-10-18 12:19:57,409 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170360000 ms
  247. 2015-10-18 12:19:57,410 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170360000 ms
  248. 2015-10-18 12:19:57,414 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170360000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170360000'
  249. 2015-10-18 12:19:57,443 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170360000
  250. 2015-10-18 12:19:57,447 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170360000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170360000', took 31281 bytes and 34 ms
  251. 2015-10-18 12:19:57,447 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170360000 ms
  252. 2015-10-18 12:19:57,452 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282718' for time 1445170160000 ms
  253. 2015-10-18 12:19:57,458 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-282983' for time 1445170320000 ms
  254. 2015-10-18 12:19:57,458 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170360000 ms
  255. 2015-10-18 12:19:59,829 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 283081 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283081, new parent is RDD 283765
  256. 2015-10-18 12:20:02,667 [JobGenerator] INFO org.apache.spark.streaming.dstream.StateDStream - Marking RDD 283781 for time 1445170800000 ms for checkpointing
  257. 2015-10-18 12:20:02,678 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170800000 ms
  258. 2015-10-18 12:20:02,678 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170800000 ms
  259. 2015-10-18 12:20:02,679 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170800000 ms
  260. 2015-10-18 12:20:02,684 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170800000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170800000'
  261. 2015-10-18 12:20:02,717 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170600000
  262. 2015-10-18 12:20:02,721 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170800000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170800000', took 31891 bytes and 37 ms
  263. 2015-10-18 12:20:04,238 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170380000 ms
  264. 2015-10-18 12:20:04,238 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170380000 ms
  265. 2015-10-18 12:20:04,239 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170380000 ms
  266. 2015-10-18 12:20:04,243 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170380000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170380000'
  267. 2015-10-18 12:20:04,275 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170380000
  268. 2015-10-18 12:20:04,279 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170380000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170380000', took 31317 bytes and 36 ms
  269. 2015-10-18 12:20:04,279 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170380000 ms
  270. 2015-10-18 12:20:04,284 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283015' for time 1445170340000 ms
  271. 2015-10-18 12:20:04,284 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170380000 ms
  272. 2015-10-18 12:20:08,668 [streaming-job-executor-0] INFO org.apache.spark.rdd.ReliableRDDCheckpointData - Done checkpointing RDD 283113 to hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283113, new parent is RDD 283798
  273. 2015-10-18 12:20:12,536 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 1445170400000 ms
  274. 2015-10-18 12:20:12,536 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updating checkpoint data for time 1445170400000 ms
  275. 2015-10-18 12:20:12,537 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Updated checkpoint data for time 1445170400000 ms
  276. 2015-10-18 12:20:12,540 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Saving checkpoint for time 1445170400000 ms to file 'hdfs://root/checkpoint/checkpoint-1445170400000'
  277. 2015-10-18 12:20:12,599 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Deleting hdfs://root/checkpoint/checkpoint-1445170400000
  278. 2015-10-18 12:20:12,603 [pool-19-thread-1] INFO org.apache.spark.streaming.CheckpointWriter - Checkpoint for time 1445170400000 ms saved to file 'hdfs://root/checkpoint/checkpoint-1445170400000', took 31029 bytes and 63 ms
  279. 2015-10-18 12:20:12,603 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Clearing checkpoint data for time 1445170400000 ms
  280. 2015-10-18 12:20:12,609 [JobGenerator] INFO org.apache.spark.streaming.dstream.DStreamCheckpointData - Deleted checkpoint file 'hdfs://root/checkpoint/4c53b3c0-ec8c-43d1-b859-9f0e42ffcbdb/rdd-283048' for time 1445170360000 ms
  281. 2015-10-18 12:20:12,609 [JobGenerator] INFO org.apache.spark.streaming.DStreamGraph - Cleared checkpoint data for time 1445170400000 ms
  282.  
  283. ...
  284.  
  285. 2015-10-18 12:20:15,203 [SIGTERM handler] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL 15: SIGTERM
  286. 2015-10-18 12:20:15,232 [Thread-3] INFO org.apache.spark.streaming.StreamingContext - Invoking stop(stopGracefully=false) from shutdown hook
  287. 2015-10-18 12:20:15,235 [Thread-3] INFO org.apache.spark.streaming.scheduler.JobGenerator - Stopping JobGenerator immediately
  288. 2015-10-18 12:20:15,236 [Thread-3] INFO org.apache.spark.streaming.util.RecurringTimer - Stopped timer for JobGenerator after time 1445170800000
  289. 2015-10-18 12:20:15,243 [dag-scheduler-event-loop] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(9920) called with curMem=2401004, maxMem=1030823608
  290.  
  291. ...
  292.  
  293. 2015-10-18 12:20:15,322 [Thread-3] INFO org.apache.spark.streaming.CheckpointWriter - CheckpointWriter executor terminated ? true, waited for 1 ms.
  294. 2015-10-18 12:20:15,324 [Thread-3] INFO org.apache.spark.streaming.scheduler.JobGenerator - Stopped JobGenerator
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!