API tools faq
paste
Advertisement
Guest User

spark-shell output

a guest
Jun 23rd, 2015
80
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 32.00 KB | None | 0 0
raw download clone embed print report
  1. scala> ssc.start()
  2. 15/06/23 21:58:04 INFO ReceiverTracker: ReceiverTracker started
  3. 15/06/23 21:58:04 INFO ForEachDStream: metadataCleanupDelay = -1
  4. 15/06/23 21:58:04 INFO MappedDStream: metadataCleanupDelay = -1
  5. 15/06/23 21:58:04 INFO KafkaInputDStream: metadataCleanupDelay = -1
  6. 15/06/23 21:58:04 INFO KafkaInputDStream: Slide time = 10000 ms
  7. 15/06/23 21:58:04 INFO KafkaInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
  8. 15/06/23 21:58:04 INFO KafkaInputDStream: Checkpoint interval = null
  9. 15/06/23 21:58:04 INFO KafkaInputDStream: Remember duration = 10000 ms
  10. 15/06/23 21:58:04 INFO KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@67836d4
  11. 15/06/23 21:58:04 INFO MappedDStream: Slide time = 10000 ms
  12. 15/06/23 21:58:04 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
  13. 15/06/23 21:58:04 INFO MappedDStream: Checkpoint interval = null
  14. 15/06/23 21:58:04 INFO MappedDStream: Remember duration = 10000 ms
  15. 15/06/23 21:58:04 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@73973e77
  16. 15/06/23 21:58:04 INFO ForEachDStream: Slide time = 10000 ms
  17. 15/06/23 21:58:04 INFO ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)
  18. 15/06/23 21:58:04 INFO ForEachDStream: Checkpoint interval = null
  19. 15/06/23 21:58:04 INFO ForEachDStream: Remember duration = 10000 ms
  20. 15/06/23 21:58:04 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@65011123
  21. 15/06/23 21:58:04 INFO RecurringTimer: Started timer for JobGenerator at time 1435089490000
  22. 15/06/23 21:58:04 INFO JobGenerator: Started JobGenerator at 1435089490000 ms
  23. 15/06/23 21:58:04 INFO JobScheduler: Started JobScheduler
  24.  
  25. scala> ssc.awaitTermination()
  26. 15/06/23 21:58:04 INFO SparkContext: Starting job: start at <console>:27
  27. 15/06/23 21:58:04 INFO DAGScheduler: Registering RDD 2 (start at <console>:27)
  28. 15/06/23 21:58:04 INFO DAGScheduler: Got job 0 (start at <console>:27) with 20 output partitions (allowLocal=false)
  29. 15/06/23 21:58:04 INFO DAGScheduler: Final stage: Stage 1(start at <console>:27)
  30. 15/06/23 21:58:04 INFO DAGScheduler: Parents of final stage: List(Stage 0)
  31. 15/06/23 21:58:04 INFO DAGScheduler: Missing parents: List(Stage 0)
  32. 15/06/23 21:58:04 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[2] at start at <console>:27), which has no missing parents
  33. 15/06/23 21:58:04 INFO MemoryStore: ensureFreeSpace(2720) called with curMem=0, maxMem=278019440
  34. 15/06/23 21:58:04 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.7 KB, free 265.1 MB)
  35. 15/06/23 21:58:04 INFO MemoryStore: ensureFreeSpace(1948) called with curMem=2720, maxMem=278019440
  36. 15/06/23 21:58:04 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1948.0 B, free 265.1 MB)
  37. 15/06/23 21:58:04 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:47906 (size: 1948.0 B, free: 265.1 MB)
  38. 15/06/23 21:58:04 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
  39. 15/06/23 21:58:04 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:838
  40. 15/06/23 21:58:04 INFO DAGScheduler: Submitting 50 missing tasks from Stage 0 (MappedRDD[2] at start at <console>:27)
  41. 15/06/23 21:58:04 INFO TaskSchedulerImpl: Adding task set 0.0 with 50 tasks
  42. 15/06/23 21:58:04 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  43. 15/06/23 21:58:07 INFO BlockManagerMasterActor: Registering block manager 192.168.168.130:58464 with 265.1 MB RAM, BlockManagerId(20150616-205038-2192091328-5050-6847-0, 192.168.168.130, 58464)
  44. 15/06/23 21:58:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.168.130:58464 (size: 1948.0 B, free: 265.1 MB)
  45. 15/06/23 21:58:07 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3227 ms on 192.168.168.130 (1/50)
  46. 15/06/23 21:58:08 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  47. 15/06/23 21:58:08 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 19 ms on 192.168.168.130 (2/50)
  48. 15/06/23 21:58:09 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  49. 15/06/23 21:58:09 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 15 ms on 192.168.168.130 (3/50)
  50. 15/06/23 21:58:10 INFO JobScheduler: Added jobs for time 1435089490000 ms
  51. 15/06/23 21:58:10 INFO JobScheduler: Starting job streaming job 1435089490000 ms.0 from job set of time 1435089490000 ms
  52. DataListener.listen() [pairRDD = MappedRDD[5] at map at <console>:28]
  53. 15/06/23 21:58:10 INFO SparkContext: Starting job: foreachRDD at <console>:31
  54. 15/06/23 21:58:10 INFO DAGScheduler: Job 1 finished: foreachRDD at <console>:31, took 0.000660 s
  55. DataListener.listen() [pairRDD.count = 0]
  56. 15/06/23 21:58:10 INFO SparkContext: Starting job: foreachRDD at <console>:31
  57. 15/06/23 21:58:10 INFO DAGScheduler: Job 2 finished: foreachRDD at <console>:31, took 0.000037 s
  58. 15/06/23 21:58:10 INFO SparkContext: Starting job: foreachRDD at <console>:31
  59. 15/06/23 21:58:10 INFO DAGScheduler: Job 3 finished: foreachRDD at <console>:31, took 0.000028 s
  60. 15/06/23 21:58:10 INFO JobScheduler: Finished job streaming job 1435089490000 ms.0 from job set of time 1435089490000 ms
  61. 15/06/23 21:58:10 INFO JobScheduler: Total delay: 0.042 s for time 1435089490000 ms (execution: 0.022 s)
  62. 15/06/23 21:58:10 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer()
  63. 15/06/23 21:58:10 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  64. 15/06/23 21:58:10 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 14 ms on 192.168.168.130 (4/50)
  65. 15/06/23 21:58:11 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  66. 15/06/23 21:58:11 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 13 ms on 192.168.168.130 (5/50)
  67. 15/06/23 21:58:12 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  68. 15/06/23 21:58:12 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 13 ms on 192.168.168.130 (6/50)
  69. 15/06/23 21:58:13 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  70. 15/06/23 21:58:13 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 14 ms on 192.168.168.130 (7/50)
  71. 15/06/23 21:58:14 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  72. 15/06/23 21:58:14 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 14 ms on 192.168.168.130 (8/50)
  73. 15/06/23 21:58:15 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  74. 15/06/23 21:58:15 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 14 ms on 192.168.168.130 (9/50)
  75. 15/06/23 21:58:16 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  76. 15/06/23 21:58:16 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 13 ms on 192.168.168.130 (10/50)
  77. 15/06/23 21:58:17 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  78. 15/06/23 21:58:17 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 13 ms on 192.168.168.130 (11/50)
  79. 15/06/23 21:58:18 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  80. 15/06/23 21:58:18 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 15 ms on 192.168.168.130 (12/50)
  81. 15/06/23 21:58:19 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  82. 15/06/23 21:58:19 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 14 ms on 192.168.168.130 (13/50)
  83. 15/06/23 21:58:20 INFO JobScheduler: Added jobs for time 1435089500000 ms
  84. DataListener.listen() [pairRDD = MappedRDD[7] at map at <console>:28]
  85. 15/06/23 21:58:20 INFO SparkContext: Starting job: foreachRDD at <console>:31
  86. 15/06/23 21:58:20 INFO DAGScheduler: Job 4 finished: foreachRDD at <console>:31, took 0.000022 s
  87. DataListener.listen() [pairRDD.count = 0]
  88. 15/06/23 21:58:20 INFO SparkContext: Starting job: foreachRDD at <console>:31
  89. 15/06/23 21:58:20 INFO DAGScheduler: Job 5 finished: foreachRDD at <console>:31, took 0.000014 s
  90. 15/06/23 21:58:20 INFO SparkContext: Starting job: foreachRDD at <console>:31
  91. 15/06/23 21:58:20 INFO DAGScheduler: Job 6 finished: foreachRDD at <console>:31, took 0.000016 s
  92. 15/06/23 21:58:20 INFO JobScheduler: Starting job streaming job 1435089500000 ms.0 from job set of time 1435089500000 ms
  93. 15/06/23 21:58:20 INFO JobScheduler: Finished job streaming job 1435089500000 ms.0 from job set of time 1435089500000 ms
  94. 15/06/23 21:58:20 INFO JobScheduler: Total delay: 0.007 s for time 1435089500000 ms (execution: 0.004 s)
  95. 15/06/23 21:58:20 INFO MappedRDD: Removing RDD 5 from persistence list
  96. 15/06/23 21:58:20 INFO BlockManager: Removing RDD 5
  97. 15/06/23 21:58:20 INFO BlockRDD: Removing RDD 4 from persistence list
  98. 15/06/23 21:58:20 INFO BlockManager: Removing RDD 4
  99. 15/06/23 21:58:20 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[4] at createStream at <console>:28 of time 1435089500000 ms
  100. 15/06/23 21:58:20 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer()
  101. 15/06/23 21:58:20 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  102. 15/06/23 21:58:20 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 11 ms on 192.168.168.130 (14/50)
  103. 15/06/23 21:58:21 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  104. 15/06/23 21:58:21 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 13 ms on 192.168.168.130 (15/50)
  105. 15/06/23 21:58:22 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  106. 15/06/23 21:58:22 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 15 ms on 192.168.168.130 (16/50)
  107. 15/06/23 21:58:23 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  108. 15/06/23 21:58:23 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 15 ms on 192.168.168.130 (17/50)
  109. 15/06/23 21:58:24 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  110. 15/06/23 21:58:24 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 15 ms on 192.168.168.130 (18/50)
  111. 15/06/23 21:58:25 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  112. 15/06/23 21:58:25 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 19 ms on 192.168.168.130 (19/50)
  113. 15/06/23 21:58:26 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  114. 15/06/23 21:58:26 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 13 ms on 192.168.168.130 (20/50)
  115. 15/06/23 21:58:27 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  116. 15/06/23 21:58:27 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 14 ms on 192.168.168.130 (21/50)
  117. 15/06/23 21:58:28 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  118. 15/06/23 21:58:28 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 22 ms on 192.168.168.130 (22/50)
  119. 15/06/23 21:58:29 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  120. 15/06/23 21:58:29 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 13 ms on 192.168.168.130 (23/50)
  121. 15/06/23 21:58:30 INFO JobScheduler: Starting job streaming job 1435089510000 ms.0 from job set of time 1435089510000 ms
  122. DataListener.listen() [pairRDD = MappedRDD[9] at map at <console>:28]
  123. 15/06/23 21:58:30 INFO JobScheduler: Added jobs for time 1435089510000 ms
  124. 15/06/23 21:58:30 INFO SparkContext: Starting job: foreachRDD at <console>:31
  125. 15/06/23 21:58:30 INFO DAGScheduler: Job 7 finished: foreachRDD at <console>:31, took 0.000027 s
  126. DataListener.listen() [pairRDD.count = 0]
  127. 15/06/23 21:58:30 INFO SparkContext: Starting job: foreachRDD at <console>:31
  128. 15/06/23 21:58:30 INFO DAGScheduler: Job 8 finished: foreachRDD at <console>:31, took 0.000019 s
  129. 15/06/23 21:58:30 INFO SparkContext: Starting job: foreachRDD at <console>:31
  130. 15/06/23 21:58:30 INFO DAGScheduler: Job 9 finished: foreachRDD at <console>:31, took 0.000024 s
  131. 15/06/23 21:58:30 INFO JobScheduler: Finished job streaming job 1435089510000 ms.0 from job set of time 1435089510000 ms
  132. 15/06/23 21:58:30 INFO JobScheduler: Total delay: 0.010 s for time 1435089510000 ms (execution: 0.008 s)
  133. 15/06/23 21:58:30 INFO MappedRDD: Removing RDD 7 from persistence list
  134. 15/06/23 21:58:30 INFO BlockManager: Removing RDD 7
  135. 15/06/23 21:58:30 INFO BlockRDD: Removing RDD 6 from persistence list
  136. 15/06/23 21:58:30 INFO BlockManager: Removing RDD 6
  137. 15/06/23 21:58:30 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[6] at createStream at <console>:28 of time 1435089510000 ms
  138. 15/06/23 21:58:30 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1435089490000 ms)
  139. 15/06/23 21:58:30 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  140. 15/06/23 21:58:30 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 14 ms on 192.168.168.130 (24/50)
  141. 15/06/23 21:58:31 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  142. 15/06/23 21:58:31 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 14 ms on 192.168.168.130 (25/50)
  143. 15/06/23 21:58:32 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  144. 15/06/23 21:58:32 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 15 ms on 192.168.168.130 (26/50)
  145. 15/06/23 21:58:33 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  146. 15/06/23 21:58:33 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 16 ms on 192.168.168.130 (27/50)
  147. 15/06/23 21:58:34 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  148. 15/06/23 21:58:34 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 13 ms on 192.168.168.130 (28/50)
  149. 15/06/23 21:58:35 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  150. 15/06/23 21:58:35 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 10 ms on 192.168.168.130 (29/50)
  151. 15/06/23 21:58:36 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  152. 15/06/23 21:58:36 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 11 ms on 192.168.168.130 (30/50)
  153. 15/06/23 21:58:37 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  154. 15/06/23 21:58:37 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 13 ms on 192.168.168.130 (31/50)
  155. 15/06/23 21:58:38 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  156. 15/06/23 21:58:38 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 15 ms on 192.168.168.130 (32/50)
  157. 15/06/23 21:58:39 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  158. 15/06/23 21:58:39 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 10 ms on 192.168.168.130 (33/50)
  159. DataListener.listen() [pairRDD = MappedRDD[11] at map at <console>:28]
  160. 15/06/23 21:58:40 INFO SparkContext: Starting job: foreachRDD at <console>:31
  161. 15/06/23 21:58:40 INFO JobScheduler: Starting job streaming job 1435089520000 ms.0 from job set of time 1435089520000 ms
  162. 15/06/23 21:58:40 INFO DAGScheduler: Job 10 finished: foreachRDD at <console>:31, took 0.000024 s
  163. DataListener.listen() [pairRDD.count = 0]
  164. 15/06/23 21:58:40 INFO SparkContext: Starting job: foreachRDD at <console>:31
  165. 15/06/23 21:58:40 INFO DAGScheduler: Job 11 finished: foreachRDD at <console>:31, took 0.000018 s
  166. 15/06/23 21:58:40 INFO JobScheduler: Added jobs for time 1435089520000 ms
  167. 15/06/23 21:58:40 INFO SparkContext: Starting job: foreachRDD at <console>:31
  168. 15/06/23 21:58:40 INFO DAGScheduler: Job 12 finished: foreachRDD at <console>:31, took 0.000015 s
  169. 15/06/23 21:58:40 INFO JobScheduler: Finished job streaming job 1435089520000 ms.0 from job set of time 1435089520000 ms
  170. 15/06/23 21:58:40 INFO MappedRDD: Removing RDD 9 from persistence list
  171. 15/06/23 21:58:40 INFO BlockManager: Removing RDD 9
  172. 15/06/23 21:58:40 INFO BlockRDD: Removing RDD 8 from persistence list
  173. 15/06/23 21:58:40 INFO BlockManager: Removing RDD 8
  174. 15/06/23 21:58:40 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[8] at createStream at <console>:28 of time 1435089520000 ms
  175. 15/06/23 21:58:40 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1435089500000 ms)
  176. 15/06/23 21:58:40 INFO JobScheduler: Total delay: 0.008 s for time 1435089520000 ms (execution: 0.004 s)
  177. 15/06/23 21:58:40 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  178. 15/06/23 21:58:40 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 12 ms on 192.168.168.130 (34/50)
  179. 15/06/23 21:58:41 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  180. 15/06/23 21:58:41 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 11 ms on 192.168.168.130 (35/50)
  181. 15/06/23 21:58:42 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  182. 15/06/23 21:58:42 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 12 ms on 192.168.168.130 (36/50)
  183. 15/06/23 21:58:43 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  184. 15/06/23 21:58:43 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 33 ms on 192.168.168.130 (37/50)
  185. 15/06/23 21:58:44 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  186. 15/06/23 21:58:44 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 9 ms on 192.168.168.130 (38/50)
  187. 15/06/23 21:58:45 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  188. 15/06/23 21:58:45 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 12 ms on 192.168.168.130 (39/50)
  189. 15/06/23 21:58:46 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  190. 15/06/23 21:58:46 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 13 ms on 192.168.168.130 (40/50)
  191. 15/06/23 21:58:47 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  192. 15/06/23 21:58:47 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 14 ms on 192.168.168.130 (41/50)
  193. 15/06/23 21:58:48 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  194. 15/06/23 21:58:48 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 14 ms on 192.168.168.130 (42/50)
  195. 15/06/23 21:58:49 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  196. 15/06/23 21:58:49 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 10 ms on 192.168.168.130 (43/50)
  197. 15/06/23 21:58:50 INFO JobScheduler: Added jobs for time 1435089530000 ms
  198. DataListener.listen() [pairRDD = MappedRDD[13] at map at <console>:28]
  199. 15/06/23 21:58:50 INFO JobScheduler: Starting job streaming job 1435089530000 ms.0 from job set of time 1435089530000 ms
  200. 15/06/23 21:58:50 INFO SparkContext: Starting job: foreachRDD at <console>:31
  201. 15/06/23 21:58:50 INFO DAGScheduler: Job 13 finished: foreachRDD at <console>:31, took 0.000029 s
  202. DataListener.listen() [pairRDD.count = 0]
  203. 15/06/23 21:58:50 INFO SparkContext: Starting job: foreachRDD at <console>:31
  204. 15/06/23 21:58:50 INFO DAGScheduler: Job 14 finished: foreachRDD at <console>:31, took 0.000024 s
  205. 15/06/23 21:58:50 INFO SparkContext: Starting job: foreachRDD at <console>:31
  206. 15/06/23 21:58:50 INFO DAGScheduler: Job 15 finished: foreachRDD at <console>:31, took 0.000014 s
  207. 15/06/23 21:58:50 INFO JobScheduler: Finished job streaming job 1435089530000 ms.0 from job set of time 1435089530000 ms
  208. 15/06/23 21:58:50 INFO JobScheduler: Total delay: 0.006 s for time 1435089530000 ms (execution: 0.003 s)
  209. 15/06/23 21:58:50 INFO MappedRDD: Removing RDD 11 from persistence list
  210. 15/06/23 21:58:50 INFO BlockManager: Removing RDD 11
  211. 15/06/23 21:58:50 INFO BlockRDD: Removing RDD 10 from persistence list
  212. 15/06/23 21:58:50 INFO BlockManager: Removing RDD 10
  213. 15/06/23 21:58:50 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[10] at createStream at <console>:28 of time 1435089530000 ms
  214. 15/06/23 21:58:50 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1435089510000 ms)
  215. 15/06/23 21:58:50 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  216. 15/06/23 21:58:50 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 13 ms on 192.168.168.130 (44/50)
  217. 15/06/23 21:58:51 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  218. 15/06/23 21:58:51 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 18 ms on 192.168.168.130 (45/50)
  219. 15/06/23 21:58:52 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  220. 15/06/23 21:58:52 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 8 ms on 192.168.168.130 (46/50)
  221. 15/06/23 21:58:53 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  222. 15/06/23 21:58:53 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 13 ms on 192.168.168.130 (47/50)
  223. 15/06/23 21:58:54 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  224. 15/06/23 21:58:54 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 12 ms on 192.168.168.130 (48/50)
  225. 15/06/23 21:58:55 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, 192.168.168.130, PROCESS_LOCAL, 1249 bytes)
  226. 15/06/23 21:58:55 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 11 ms on 192.168.168.130 (49/50)
  227. 15/06/23 21:58:56 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, 192.168.168.130, PROCESS_LOCAL, 1306 bytes)
  228. 15/06/23 21:58:56 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 13 ms on 192.168.168.130 (50/50)
  229. 15/06/23 21:58:56 INFO DAGScheduler: Stage 0 (start at <console>:27) finished in 52.071 s
  230. 15/06/23 21:58:56 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
  231. 15/06/23 21:58:56 INFO DAGScheduler: looking for newly runnable stages
  232. 15/06/23 21:58:56 INFO DAGScheduler: running: Set()
  233. 15/06/23 21:58:56 INFO DAGScheduler: waiting: Set(Stage 1)
  234. 15/06/23 21:58:56 INFO DAGScheduler: failed: Set()
  235. 15/06/23 21:58:56 INFO DAGScheduler: Missing parents for Stage 1: List()
  236. 15/06/23 21:58:56 INFO DAGScheduler: Submitting Stage 1 (ShuffledRDD[3] at start at <console>:27), which is now runnable
  237. 15/06/23 21:58:56 INFO MemoryStore: ensureFreeSpace(2232) called with curMem=4668, maxMem=278019440
  238. 15/06/23 21:58:56 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.2 KB, free 265.1 MB)
  239. 15/06/23 21:58:56 INFO MemoryStore: ensureFreeSpace(1639) called with curMem=6900, maxMem=278019440
  240. 15/06/23 21:58:56 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1639.0 B, free 265.1 MB)
  241. 15/06/23 21:58:56 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:47906 (size: 1639.0 B, free: 265.1 MB)
  242. 15/06/23 21:58:56 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
  243. 15/06/23 21:58:56 INFO SparkContext: Created broadcast 1 from getCallSite at DStream.scala:294
  244. 15/06/23 21:58:56 INFO DAGScheduler: Submitting 20 missing tasks from Stage 1 (ShuffledRDD[3] at start at <console>:27)
  245. 15/06/23 21:58:56 INFO TaskSchedulerImpl: Adding task set 1.0 with 20 tasks
  246. 15/06/23 21:58:56 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 50, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  247. 15/06/23 21:58:56 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.168.130:58464 (size: 1639.0 B, free: 265.1 MB)
  248. 15/06/23 21:58:56 INFO MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to [email protected]:45453
  249. 15/06/23 21:58:56 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 302 bytes
  250. 15/06/23 21:58:56 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 50) in 81 ms on 192.168.168.130 (1/20)
  251. 15/06/23 21:58:57 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 51, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  252. 15/06/23 21:58:57 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 51) in 14 ms on 192.168.168.130 (2/20)
  253. 15/06/23 21:58:58 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 52, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  254. 15/06/23 21:58:58 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 52) in 15 ms on 192.168.168.130 (3/20)
  255. 15/06/23 21:58:59 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 53, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  256. 15/06/23 21:58:59 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 53) in 12 ms on 192.168.168.130 (4/20)
  257. 15/06/23 21:59:00 INFO JobScheduler: Added jobs for time 1435089540000 ms
  258. DataListener.listen() [pairRDD = MappedRDD[15] at map at <console>:28]
  259. 15/06/23 21:59:00 INFO SparkContext: Starting job: foreachRDD at <console>:31
  260. 15/06/23 21:59:00 INFO DAGScheduler: Job 16 finished: foreachRDD at <console>:31, took 0.000047 s
  261. DataListener.listen() [pairRDD.count = 0]
  262. 15/06/23 21:59:00 INFO SparkContext: Starting job: foreachRDD at <console>:31
  263. 15/06/23 21:59:00 INFO DAGScheduler: Job 17 finished: foreachRDD at <console>:31, took 0.000016 s
  264. 15/06/23 21:59:00 INFO SparkContext: Starting job: foreachRDD at <console>:31
  265. 15/06/23 21:59:00 INFO DAGScheduler: Job 18 finished: foreachRDD at <console>:31, took 0.000013 s
  266. 15/06/23 21:59:00 INFO JobScheduler: Starting job streaming job 1435089540000 ms.0 from job set of time 1435089540000 ms
  267. 15/06/23 21:59:00 INFO JobScheduler: Finished job streaming job 1435089540000 ms.0 from job set of time 1435089540000 ms
  268. 15/06/23 21:59:00 INFO MappedRDD: Removing RDD 13 from persistence list
  269. 15/06/23 21:59:00 INFO JobScheduler: Total delay: 0.007 s for time 1435089540000 ms (execution: 0.000 s)
  270. 15/06/23 21:59:00 INFO BlockRDD: Removing RDD 12 from persistence list
  271. 15/06/23 21:59:00 INFO BlockManager: Removing RDD 13
  272. 15/06/23 21:59:00 INFO BlockManager: Removing RDD 12
  273. 15/06/23 21:59:00 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[12] at createStream at <console>:28 of time 1435089540000 ms
  274. 15/06/23 21:59:00 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1435089520000 ms)
  275. 15/06/23 21:59:00 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 54, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  276. 15/06/23 21:59:00 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 54) in 12 ms on 192.168.168.130 (5/20)
  277. 15/06/23 21:59:01 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 55, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  278. 15/06/23 21:59:01 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 55) in 12 ms on 192.168.168.130 (6/20)
  279. 15/06/23 21:59:02 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 56, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  280. 15/06/23 21:59:02 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 56) in 13 ms on 192.168.168.130 (7/20)
  281. 15/06/23 21:59:03 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 57, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  282. 15/06/23 21:59:03 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 57) in 11 ms on 192.168.168.130 (8/20)
  283. 15/06/23 21:59:04 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 58, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  284. 15/06/23 21:59:04 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 58) in 11 ms on 192.168.168.130 (9/20)
  285. 15/06/23 21:59:05 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 59, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  286. 15/06/23 21:59:05 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 59) in 12 ms on 192.168.168.130 (10/20)
  287. 15/06/23 21:59:06 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 60, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  288. 15/06/23 21:59:06 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 60) in 15 ms on 192.168.168.130 (11/20)
  289. 15/06/23 21:59:07 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 61, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  290. 15/06/23 21:59:07 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 61) in 10 ms on 192.168.168.130 (12/20)
  291. 15/06/23 21:59:08 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 62, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  292. 15/06/23 21:59:08 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 62) in 13 ms on 192.168.168.130 (13/20)
  293. 15/06/23 21:59:09 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 63, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  294. 15/06/23 21:59:09 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 63) in 15 ms on 192.168.168.130 (14/20)
  295. DataListener.listen() [pairRDD = MappedRDD[17] at map at <console>:28]
  296. 15/06/23 21:59:10 INFO JobScheduler: Starting job streaming job 1435089550000 ms.0 from job set of time 1435089550000 ms
  297. 15/06/23 21:59:10 INFO SparkContext: Starting job: foreachRDD at <console>:31
  298. 15/06/23 21:59:10 INFO JobScheduler: Added jobs for time 1435089550000 ms
  299. 15/06/23 21:59:10 INFO DAGScheduler: Job 19 finished: foreachRDD at <console>:31, took 0.000021 s
  300. DataListener.listen() [pairRDD.count = 0]
  301. 15/06/23 21:59:10 INFO SparkContext: Starting job: foreachRDD at <console>:31
  302. 15/06/23 21:59:10 INFO DAGScheduler: Job 20 finished: foreachRDD at <console>:31, took 0.000016 s
  303. 15/06/23 21:59:10 INFO SparkContext: Starting job: foreachRDD at <console>:31
  304. 15/06/23 21:59:10 INFO DAGScheduler: Job 21 finished: foreachRDD at <console>:31, took 0.000018 s
  305. 15/06/23 21:59:10 INFO JobScheduler: Finished job streaming job 1435089550000 ms.0 from job set of time 1435089550000 ms
  306. 15/06/23 21:59:10 INFO JobScheduler: Total delay: 0.010 s for time 1435089550000 ms (execution: 0.006 s)
  307. 15/06/23 21:59:10 INFO MappedRDD: Removing RDD 15 from persistence list
  308. 15/06/23 21:59:10 INFO BlockManager: Removing RDD 15
  309. 15/06/23 21:59:10 INFO BlockRDD: Removing RDD 14 from persistence list
  310. 15/06/23 21:59:10 INFO BlockManager: Removing RDD 14
  311. 15/06/23 21:59:10 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[14] at createStream at <console>:28 of time 1435089550000 ms
  312. 15/06/23 21:59:10 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1435089530000 ms)
  313. 15/06/23 21:59:10 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 64, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  314. 15/06/23 21:59:10 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 64) in 10 ms on 192.168.168.130 (15/20)
  315. 15/06/23 21:59:11 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 65, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  316. 15/06/23 21:59:11 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 65) in 16 ms on 192.168.168.130 (16/20)
  317. 15/06/23 21:59:12 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 66, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  318. 15/06/23 21:59:12 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 66) in 11 ms on 192.168.168.130 (17/20)
  319. 15/06/23 21:59:13 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 67, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  320. 15/06/23 21:59:13 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 67) in 13 ms on 192.168.168.130 (18/20)
  321. 15/06/23 21:59:14 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 68, 192.168.168.130, PROCESS_LOCAL, 1056 bytes)
  322. 15/06/23 21:59:14 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 68) in 9 ms on 192.168.168.130 (19/20)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!