API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 23rd, 2015
160
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.13 KB | None | 0 0
raw download clone embed print report
  1. > people <- read.df(sqlContext, "/tmp/people.json", "json")
  2. 15/12/23 15:47:51 INFO JSONRelation: Listing hdfs://phdns01.cloud.hortonworks.com:8020/tmp/people.json on driver
  3. 15/12/23 15:47:52 INFO MemoryStore: ensureFreeSpace(307688) called with curMem=340225, maxMem=555755765
  4. 15/12/23 15:47:52 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 300.5 KB, free 529.4 MB)
  5. 15/12/23 15:47:52 INFO MemoryStore: ensureFreeSpace(26198) called with curMem=647913, maxMem=555755765
  6. 15/12/23 15:47:52 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 25.6 KB, free 529.4 MB)
  7. 15/12/23 15:47:52 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:32818 (size: 25.6 KB, free: 530.0 MB)
  8. 15/12/23 15:47:52 INFO SparkContext: Created broadcast 4 from loadDF at NativeMethodAccessorImpl.java:-2
  9. 15/12/23 15:47:52 INFO FileInputFormat: Total input paths to process : 1
  10. 15/12/23 15:47:52 INFO SparkContext: Starting job: loadDF at NativeMethodAccessorImpl.java:-2
  11. 15/12/23 15:47:52 INFO DAGScheduler: Got job 1 (loadDF at NativeMethodAccessorImpl.java:-2) with 2 output partitions
  12. 15/12/23 15:47:52 INFO DAGScheduler: Final stage: ResultStage 1(loadDF at NativeMethodAccessorImpl.java:-2)
  13. 15/12/23 15:47:52 INFO DAGScheduler: Parents of final stage: List()
  14. 15/12/23 15:47:52 INFO DAGScheduler: Missing parents: List()
  15. 15/12/23 15:47:52 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[13] at loadDF at NativeMethodAccessorImpl.java:-2), which has no missing parents
  16. 15/12/23 15:47:52 INFO MemoryStore: ensureFreeSpace(4056) called with curMem=674111, maxMem=555755765
  17. 15/12/23 15:47:52 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.0 KB, free 529.4 MB)
  18. 15/12/23 15:47:52 INFO MemoryStore: ensureFreeSpace(2294) called with curMem=678167, maxMem=555755765
  19. 15/12/23 15:47:52 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.2 KB, free 529.4 MB)
  20. 15/12/23 15:47:52 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:32818 (size: 2.2 KB, free: 530.0 MB)
  21. 15/12/23 15:47:52 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:861
  22. 15/12/23 15:47:52 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[13] at loadDF at NativeMethodAccessorImpl.java:-2)
  23. 15/12/23 15:47:52 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
  24. 15/12/23 15:47:52 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, ANY, 2166 bytes)
  25. 15/12/23 15:47:52 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, localhost, ANY, 2166 bytes)
  26. 15/12/23 15:47:52 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
  27. 15/12/23 15:47:52 INFO Executor: Running task 1.0 in stage 1.0 (TID 3)
  28. 15/12/23 15:47:52 INFO HadoopRDD: Input split: hdfs://phdns01.cloud.hortonworks.com:8020/tmp/people.json:0+36
  29. 15/12/23 15:47:52 INFO HadoopRDD: Input split: hdfs://phdns01.cloud.hortonworks.com:8020/tmp/people.json:36+37
  30. 15/12/23 15:47:52 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 2845 bytes result sent to driver
  31. 15/12/23 15:47:52 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 2845 bytes result sent to driver
  32. 15/12/23 15:47:52 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 40 ms on localhost (1/2)
  33. 15/12/23 15:47:52 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 42 ms on localhost (2/2)
  34. 15/12/23 15:47:52 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
  35. 15/12/23 15:47:52 INFO DAGScheduler: ResultStage 1 (loadDF at NativeMethodAccessorImpl.java:-2) finished in 0.043 s
  36. 15/12/23 15:47:52 INFO DAGScheduler: Job 1 finished: loadDF at NativeMethodAccessorImpl.java:-2, took 0.055354 s
  37. > registerTempTable(people, "people")
  38. > teenagers <- sql(sqlContext, "SELECT name FROM people WHERE age >= 13 AND age <= 19")
  39. > head(teenagers)
  40. 15/12/23 15:48:13 INFO MemoryStore: ensureFreeSpace(78632) called with curMem=680461, maxMem=555755765
  41. 15/12/23 15:48:13 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 76.8 KB, free 529.3 MB)
  42. 15/12/23 15:48:13 INFO MemoryStore: ensureFreeSpace(26101) called with curMem=759093, maxMem=555755765
  43. 15/12/23 15:48:13 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 25.5 KB, free 529.3 MB)
  44. 15/12/23 15:48:13 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:32818 (size: 25.5 KB, free: 529.9 MB)
  45. 15/12/23 15:48:13 INFO SparkContext: Created broadcast 6 from dfToCols at NativeMethodAccessorImpl.java:-2
  46. 15/12/23 15:48:13 INFO MemoryStore: ensureFreeSpace(307688) called with curMem=785194, maxMem=555755765
  47. 15/12/23 15:48:13 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 300.5 KB, free 529.0 MB)
  48. 15/12/23 15:48:13 INFO MemoryStore: ensureFreeSpace(26198) called with curMem=1092882, maxMem=555755765
  49. 15/12/23 15:48:13 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 25.6 KB, free 528.9 MB)
  50. 15/12/23 15:48:13 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:32818 (size: 25.6 KB, free: 529.9 MB)
  51. 15/12/23 15:48:13 INFO SparkContext: Created broadcast 7 from dfToCols at NativeMethodAccessorImpl.java:-2
  52. 15/12/23 15:48:13 INFO FileInputFormat: Total input paths to process : 1
  53. 15/12/23 15:48:13 INFO SparkContext: Starting job: dfToCols at NativeMethodAccessorImpl.java:-2
  54. 15/12/23 15:48:13 INFO DAGScheduler: Got job 2 (dfToCols at NativeMethodAccessorImpl.java:-2) with 1 output partitions
  55. 15/12/23 15:48:13 INFO DAGScheduler: Final stage: ResultStage 2(dfToCols at NativeMethodAccessorImpl.java:-2)
  56. 15/12/23 15:48:13 INFO DAGScheduler: Parents of final stage: List()
  57. 15/12/23 15:48:13 INFO DAGScheduler: Missing parents: List()
  58. 15/12/23 15:48:13 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[19] at dfToCols at NativeMethodAccessorImpl.java:-2), which has no missing parents
  59. 15/12/23 15:48:13 INFO MemoryStore: ensureFreeSpace(7264) called with curMem=1119080, maxMem=555755765
  60. 15/12/23 15:48:13 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 7.1 KB, free 528.9 MB)
  61. 15/12/23 15:48:13 INFO MemoryStore: ensureFreeSpace(3848) called with curMem=1126344, maxMem=555755765
  62. 15/12/23 15:48:13 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.8 KB, free 528.9 MB)
  63. 15/12/23 15:48:13 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:32818 (size: 3.8 KB, free: 529.9 MB)
  64. 15/12/23 15:48:13 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:861
  65. 15/12/23 15:48:13 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[19] at dfToCols at NativeMethodAccessorImpl.java:-2)
  66. 15/12/23 15:48:13 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
  67. 15/12/23 15:48:13 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, ANY, 2166 bytes)
  68. 15/12/23 15:48:13 INFO Executor: Running task 0.0 in stage 2.0 (TID 4)
  69. 15/12/23 15:48:13 INFO HadoopRDD: Input split: hdfs://phdns01.cloud.hortonworks.com:8020/tmp/people.json:0+36
  70. 15/12/23 15:48:14 INFO GeneratePredicate: Code generated in 198.977883 ms
  71. 15/12/23 15:48:14 INFO GenerateMutableProjection: Code generated in 18.126055 ms
  72. 15/12/23 15:48:14 INFO Executor: Finished task 0.0 in stage 2.0 (TID 4). 2352 bytes result sent to driver
  73. 15/12/23 15:48:14 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 271 ms on localhost (1/1)
  74. 15/12/23 15:48:14 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
  75. 15/12/23 15:48:14 INFO DAGScheduler: ResultStage 2 (dfToCols at NativeMethodAccessorImpl.java:-2) finished in 0.273 s
  76. 15/12/23 15:48:14 INFO DAGScheduler: Job 2 finished: dfToCols at NativeMethodAccessorImpl.java:-2, took 0.287883 s
  77. 15/12/23 15:48:14 INFO SparkContext: Starting job: dfToCols at NativeMethodAccessorImpl.java:-2
  78. 15/12/23 15:48:14 INFO DAGScheduler: Got job 3 (dfToCols at NativeMethodAccessorImpl.java:-2) with 1 output partitions
  79. 15/12/23 15:48:14 INFO DAGScheduler: Final stage: ResultStage 3(dfToCols at NativeMethodAccessorImpl.java:-2)
  80. 15/12/23 15:48:14 INFO DAGScheduler: Parents of final stage: List()
  81. 15/12/23 15:48:14 INFO DAGScheduler: Missing parents: List()
  82. 15/12/23 15:48:14 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[19] at dfToCols at NativeMethodAccessorImpl.java:-2), which has no missing parents
  83. 15/12/23 15:48:14 INFO MemoryStore: ensureFreeSpace(7264) called with curMem=1130192, maxMem=555755765
  84. 15/12/23 15:48:14 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 7.1 KB, free 528.9 MB)
  85. 15/12/23 15:48:14 INFO MemoryStore: ensureFreeSpace(3848) called with curMem=1137456, maxMem=555755765
  86. 15/12/23 15:48:14 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 3.8 KB, free 528.9 MB)
  87. 15/12/23 15:48:14 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:32818 (size: 3.8 KB, free: 529.9 MB)
  88. 15/12/23 15:48:14 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:861
  89. 15/12/23 15:48:14 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[19] at dfToCols at NativeMethodAccessorImpl.java:-2)
  90. 15/12/23 15:48:14 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
  91. 15/12/23 15:48:14 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, ANY, 2166 bytes)
  92. 15/12/23 15:48:14 INFO Executor: Running task 0.0 in stage 3.0 (TID 5)
  93. 15/12/23 15:48:14 INFO HadoopRDD: Input split: hdfs://phdns01.cloud.hortonworks.com:8020/tmp/people.json:36+37
  94. 15/12/23 15:48:14 INFO Executor: Finished task 0.0 in stage 3.0 (TID 5). 2629 bytes result sent to driver
  95. 15/12/23 15:48:14 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 26 ms on localhost (1/1)
  96. 15/12/23 15:48:14 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
  97. 15/12/23 15:48:14 INFO DAGScheduler: ResultStage 3 (dfToCols at NativeMethodAccessorImpl.java:-2) finished in 0.026 s
  98. 15/12/23 15:48:14 INFO DAGScheduler: Job 3 finished: dfToCols at NativeMethodAccessorImpl.java:-2, took 0.037463 s
  99. name
  100. 1 Justin
  101. >
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!