API tools faq
paste
Advertisement
Guest User

Spark log

a guest
Feb 17th, 2016
102
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 93.94 KB | None | 0 0
raw download clone embed print report
  1. log4j:ERROR setFile(null,true) call failed.
  2. java.io.FileNotFoundException: /var/log/spark/sparkdriver_abshkmodi-ssh.log (Permission denied)
  3. at java.io.FileOutputStream.open(Native Method)
  4. at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
  5. at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
  6. at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
  7. at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
  8. at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
  9. at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
  10. at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
  11. at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
  12. at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
  13. at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
  14. at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
  15. at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
  16. at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
  17. at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
  18. at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
  19. at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
  20. at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
  21. at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
  22. at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
  23. at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:275)
  24. at org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
  25. at java.lang.Class.forName0(Native Method)
  26. at java.lang.Class.forName(Class.java:278)
  27. at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
  28. at org.apache.spark.util.SparkShutdownHookManager$$anonfun$4.apply(ShutdownHookManager.scala:218)
  29. at org.apache.spark.util.SparkShutdownHookManager$$anonfun$4.apply(ShutdownHookManager.scala:218)
  30. at scala.util.Try$.apply(Try.scala:161)
  31. at org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:218)
  32. at org.apache.spark.util.ShutdownHookManager$.shutdownHooks$lzycompute(ShutdownHookManager.scala:50)
  33. at org.apache.spark.util.ShutdownHookManager$.shutdownHooks(ShutdownHookManager.scala:48)
  34. at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:189)
  35. at org.apache.spark.util.ShutdownHookManager$.<init>(ShutdownHookManager.scala:58)
  36. at org.apache.spark.util.ShutdownHookManager$.<clinit>(ShutdownHookManager.scala)
  37. at org.apache.spark.util.Utils$.createTempDir(Utils.scala:238)
  38. at org.apache.spark.repl.SparkIMain.outputDir$lzycompute(SparkIMain.scala:102)
  39. at org.apache.spark.repl.SparkIMain.outputDir(SparkIMain.scala:99)
  40. at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:115)
  41. at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
  42. at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:217)
  43. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:949)
  44. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
  45. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
  46. at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
  47. at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
  48. at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
  49. at org.apache.spark.repl.Main$.main(Main.scala:31)
  50. at org.apache.spark.repl.Main.main(Main.scala)
  51. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  52. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  53. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  54. at java.lang.reflect.Method.invoke(Method.java:606)
  55. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
  56. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  57. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  58. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
  59. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  60. log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].
  61. 16/02/14 20:19:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  62. 16/02/14 20:19:34 INFO SecurityManager: Changing view acls to: abshkmodi-ssh
  63. 16/02/14 20:19:34 INFO SecurityManager: Changing modify acls to: abshkmodi-ssh
  64. 16/02/14 20:19:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(abshkmodi-ssh); users with modify permissions: Set(abshkmodi-ssh)
  65. 16/02/14 20:19:35 INFO HttpServer: Starting HTTP Server
  66. 16/02/14 20:19:36 INFO Server: jetty-8.y.z-SNAPSHOT
  67. 16/02/14 20:19:36 INFO AbstractConnector: Started [email protected]:44188
  68. 16/02/14 20:19:36 INFO Utils: Successfully started service 'HTTP class server' on port 44188.
  69. 16/02/14 20:19:52 INFO SparkContext: Running Spark version 1.5.2
  70. 16/02/14 20:19:52 INFO SecurityManager: Changing view acls to: abshkmodi-ssh
  71. 16/02/14 20:19:52 INFO SecurityManager: Changing modify acls to: abshkmodi-ssh
  72. 16/02/14 20:19:52 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(abshkmodi-ssh); users with modify permissions: Set(abshkmodi-ssh)
  73. 16/02/14 20:19:55 INFO Slf4jLogger: Slf4jLogger started
  74. 16/02/14 20:19:55 INFO Remoting: Starting remoting
  75. 16/02/14 20:19:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:54953]
  76. 16/02/14 20:19:56 INFO Utils: Successfully started service 'sparkDriver' on port 54953.
  77. 16/02/14 20:19:56 INFO SparkEnv: Registering MapOutputTracker
  78. 16/02/14 20:19:56 INFO SparkEnv: Registering BlockManagerMaster
  79. 16/02/14 20:19:56 INFO DiskBlockManager: Created local directory at /var/tmp/spark/blockmgr-c3147fee-dd1e-42c5-bfe0-3bfdae3d6f44
  80. 16/02/14 20:19:56 INFO MemoryStore: MemoryStore started with capacity 530.3 MB
  81. 16/02/14 20:19:57 INFO HttpFileServer: HTTP File server directory is /var/tmp/spark/spark-fb3a3311-7aaf-43d4-a9db-6dfb51f97422/httpd-aa6356b3-2f9d-4a4b-ba47-1f86490d1539
  82. 16/02/14 20:19:57 INFO HttpServer: Starting HTTP Server
  83. 16/02/14 20:19:57 INFO Server: jetty-8.y.z-SNAPSHOT
  84. 16/02/14 20:19:57 INFO AbstractConnector: Started [email protected]:36615
  85. 16/02/14 20:19:57 INFO Utils: Successfully started service 'HTTP file server' on port 36615.
  86. 16/02/14 20:19:57 INFO SparkEnv: Registering OutputCommitCoordinator
  87. 16/02/14 20:19:58 INFO Server: jetty-8.y.z-SNAPSHOT
  88. 16/02/14 20:19:58 WARN AbstractLifeCycle: FAILED [email protected]:4040: java.net.BindException: Address already in use
  89. java.net.BindException: Address already in use
  90. at sun.nio.ch.Net.bind0(Native Method)
  91. at sun.nio.ch.Net.bind(Net.java:463)
  92. at sun.nio.ch.Net.bind(Net.java:455)
  93. at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
  94. at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  95. at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
  96. at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
  97. at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
  98. at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
  99. at org.spark-project.jetty.server.Server.doStart(Server.java:293)
  100. at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
  101. at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:237)
  102. at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
  103. at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
  104. at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
  105. at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
  106. at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
  107. at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
  108. at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
  109. at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
  110. at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
  111. at scala.Option.foreach(Option.scala:236)
  112. at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
  113. at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
  114. at $line3.$read$$iwC$$iwC.<init>(<console>:9)
  115. at $line3.$read$$iwC.<init>(<console>:18)
  116. at $line3.$read.<init>(<console>:20)
  117. at $line3.$read$.<init>(<console>:24)
  118. at $line3.$read$.<clinit>(<console>)
  119. at $line3.$eval$.<init>(<console>:7)
  120. at $line3.$eval$.<clinit>(<console>)
  121. at $line3.$eval.$print(<console>)
  122. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  123. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  124. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  125. at java.lang.reflect.Method.invoke(Method.java:606)
  126. at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
  127. at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
  128. at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
  129. at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
  130. at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
  131. at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
  132. at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
  133. at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
  134. at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
  135. at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
  136. at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
  137. at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
  138. at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
  139. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
  140. at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
  141. at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
  142. at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
  143. at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
  144. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
  145. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
  146. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
  147. at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
  148. at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
  149. at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
  150. at org.apache.spark.repl.Main$.main(Main.scala:31)
  151. at org.apache.spark.repl.Main.main(Main.scala)
  152. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  153. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  154. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  155. at java.lang.reflect.Method.invoke(Method.java:606)
  156. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
  157. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  158. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  159. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
  160. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  161. 16/02/14 20:19:58 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@74561c24: java.net.BindException: Address already in use
  162. java.net.BindException: Address already in use
  163. at sun.nio.ch.Net.bind0(Native Method)
  164. at sun.nio.ch.Net.bind(Net.java:463)
  165. at sun.nio.ch.Net.bind(Net.java:455)
  166. at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
  167. at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  168. at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
  169. at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
  170. at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
  171. at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
  172. at org.spark-project.jetty.server.Server.doStart(Server.java:293)
  173. at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
  174. at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:237)
  175. at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
  176. at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
  177. at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
  178. at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
  179. at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
  180. at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
  181. at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
  182. at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
  183. at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
  184. at scala.Option.foreach(Option.scala:236)
  185. at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
  186. at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
  187. at $line3.$read$$iwC$$iwC.<init>(<console>:9)
  188. at $line3.$read$$iwC.<init>(<console>:18)
  189. at $line3.$read.<init>(<console>:20)
  190. at $line3.$read$.<init>(<console>:24)
  191. at $line3.$read$.<clinit>(<console>)
  192. at $line3.$eval$.<init>(<console>:7)
  193. at $line3.$eval$.<clinit>(<console>)
  194. at $line3.$eval.$print(<console>)
  195. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  196. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  197. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  198. at java.lang.reflect.Method.invoke(Method.java:606)
  199. at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
  200. at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
  201. at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
  202. at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
  203. at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
  204. at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
  205. at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
  206. at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
  207. at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
  208. at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
  209. at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
  210. at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
  211. at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
  212. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
  213. at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
  214. at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
  215. at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
  216. at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
  217. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
  218. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
  219. at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
  220. at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
  221. at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
  222. at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
  223. at org.apache.spark.repl.Main$.main(Main.scala:31)
  224. at org.apache.spark.repl.Main.main(Main.scala)
  225. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  226. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  227. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  228. at java.lang.reflect.Method.invoke(Method.java:606)
  229. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
  230. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  231. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  232. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
  233. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  234. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
  235. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
  236. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
  237. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
  238. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
  239. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
  240. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
  241. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
  242. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
  243. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
  244. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
  245. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
  246. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
  247. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
  248. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
  249. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
  250. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
  251. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
  252. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
  253. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
  254. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
  255. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
  256. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
  257. 16/02/14 20:19:58 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
  258. 16/02/14 20:19:58 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
  259. 16/02/14 20:19:58 INFO Server: jetty-8.y.z-SNAPSHOT
  260. 16/02/14 20:19:58 INFO AbstractConnector: Started [email protected]:4041
  261. 16/02/14 20:19:58 INFO Utils: Successfully started service 'SparkUI' on port 4041.
  262. 16/02/14 20:19:58 INFO SparkUI: Started SparkUI at http://10.0.0.13:4041
  263. 16/02/14 20:19:58 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
  264. 16/02/14 20:20:00 INFO TimelineClientImpl: Timeline service address: http://hn0-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net:8188/ws/v1/timeline/
  265. 16/02/14 20:20:01 INFO AHSProxy: Connecting to Application History server at hn0-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net/10.0.0.13:10200
  266. 16/02/14 20:20:01 INFO ConfiguredRMFailoverProxyProvider: Failing over to rm2
  267. 16/02/14 20:20:02 INFO Client: Requesting a new application from cluster with 30 NodeManagers
  268. 16/02/14 20:20:02 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
  269. 16/02/14 20:20:02 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
  270. 16/02/14 20:20:02 INFO Client: Setting up container launch context for our AM
  271. 16/02/14 20:20:02 INFO Client: Setting up the launch environment for our AM container
  272. 16/02/14 20:20:02 INFO Client: Preparing resources for our AM container
  273. 16/02/14 20:20:03 INFO MetricsConfig: loaded properties from hadoop-metrics2-azure-file-system.properties
  274. 16/02/14 20:20:03 INFO WasbAzureIaasSink: Init starting.
  275. 16/02/14 20:20:03 INFO AzureIaasSink: Init starting. Initializing MdsLogger.
  276. 16/02/14 20:20:03 INFO AzureIaasSink: Init completed.
  277. 16/02/14 20:20:03 INFO WasbAzureIaasSink: Init completed.
  278. 16/02/14 20:20:03 INFO MetricsSinkAdapter: Sink azurefs2 started
  279. 16/02/14 20:20:03 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
  280. 16/02/14 20:20:03 INFO MetricsSystemImpl: azure-file-system metrics system started
  281. 16/02/14 20:20:04 INFO Client: Uploading resource file:/var/tmp/spark/spark-fb3a3311-7aaf-43d4-a9db-6dfb51f97422/__spark_conf__4643401418066494768.zip -> wasb://[email protected]/user/abshkmodi-ssh/.sparkStaging/application_1455282642107_0043/__spark_conf__4643401418066494768.zip
  282. 16/02/14 20:20:05 INFO SecurityManager: Changing view acls to: abshkmodi-ssh
  283. 16/02/14 20:20:05 INFO SecurityManager: Changing modify acls to: abshkmodi-ssh
  284. 16/02/14 20:20:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(abshkmodi-ssh); users with modify permissions: Set(abshkmodi-ssh)
  285. 16/02/14 20:20:05 INFO Client: Submitting application 43 to ResourceManager
  286. 16/02/14 20:20:05 INFO YarnClientImpl: Submitted application application_1455282642107_0043
  287. 16/02/14 20:20:05 INFO YarnExtensionServices: Starting Yarn extension services with app application_1455282642107_0043 and attemptId None
  288. 16/02/14 20:20:06 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  289. 16/02/14 20:20:06 INFO Client:
  290. client token: N/A
  291. diagnostics: N/A
  292. ApplicationMaster host: N/A
  293. ApplicationMaster RPC port: -1
  294. queue: default
  295. start time: 1455481205403
  296. final status: UNDEFINED
  297. tracking URL: http://hn1-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net:8088/proxy/application_1455282642107_0043/
  298. user: abshkmodi-ssh
  299. 16/02/14 20:20:07 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  300. 16/02/14 20:20:08 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  301. 16/02/14 20:20:09 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  302. 16/02/14 20:20:10 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  303. 16/02/14 20:20:11 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  304. 16/02/14 20:20:12 INFO Client: Application report for application_1455282642107_0043 (state: ACCEPTED)
  305. 16/02/14 20:20:12 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:38031/user/YarnAM#-1017099691])
  306. 16/02/14 20:20:12 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hn0-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net,hn1-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net, PROXY_URI_BASES -> http://hn0-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net:8088/proxy/application_1455282642107_0043,http://hn1-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net:8088/proxy/application_1455282642107_0043), /proxy/application_1455282642107_0043
  307. 16/02/14 20:20:12 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  308. 16/02/14 20:20:13 INFO Client: Application report for application_1455282642107_0043 (state: RUNNING)
  309. 16/02/14 20:20:13 INFO Client:
  310. client token: N/A
  311. diagnostics: N/A
  312. ApplicationMaster host: 10.0.0.35
  313. ApplicationMaster RPC port: 0
  314. queue: default
  315. start time: 1455481205403
  316. final status: UNDEFINED
  317. 16/02/14 20:20:13 INFO YarnClientSchedulerBackend: Application application_1455282642107_0043 has started running.
  318. 16/02/14 20:20:13 INFO YarnScheduler: Starting speculative execution thread
  319. 16/02/14 20:20:14 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33073.
  320. 16/02/14 20:20:14 INFO NettyBlockTransferService: Server created on 33073
  321. 16/02/14 20:20:14 INFO BlockManagerMaster: Trying to register BlockManager
  322. 16/02/14 20:20:14 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.13:33073 with 530.3 MB RAM, BlockManagerId(driver, 10.0.0.13, 33073)
  323. 16/02/14 20:20:14 INFO BlockManagerMaster: Registered BlockManager
  324. 16/02/14 20:20:15 INFO EventLoggingListener: Logging events to wasb:///hdp/spark-events/application_1455282642107_0043
  325. 16/02/14 20:20:20 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:53238/user/Executor#853825945]) with ID 3
  326. 16/02/14 20:20:20 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58269/user/Executor#-773316969]) with ID 2
  327. 16/02/14 20:20:20 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:43504/user/Executor#-1047437222]) with ID 5
  328. 16/02/14 20:20:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.19:51776 with 530.3 MB RAM, BlockManagerId(3, 10.0.0.19, 51776)
  329. 16/02/14 20:20:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.35:57975 with 530.3 MB RAM, BlockManagerId(2, 10.0.0.35, 57975)
  330. 16/02/14 20:20:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.30:34041 with 530.3 MB RAM, BlockManagerId(5, 10.0.0.30, 34041)
  331. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:32788/user/Executor#634570555]) with ID 9
  332. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:38798/user/Executor#564077469]) with ID 12
  333. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:41686/user/Executor#-59736481]) with ID 11
  334. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:45180/user/Executor#-1381121524]) with ID 16
  335. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:47720/user/Executor#-68849180]) with ID 8
  336. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:57484/user/Executor#-1600025409]) with ID 15
  337. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:41720/user/Executor#485687162]) with ID 19
  338. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:32882/user/Executor#121063705]) with ID 14
  339. 16/02/14 20:20:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.27:60829 with 530.3 MB RAM, BlockManagerId(9, 10.0.0.27, 60829)
  340. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:37330/user/Executor#-45055840]) with ID 7
  341. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:55617/user/Executor#-240363421]) with ID 17
  342. 16/02/14 20:20:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.22:49473 with 530.3 MB RAM, BlockManagerId(12, 10.0.0.22, 49473)
  343. 16/02/14 20:20:21 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:41144/user/Executor#-687884251]) with ID 10
  344. 16/02/14 20:20:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.34:48478 with 530.3 MB RAM, BlockManagerId(11, 10.0.0.34, 48478)
  345. 16/02/14 20:20:21 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.38:55189 with 530.3 MB RAM, BlockManagerId(16, 10.0.0.38, 55189)
  346. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.10:39402 with 530.3 MB RAM, BlockManagerId(8, 10.0.0.10, 39402)
  347. 16/02/14 20:20:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:41273/user/Executor#1411534210]) with ID 18
  348. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.11:41508 with 530.3 MB RAM, BlockManagerId(15, 10.0.0.11, 41508)
  349. 16/02/14 20:20:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:41749/user/Executor#1914515299]) with ID 1
  350. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.4:41704 with 530.3 MB RAM, BlockManagerId(19, 10.0.0.4, 41704)
  351. 16/02/14 20:20:22 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
  352. 16/02/14 20:20:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:46615/user/Executor#-1241670627]) with ID 4
  353. 16/02/14 20:20:22 INFO SparkILoop: Created spark context..
  354. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.39:47472 with 530.3 MB RAM, BlockManagerId(14, 10.0.0.39, 47472)
  355. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.18:56267 with 530.3 MB RAM, BlockManagerId(7, 10.0.0.18, 56267)
  356. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.7:41881 with 530.3 MB RAM, BlockManagerId(17, 10.0.0.7, 41881)
  357. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.37:59838 with 530.3 MB RAM, BlockManagerId(10, 10.0.0.37, 59838)
  358. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.26:56362 with 530.3 MB RAM, BlockManagerId(18, 10.0.0.26, 56362)
  359. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.21:38437 with 530.3 MB RAM, BlockManagerId(1, 10.0.0.21, 38437)
  360. 16/02/14 20:20:22 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.25:40063 with 530.3 MB RAM, BlockManagerId(4, 10.0.0.25, 40063)
  361. 16/02/14 20:20:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:37294/user/Executor#294332242]) with ID 13
  362. 16/02/14 20:20:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:55698/user/Executor#55032407]) with ID 6
  363. 16/02/14 20:20:23 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.28:46101 with 530.3 MB RAM, BlockManagerId(13, 10.0.0.28, 46101)
  364. 16/02/14 20:20:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:53040/user/Executor#-1448246698]) with ID 20
  365. 16/02/14 20:20:23 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.20:51761 with 530.3 MB RAM, BlockManagerId(6, 10.0.0.20, 51761)
  366. 16/02/14 20:20:24 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.32:35226 with 530.3 MB RAM, BlockManagerId(20, 10.0.0.32, 35226)
  367. 16/02/14 20:20:24 INFO HiveContext: Initializing execution hive, version 1.2.1
  368. 16/02/14 20:20:24 INFO ClientWrapper: Inspected Hadoop version: 2.7.1.2.3.3.1-4
  369. 16/02/14 20:20:24 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.7.1.2.3.3.1-4
  370. 16/02/14 20:20:26 INFO metastore: Trying to connect to metastore with URI thrift://hn0-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net:9083
  371. 16/02/14 20:20:26 INFO metastore: Connected to metastore.
  372. 16/02/14 20:20:26 INFO SessionState: Created local directory: /var/tmp/spark/330fbd35-0566-49cf-956f-21e134d1f33a_resources
  373. 16/02/14 20:20:26 INFO SessionState: Created HDFS directory: /tmp/hive/abshkmodi-ssh/330fbd35-0566-49cf-956f-21e134d1f33a
  374. 16/02/14 20:20:26 INFO SessionState: Created local directory: /var/tmp/spark/abshkmodi-ssh/330fbd35-0566-49cf-956f-21e134d1f33a
  375. 16/02/14 20:20:27 INFO SessionState: Created HDFS directory: /tmp/hive/abshkmodi-ssh/330fbd35-0566-49cf-956f-21e134d1f33a/_tmp_space.db
  376. 16/02/14 20:20:27 INFO HiveContext: default warehouse location is /user/hive/warehouse
  377. 16/02/14 20:20:27 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
  378. 16/02/14 20:20:27 INFO ClientWrapper: Inspected Hadoop version: 2.7.1.2.3.3.1-4
  379. 16/02/14 20:20:27 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.7.1.2.3.3.1-4
  380. 16/02/14 20:20:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  381. 16/02/14 20:20:29 INFO metastore: Trying to connect to metastore with URI thrift://hn0-abshkm.auusdajhslxujb22jxnw3kpa5c.jx.internal.cloudapp.net:9083
  382. 16/02/14 20:20:29 INFO metastore: Connected to metastore.
  383. 16/02/14 20:20:30 INFO MetricsConfig: loaded properties from hadoop-metrics2-azure-file-system.properties
  384. 16/02/14 20:20:30 INFO WasbAzureIaasSink: Init starting.
  385. 16/02/14 20:20:30 INFO AzureIaasSink: Init starting. Initializing MdsLogger.
  386. 16/02/14 20:20:30 INFO AzureIaasSink: Init completed.
  387. 16/02/14 20:20:30 INFO WasbAzureIaasSink: Init completed.
  388. 16/02/14 20:20:30 INFO MetricsSinkAdapter: Sink azurefs2 started
  389. 16/02/14 20:20:30 WARN MBeans: Failed to register MBean "Hadoop:service=azure-file-system,name=MetricsSystem,sub=Stats": Instance already exists.
  390. 16/02/14 20:20:30 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
  391. 16/02/14 20:20:30 INFO MetricsSystemImpl: azure-file-system metrics system started
  392. 16/02/14 20:20:30 WARN MBeans: Failed to register MBean "Hadoop:service=azure-file-system,name=MetricsSystem,sub=Control": Instance already exists.
  393. 16/02/14 20:20:30 WARN MBeans: Failed to register MBean "Hadoop:service=azure-file-system,name=AzureFileSystemMetrics": Instance already exists.
  394. 16/02/14 20:20:30 INFO SessionState: Created local directory: /var/tmp/spark/0741d3a9-ab1d-4be5-a7b3-347dd90c2e13_resources
  395. 16/02/14 20:20:30 INFO SessionState: Created HDFS directory: /tmp/hive/abshkmodi-ssh/0741d3a9-ab1d-4be5-a7b3-347dd90c2e13
  396. 16/02/14 20:20:30 INFO SessionState: Created local directory: /var/tmp/spark/abshkmodi-ssh/0741d3a9-ab1d-4be5-a7b3-347dd90c2e13
  397. 16/02/14 20:20:30 INFO SessionState: Created HDFS directory: /tmp/hive/abshkmodi-ssh/0741d3a9-ab1d-4be5-a7b3-347dd90c2e13/_tmp_space.db
  398. 16/02/14 20:20:31 INFO SparkILoop: Created sql context (with Hive support)..
  399. 16/02/14 20:20:47 INFO MemoryStore: ensureFreeSpace(234336) called with curMem=0, maxMem=556038881
  400. 16/02/14 20:20:47 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 228.8 KB, free 530.1 MB)
  401. 16/02/14 20:20:47 INFO MemoryStore: ensureFreeSpace(28592) called with curMem=234336, maxMem=556038881
  402. 16/02/14 20:20:47 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 27.9 KB, free 530.0 MB)
  403. 16/02/14 20:20:47 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.13:33073 (size: 27.9 KB, free: 530.3 MB)
  404. 16/02/14 20:20:47 INFO SparkContext: Created broadcast 0 from textFile at <console>:27
  405. 16/02/14 20:20:51 INFO FileInputFormat: Total input paths to process : 20
  406. 16/02/14 20:20:51 INFO SparkContext: Starting job: reduce at <console>:31
  407. 16/02/14 20:20:51 INFO DAGScheduler: Got job 0 (reduce at <console>:31) with 60 output partitions
  408. 16/02/14 20:20:51 INFO DAGScheduler: Final stage: ResultStage 0(reduce at <console>:31)
  409. 16/02/14 20:20:51 INFO DAGScheduler: Parents of final stage: List()
  410. 16/02/14 20:20:51 INFO DAGScheduler: Missing parents: List()
  411. 16/02/14 20:20:51 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at <console>:31), which has no missing parents
  412. 16/02/14 20:20:51 INFO MemoryStore: ensureFreeSpace(8120) called with curMem=262928, maxMem=556038881
  413. 16/02/14 20:20:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.9 KB, free 530.0 MB)
  414. 16/02/14 20:20:51 INFO MemoryStore: ensureFreeSpace(3433) called with curMem=271048, maxMem=556038881
  415. 16/02/14 20:20:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.4 KB, free 530.0 MB)
  416. 16/02/14 20:20:51 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.13:33073 (size: 3.4 KB, free: 530.2 MB)
  417. 16/02/14 20:20:51 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
  418. 16/02/14 20:20:51 INFO DAGScheduler: Submitting 60 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at <console>:31)
  419. 16/02/14 20:20:51 INFO YarnScheduler: Adding task set 0.0 with 60 tasks
  420. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 10.0.0.20, PROCESS_LOCAL, 2208 bytes)
  421. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 10.0.0.25, PROCESS_LOCAL, 2208 bytes)
  422. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  423. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 10.0.0.10, PROCESS_LOCAL, 2208 bytes)
  424. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, 10.0.0.34, PROCESS_LOCAL, 2208 bytes)
  425. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, 10.0.0.21, PROCESS_LOCAL, 2208 bytes)
  426. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, 10.0.0.27, PROCESS_LOCAL, 2208 bytes)
  427. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, 10.0.0.19, PROCESS_LOCAL, 2208 bytes)
  428. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  429. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, 10.0.0.35, PROCESS_LOCAL, 2208 bytes)
  430. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, 10.0.0.18, PROCESS_LOCAL, 2208 bytes)
  431. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  432. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, 10.0.0.7, PROCESS_LOCAL, 2208 bytes)
  433. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, 10.0.0.11, PROCESS_LOCAL, 2208 bytes)
  434. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, 10.0.0.39, PROCESS_LOCAL, 2208 bytes)
  435. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  436. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, 10.0.0.22, PROCESS_LOCAL, 2208 bytes)
  437. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, 10.0.0.28, PROCESS_LOCAL, 2208 bytes)
  438. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, 10.0.0.32, PROCESS_LOCAL, 2208 bytes)
  439. 16/02/14 20:20:51 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  440. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.27:60829 (size: 3.4 KB, free: 530.3 MB)
  441. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.4:41704 (size: 3.4 KB, free: 530.3 MB)
  442. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.11:41508 (size: 3.4 KB, free: 530.3 MB)
  443. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.10:39402 (size: 3.4 KB, free: 530.3 MB)
  444. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.30:34041 (size: 3.4 KB, free: 530.3 MB)
  445. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.18:56267 (size: 3.4 KB, free: 530.3 MB)
  446. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.22:49473 (size: 3.4 KB, free: 530.3 MB)
  447. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.19:51776 (size: 3.4 KB, free: 530.3 MB)
  448. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.38:55189 (size: 3.4 KB, free: 530.3 MB)
  449. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.39:47472 (size: 3.4 KB, free: 530.3 MB)
  450. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.34:48478 (size: 3.4 KB, free: 530.3 MB)
  451. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.7:41881 (size: 3.4 KB, free: 530.3 MB)
  452. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.37:59838 (size: 3.4 KB, free: 530.3 MB)
  453. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.26:56362 (size: 3.4 KB, free: 530.3 MB)
  454. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.35:57975 (size: 3.4 KB, free: 530.3 MB)
  455. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.21:38437 (size: 3.4 KB, free: 530.3 MB)
  456. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.25:40063 (size: 3.4 KB, free: 530.3 MB)
  457. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.20:51761 (size: 3.4 KB, free: 530.3 MB)
  458. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.32:35226 (size: 3.4 KB, free: 530.3 MB)
  459. 16/02/14 20:20:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.0.28:46101 (size: 3.4 KB, free: 530.3 MB)
  460. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.7:41881 (size: 27.9 KB, free: 530.2 MB)
  461. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.27:60829 (size: 27.9 KB, free: 530.2 MB)
  462. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.19:51776 (size: 27.9 KB, free: 530.2 MB)
  463. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.4:41704 (size: 27.9 KB, free: 530.2 MB)
  464. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.37:59838 (size: 27.9 KB, free: 530.2 MB)
  465. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.30:34041 (size: 27.9 KB, free: 530.2 MB)
  466. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.18:56267 (size: 27.9 KB, free: 530.2 MB)
  467. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.22:49473 (size: 27.9 KB, free: 530.2 MB)
  468. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.35:57975 (size: 27.9 KB, free: 530.2 MB)
  469. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.38:55189 (size: 27.9 KB, free: 530.2 MB)
  470. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.11:41508 (size: 27.9 KB, free: 530.2 MB)
  471. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.25:40063 (size: 27.9 KB, free: 530.2 MB)
  472. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.10:39402 (size: 27.9 KB, free: 530.2 MB)
  473. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.34:48478 (size: 27.9 KB, free: 530.2 MB)
  474. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.21:38437 (size: 27.9 KB, free: 530.2 MB)
  475. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.39:47472 (size: 27.9 KB, free: 530.2 MB)
  476. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.26:56362 (size: 27.9 KB, free: 530.2 MB)
  477. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.20:51761 (size: 27.9 KB, free: 530.2 MB)
  478. 16/02/14 20:20:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.32:35226 (size: 27.9 KB, free: 530.2 MB)
  479. 16/02/14 20:21:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.0.28:46101 (size: 27.9 KB, free: 530.2 MB)
  480. 16/02/14 20:22:26 INFO BlockManagerInfo: Added rdd_2_11 in memory on 10.0.0.26:56362 (size: 19.0 MB, free: 511.2 MB)
  481. 16/02/14 20:22:26 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  482. 16/02/14 20:22:26 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 94793 ms on 10.0.0.26 (1/60)
  483. 16/02/14 20:22:30 INFO BlockManagerInfo: Added rdd_2_5 in memory on 10.0.0.21:38437 (size: 18.8 MB, free: 511.5 MB)
  484. 16/02/14 20:22:31 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, 10.0.0.21, PROCESS_LOCAL, 2208 bytes)
  485. 16/02/14 20:22:31 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 99346 ms on 10.0.0.21 (2/60)
  486. 16/02/14 20:22:31 INFO BlockManagerInfo: Added rdd_2_14 in memory on 10.0.0.39:47472 (size: 19.1 MB, free: 511.1 MB)
  487. 16/02/14 20:22:31 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, 10.0.0.39, PROCESS_LOCAL, 2208 bytes)
  488. 16/02/14 20:22:31 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 99976 ms on 10.0.0.39 (3/60)
  489. 16/02/14 20:22:33 INFO BlockManagerInfo: Added rdd_2_2 in memory on 10.0.0.4:41704 (size: 18.7 MB, free: 511.6 MB)
  490. 16/02/14 20:22:33 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  491. 16/02/14 20:22:33 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 102024 ms on 10.0.0.4 (4/60)
  492. 16/02/14 20:22:37 INFO BlockManagerInfo: Added rdd_2_8 in memory on 10.0.0.30:34041 (size: 18.9 MB, free: 511.4 MB)
  493. 16/02/14 20:22:37 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  494. 16/02/14 20:22:37 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 105521 ms on 10.0.0.30 (5/60)
  495. 16/02/14 20:22:42 INFO BlockManagerInfo: Added rdd_2_17 in memory on 10.0.0.28:46101 (size: 19.1 MB, free: 511.2 MB)
  496. 16/02/14 20:22:42 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, 10.0.0.28, PROCESS_LOCAL, 2208 bytes)
  497. 16/02/14 20:22:42 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 110881 ms on 10.0.0.28 (6/60)
  498. 16/02/14 20:23:42 INFO BlockManagerInfo: Added rdd_2_20 in memory on 10.0.0.26:56362 (size: 19.1 MB, free: 492.1 MB)
  499. 16/02/14 20:23:42 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  500. 16/02/14 20:23:42 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 75743 ms on 10.0.0.26 (7/60)
  501. 16/02/14 20:23:52 INFO BlockManagerInfo: Added rdd_2_23 in memory on 10.0.0.4:41704 (size: 19.1 MB, free: 492.5 MB)
  502. 16/02/14 20:23:52 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  503. 16/02/14 20:23:52 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 78934 ms on 10.0.0.4 (8/60)
  504. 16/02/14 20:25:08 INFO BlockManagerInfo: Added rdd_2_26 in memory on 10.0.0.26:56362 (size: 19.2 MB, free: 472.9 MB)
  505. 16/02/14 20:25:08 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  506. 16/02/14 20:25:08 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 86449 ms on 10.0.0.26 (9/60)
  507. 16/02/14 20:25:09 INFO BlockManagerInfo: Added rdd_2_9 in memory on 10.0.0.35:57975 (size: 59.9 MB, free: 470.4 MB)
  508. 16/02/14 20:25:09 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, 10.0.0.35, PROCESS_LOCAL, 2208 bytes)
  509. 16/02/14 20:25:09 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 258009 ms on 10.0.0.35 (10/60)
  510. 16/02/14 20:25:15 INFO BlockManagerInfo: Added rdd_2_3 in memory on 10.0.0.10:39402 (size: 60.1 MB, free: 470.2 MB)
  511. 16/02/14 20:25:15 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, 10.0.0.10, PROCESS_LOCAL, 2208 bytes)
  512. 16/02/14 20:25:15 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 264249 ms on 10.0.0.10 (11/60)
  513. 16/02/14 20:25:16 INFO BlockManagerInfo: Added rdd_2_12 in memory on 10.0.0.7:41881 (size: 59.9 MB, free: 470.3 MB)
  514. 16/02/14 20:25:16 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, 10.0.0.7, PROCESS_LOCAL, 2208 bytes)
  515. 16/02/14 20:25:16 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 264453 ms on 10.0.0.7 (12/60)
  516. 16/02/14 20:25:16 INFO BlockManagerInfo: Added rdd_2_15 in memory on 10.0.0.38:55189 (size: 59.9 MB, free: 470.4 MB)
  517. 16/02/14 20:25:16 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  518. 16/02/14 20:25:16 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 264729 ms on 10.0.0.38 (13/60)
  519. 16/02/14 20:25:19 INFO BlockManagerInfo: Added rdd_2_7 in memory on 10.0.0.19:51776 (size: 60.0 MB, free: 470.3 MB)
  520. 16/02/14 20:25:19 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, 10.0.0.19, PROCESS_LOCAL, 2208 bytes)
  521. 16/02/14 20:25:19 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 267847 ms on 10.0.0.19 (14/60)
  522. 16/02/14 20:25:21 INFO BlockManagerInfo: Added rdd_2_6 in memory on 10.0.0.27:60829 (size: 60.1 MB, free: 470.2 MB)
  523. 16/02/14 20:25:21 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, 10.0.0.27, PROCESS_LOCAL, 2208 bytes)
  524. 16/02/14 20:25:21 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 269406 ms on 10.0.0.27 (15/60)
  525. 16/02/14 20:25:25 INFO BlockManagerInfo: Added rdd_2_4 in memory on 10.0.0.34:48478 (size: 60.2 MB, free: 470.1 MB)
  526. 16/02/14 20:25:25 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, 10.0.0.34, PROCESS_LOCAL, 2208 bytes)
  527. 16/02/14 20:25:25 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 274229 ms on 10.0.0.34 (16/60)
  528. 16/02/14 20:25:28 INFO BlockManagerInfo: Added rdd_2_18 in memory on 10.0.0.32:35226 (size: 60.0 MB, free: 470.3 MB)
  529. 16/02/14 20:25:28 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, 10.0.0.32, PROCESS_LOCAL, 2208 bytes)
  530. 16/02/14 20:25:28 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 276467 ms on 10.0.0.32 (17/60)
  531. 16/02/14 20:25:29 INFO BlockManagerInfo: Added rdd_2_16 in memory on 10.0.0.22:49473 (size: 59.9 MB, free: 470.3 MB)
  532. 16/02/14 20:25:29 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, 10.0.0.22, PROCESS_LOCAL, 2208 bytes)
  533. 16/02/14 20:25:29 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 277670 ms on 10.0.0.22 (18/60)
  534. 16/02/14 20:25:30 INFO BlockManagerInfo: Added rdd_2_13 in memory on 10.0.0.11:41508 (size: 59.9 MB, free: 470.3 MB)
  535. 16/02/14 20:25:30 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, 10.0.0.11, PROCESS_LOCAL, 2208 bytes)
  536. 16/02/14 20:25:30 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 278384 ms on 10.0.0.11 (19/60)
  537. 16/02/14 20:25:30 INFO BlockManagerInfo: Added rdd_2_1 in memory on 10.0.0.25:40063 (size: 60.1 MB, free: 470.2 MB)
  538. 16/02/14 20:25:30 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, 10.0.0.25, PROCESS_LOCAL, 2208 bytes)
  539. 16/02/14 20:25:30 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 278599 ms on 10.0.0.25 (20/60)
  540. 16/02/14 20:25:31 INFO BlockManagerInfo: Added rdd_2_0 in memory on 10.0.0.20:51761 (size: 60.2 MB, free: 470.0 MB)
  541. 16/02/14 20:25:31 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, 10.0.0.20, PROCESS_LOCAL, 2208 bytes)
  542. 16/02/14 20:25:31 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 279709 ms on 10.0.0.20 (21/60)
  543. 16/02/14 20:25:33 INFO BlockManagerInfo: Added rdd_2_19 in memory on 10.0.0.37:59838 (size: 59.9 MB, free: 470.4 MB)
  544. 16/02/14 20:25:33 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  545. 16/02/14 20:25:33 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 281934 ms on 10.0.0.37 (22/60)
  546. 16/02/14 20:25:38 INFO BlockManagerInfo: Added rdd_2_10 in memory on 10.0.0.18:56267 (size: 60.0 MB, free: 470.3 MB)
  547. 16/02/14 20:25:38 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, 10.0.0.18, PROCESS_LOCAL, 2208 bytes)
  548. 16/02/14 20:25:38 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 287126 ms on 10.0.0.18 (23/60)
  549. 16/02/14 20:26:30 INFO BlockManagerInfo: Added rdd_2_29 in memory on 10.0.0.35:57975 (size: 19.0 MB, free: 451.3 MB)
  550. 16/02/14 20:26:30 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, 10.0.0.35, PROCESS_LOCAL, 2208 bytes)
  551. 16/02/14 20:26:30 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 80458 ms on 10.0.0.35 (24/60)
  552. 16/02/14 20:26:43 INFO BlockManagerInfo: Added rdd_2_32 in memory on 10.0.0.38:55189 (size: 19.1 MB, free: 451.3 MB)
  553. 16/02/14 20:26:44 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  554. 16/02/14 20:26:44 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 87558 ms on 10.0.0.38 (25/60)
  555. 16/02/14 20:26:49 INFO BlockManagerInfo: Added rdd_2_22 in memory on 10.0.0.39:47472 (size: 59.9 MB, free: 451.2 MB)
  556. 16/02/14 20:26:49 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, 10.0.0.39, PROCESS_LOCAL, 2208 bytes)
  557. 16/02/14 20:26:49 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 257684 ms on 10.0.0.39 (26/60)
  558. 16/02/14 20:26:49 INFO BlockManagerInfo: Added rdd_2_35 in memory on 10.0.0.34:48478 (size: 19.0 MB, free: 451.0 MB)
  559. 16/02/14 20:26:50 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, 10.0.0.34, PROCESS_LOCAL, 2208 bytes)
  560. 16/02/14 20:26:50 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 84103 ms on 10.0.0.34 (27/60)
  561. 16/02/14 20:26:59 INFO BlockManagerInfo: Added rdd_2_41 in memory on 10.0.0.37:59838 (size: 19.1 MB, free: 451.3 MB)
  562. 16/02/14 20:26:59 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  563. 16/02/14 20:26:59 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 85533 ms on 10.0.0.37 (28/60)
  564. 16/02/14 20:27:02 INFO BlockManagerInfo: Added rdd_2_21 in memory on 10.0.0.21:38437 (size: 59.9 MB, free: 451.6 MB)
  565. 16/02/14 20:27:02 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, 10.0.0.21, PROCESS_LOCAL, 2208 bytes)
  566. 16/02/14 20:27:02 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 271625 ms on 10.0.0.21 (29/60)
  567. 16/02/14 20:27:05 INFO BlockManagerInfo: Added rdd_2_38 in memory on 10.0.0.11:41508 (size: 19.2 MB, free: 451.1 MB)
  568. 16/02/14 20:27:05 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, 10.0.0.11, PROCESS_LOCAL, 2208 bytes)
  569. 16/02/14 20:27:05 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 95142 ms on 10.0.0.11 (30/60)
  570. 16/02/14 20:27:12 INFO BlockManagerInfo: Added rdd_2_24 in memory on 10.0.0.30:34041 (size: 60.0 MB, free: 451.4 MB)
  571. 16/02/14 20:27:12 INFO TaskSetManager: Starting task 50.0 in stage 0.0 (TID 50, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  572. 16/02/14 20:27:12 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 275126 ms on 10.0.0.30 (31/60)
  573. 16/02/14 20:27:30 INFO BlockManagerInfo: Added rdd_2_25 in memory on 10.0.0.28:46101 (size: 59.9 MB, free: 451.3 MB)
  574. 16/02/14 20:27:30 INFO TaskSetManager: Starting task 51.0 in stage 0.0 (TID 51, 10.0.0.28, PROCESS_LOCAL, 2208 bytes)
  575. 16/02/14 20:27:30 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 287633 ms on 10.0.0.28 (32/60)
  576. 16/02/14 20:28:32 INFO BlockManagerInfo: Added rdd_2_44 in memory on 10.0.0.38:55189 (size: 19.2 MB, free: 432.1 MB)
  577. 16/02/14 20:28:32 INFO TaskSetManager: Starting task 52.0 in stage 0.0 (TID 52, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  578. 16/02/14 20:28:32 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 108913 ms on 10.0.0.38 (33/60)
  579. 16/02/14 20:28:48 INFO BlockManagerInfo: Added rdd_2_47 in memory on 10.0.0.37:59838 (size: 19.1 MB, free: 432.2 MB)
  580. 16/02/14 20:28:48 INFO TaskSetManager: Starting task 53.0 in stage 0.0 (TID 53, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  581. 16/02/14 20:28:48 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 109334 ms on 10.0.0.37 (34/60)
  582. 16/02/14 20:28:55 INFO BlockManagerInfo: Added rdd_2_27 in memory on 10.0.0.4:41704 (size: 59.9 MB, free: 432.6 MB)
  583. 16/02/14 20:28:55 INFO TaskSetManager: Starting task 54.0 in stage 0.0 (TID 54, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  584. 16/02/14 20:28:55 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 303186 ms on 10.0.0.4 (35/60)
  585. 16/02/14 20:28:56 INFO BlockManagerInfo: Added rdd_2_50 in memory on 10.0.0.30:34041 (size: 19.2 MB, free: 432.3 MB)
  586. 16/02/14 20:28:56 INFO TaskSetManager: Starting task 55.0 in stage 0.0 (TID 55, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  587. 16/02/14 20:28:56 INFO TaskSetManager: Finished task 50.0 in stage 0.0 (TID 50) in 104380 ms on 10.0.0.30 (36/60)
  588. 16/02/14 20:29:59 INFO BlockManagerInfo: Added rdd_2_28 in memory on 10.0.0.26:56362 (size: 59.9 MB, free: 413.0 MB)
  589. 16/02/14 20:29:59 INFO TaskSetManager: Starting task 56.0 in stage 0.0 (TID 56, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  590. 16/02/14 20:29:59 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 291197 ms on 10.0.0.26 (37/60)
  591. 16/02/14 20:30:20 INFO BlockManagerInfo: Added rdd_2_30 in memory on 10.0.0.10:39402 (size: 60.0 MB, free: 410.2 MB)
  592. 16/02/14 20:30:20 INFO TaskSetManager: Starting task 57.0 in stage 0.0 (TID 57, 10.0.0.10, PROCESS_LOCAL, 2208 bytes)
  593. 16/02/14 20:30:20 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 304182 ms on 10.0.0.10 (38/60)
  594. 16/02/14 20:30:21 INFO BlockManagerInfo: Added rdd_2_31 in memory on 10.0.0.7:41881 (size: 60.0 MB, free: 410.4 MB)
  595. 16/02/14 20:30:22 INFO TaskSetManager: Starting task 58.0 in stage 0.0 (TID 58, 10.0.0.7, PROCESS_LOCAL, 2208 bytes)
  596. 16/02/14 20:30:22 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 305895 ms on 10.0.0.7 (39/60)
  597. 16/02/14 20:30:23 INFO BlockManagerInfo: Added rdd_2_53 in memory on 10.0.0.37:59838 (size: 19.1 MB, free: 413.1 MB)
  598. 16/02/14 20:30:23 INFO TaskSetManager: Starting task 59.0 in stage 0.0 (TID 59, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  599. 16/02/14 20:30:23 INFO TaskSetManager: Finished task 53.0 in stage 0.0 (TID 53) in 95271 ms on 10.0.0.37 (40/60)
  600. 16/02/14 20:30:25 INFO BlockManagerInfo: Added rdd_2_36 in memory on 10.0.0.32:35226 (size: 60.0 MB, free: 410.3 MB)
  601. 16/02/14 20:30:25 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 297687 ms on 10.0.0.32 (41/60)
  602. 16/02/14 20:30:26 INFO BlockManagerInfo: Added rdd_2_33 in memory on 10.0.0.19:51776 (size: 59.9 MB, free: 410.4 MB)
  603. 16/02/14 20:30:26 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 306610 ms on 10.0.0.19 (42/60)
  604. 16/02/14 20:30:35 INFO BlockManagerInfo: Added rdd_2_39 in memory on 10.0.0.25:40063 (size: 59.9 MB, free: 410.3 MB)
  605. 16/02/14 20:30:35 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 305469 ms on 10.0.0.25 (43/60)
  606. 16/02/14 20:30:46 INFO BlockManagerInfo: Added rdd_2_34 in memory on 10.0.0.27:60829 (size: 60.0 MB, free: 410.2 MB)
  607. 16/02/14 20:30:46 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 325117 ms on 10.0.0.27 (44/60)
  608. 16/02/14 20:30:52 INFO BlockManagerInfo: Added rdd_2_37 in memory on 10.0.0.22:49473 (size: 59.9 MB, free: 410.4 MB)
  609. 16/02/14 20:30:52 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 323176 ms on 10.0.0.22 (45/60)
  610. 16/02/14 20:31:04 INFO BlockManagerInfo: Added rdd_2_42 in memory on 10.0.0.18:56267 (size: 60.0 MB, free: 410.3 MB)
  611. 16/02/14 20:31:04 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 325296 ms on 10.0.0.18 (46/60)
  612. 16/02/14 20:31:06 INFO BlockManagerInfo: Added rdd_2_40 in memory on 10.0.0.20:51761 (size: 59.9 MB, free: 410.1 MB)
  613. 16/02/14 20:31:06 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 335087 ms on 10.0.0.20 (47/60)
  614. 16/02/14 20:31:23 INFO BlockManagerInfo: Added rdd_2_56 in memory on 10.0.0.26:56362 (size: 19.2 MB, free: 393.8 MB)
  615. 16/02/14 20:31:23 INFO TaskSetManager: Finished task 56.0 in stage 0.0 (TID 56) in 84040 ms on 10.0.0.26 (48/60)
  616. 16/02/14 20:31:34 INFO BlockManagerInfo: Added rdd_2_45 in memory on 10.0.0.39:47472 (size: 59.9 MB, free: 391.3 MB)
  617. 16/02/14 20:31:34 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 285024 ms on 10.0.0.39 (49/60)
  618. 16/02/14 20:31:44 INFO BlockManagerInfo: Added rdd_2_43 in memory on 10.0.0.35:57975 (size: 59.9 MB, free: 391.4 MB)
  619. 16/02/14 20:31:44 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 314012 ms on 10.0.0.35 (50/60)
  620. 16/02/14 20:31:45 INFO BlockManagerInfo: Added rdd_2_59 in memory on 10.0.0.37:59838 (size: 19.1 MB, free: 394.0 MB)
  621. 16/02/14 20:31:45 INFO TaskSetManager: Finished task 59.0 in stage 0.0 (TID 59) in 81308 ms on 10.0.0.37 (51/60)
  622. 16/02/14 20:31:49 INFO BlockManagerInfo: Added rdd_2_46 in memory on 10.0.0.34:48478 (size: 60.0 MB, free: 391.1 MB)
  623. 16/02/14 20:31:49 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 299654 ms on 10.0.0.34 (52/60)
  624. 16/02/14 20:31:52 INFO BlockManagerInfo: Added rdd_2_48 in memory on 10.0.0.21:38437 (size: 60.0 MB, free: 391.6 MB)
  625. 16/02/14 20:31:52 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 289365 ms on 10.0.0.21 (53/60)
  626. 16/02/14 20:32:02 INFO BlockManagerInfo: Added rdd_2_49 in memory on 10.0.0.11:41508 (size: 59.9 MB, free: 391.3 MB)
  627. 16/02/14 20:32:02 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 297249 ms on 10.0.0.11 (54/60)
  628. 16/02/14 20:32:13 INFO BlockManagerInfo: Added rdd_2_51 in memory on 10.0.0.28:46101 (size: 59.9 MB, free: 391.4 MB)
  629. 16/02/14 20:32:13 INFO TaskSetManager: Finished task 51.0 in stage 0.0 (TID 51) in 283157 ms on 10.0.0.28 (55/60)
  630. 16/02/14 20:32:43 INFO BlockManagerInfo: Added rdd_2_52 in memory on 10.0.0.38:55189 (size: 60.0 MB, free: 372.2 MB)
  631. 16/02/14 20:32:43 INFO TaskSetManager: Finished task 52.0 in stage 0.0 (TID 52) in 250142 ms on 10.0.0.38 (56/60)
  632. 16/02/14 20:32:46 INFO BlockManagerInfo: Added rdd_2_55 in memory on 10.0.0.30:34041 (size: 59.9 MB, free: 372.4 MB)
  633. 16/02/14 20:32:46 INFO TaskSetManager: Finished task 55.0 in stage 0.0 (TID 55) in 230205 ms on 10.0.0.30 (57/60)
  634. 16/02/14 20:32:58 INFO BlockManagerInfo: Added rdd_2_54 in memory on 10.0.0.4:41704 (size: 59.9 MB, free: 372.6 MB)
  635. 16/02/14 20:32:58 INFO TaskSetManager: Finished task 54.0 in stage 0.0 (TID 54) in 242323 ms on 10.0.0.4 (58/60)
  636. 16/02/14 20:33:28 INFO BlockManagerInfo: Added rdd_2_57 in memory on 10.0.0.10:39402 (size: 59.9 MB, free: 350.3 MB)
  637. 16/02/14 20:33:28 INFO TaskSetManager: Finished task 57.0 in stage 0.0 (TID 57) in 188517 ms on 10.0.0.10 (59/60)
  638. 16/02/14 20:33:34 INFO BlockManagerInfo: Added rdd_2_58 in memory on 10.0.0.7:41881 (size: 60.0 MB, free: 350.4 MB)
  639. 16/02/14 20:33:34 INFO TaskSetManager: Finished task 58.0 in stage 0.0 (TID 58) in 192703 ms on 10.0.0.7 (60/60)
  640. 16/02/14 20:33:34 INFO DAGScheduler: ResultStage 0 (reduce at <console>:31) finished in 763.219 s
  641. 16/02/14 20:33:34 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
  642. 16/02/14 20:33:35 INFO DAGScheduler: Job 0 finished: reduce at <console>:31, took 763.850678 s
  643. 16/02/14 20:33:37 INFO SparkContext: Starting job: reduce at <console>:31
  644. 16/02/14 20:33:37 INFO DAGScheduler: Got job 1 (reduce at <console>:31) with 60 output partitions
  645. 16/02/14 20:33:37 INFO DAGScheduler: Final stage: ResultStage 1(reduce at <console>:31)
  646. 16/02/14 20:33:37 INFO DAGScheduler: Parents of final stage: List()
  647. 16/02/14 20:33:37 INFO DAGScheduler: Missing parents: List()
  648. 16/02/14 20:33:37 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[4] at mapPartitions at <console>:31), which has no missing parents
  649. 16/02/14 20:33:37 INFO MemoryStore: ensureFreeSpace(8120) called with curMem=274481, maxMem=556038881
  650. 16/02/14 20:33:37 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 7.9 KB, free 530.0 MB)
  651. 16/02/14 20:33:37 INFO MemoryStore: ensureFreeSpace(3439) called with curMem=282601, maxMem=556038881
  652. 16/02/14 20:33:37 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.4 KB, free 530.0 MB)
  653. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.13:33073 (size: 3.4 KB, free: 530.2 MB)
  654. 16/02/14 20:33:37 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:861
  655. 16/02/14 20:33:37 INFO DAGScheduler: Submitting 60 missing tasks from ResultStage 1 (MapPartitionsRDD[4] at mapPartitions at <console>:31)
  656. 16/02/14 20:33:37 INFO YarnScheduler: Adding task set 1.0 with 60 tasks
  657. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 60, 10.0.0.20, PROCESS_LOCAL, 2208 bytes)
  658. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 61, 10.0.0.28, PROCESS_LOCAL, 2208 bytes)
  659. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 62, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  660. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 63, 10.0.0.10, PROCESS_LOCAL, 2208 bytes)
  661. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 64, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  662. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 65, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  663. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 66, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  664. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 67, 10.0.0.39, PROCESS_LOCAL, 2208 bytes)
  665. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 68, 10.0.0.35, PROCESS_LOCAL, 2208 bytes)
  666. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 69, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  667. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 70, 10.0.0.18, PROCESS_LOCAL, 2208 bytes)
  668. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 71, 10.0.0.27, PROCESS_LOCAL, 2208 bytes)
  669. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 72, 10.0.0.7, PROCESS_LOCAL, 2208 bytes)
  670. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 73, 10.0.0.32, PROCESS_LOCAL, 2208 bytes)
  671. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 74, 10.0.0.22, PROCESS_LOCAL, 2208 bytes)
  672. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 75, 10.0.0.19, PROCESS_LOCAL, 2208 bytes)
  673. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 76, 10.0.0.25, PROCESS_LOCAL, 2208 bytes)
  674. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 77, 10.0.0.34, PROCESS_LOCAL, 2208 bytes)
  675. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 78, 10.0.0.11, PROCESS_LOCAL, 2208 bytes)
  676. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 79, 10.0.0.21, PROCESS_LOCAL, 2208 bytes)
  677. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.10:39402 (size: 3.4 KB, free: 350.3 MB)
  678. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.38:55189 (size: 3.4 KB, free: 372.1 MB)
  679. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.18:56267 (size: 3.4 KB, free: 410.3 MB)
  680. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.26:56362 (size: 3.4 KB, free: 393.8 MB)
  681. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.30:34041 (size: 3.4 KB, free: 372.4 MB)
  682. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.21:38437 (size: 3.4 KB, free: 391.6 MB)
  683. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.27:60829 (size: 3.4 KB, free: 410.2 MB)
  684. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.7:41881 (size: 3.4 KB, free: 350.4 MB)
  685. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.32:35226 (size: 3.4 KB, free: 410.3 MB)
  686. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.4:41704 (size: 3.4 KB, free: 372.6 MB)
  687. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.39:47472 (size: 3.4 KB, free: 391.3 MB)
  688. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.28:46101 (size: 3.4 KB, free: 391.4 MB)
  689. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.37:59838 (size: 3.4 KB, free: 394.0 MB)
  690. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.20:51761 (size: 3.4 KB, free: 410.1 MB)
  691. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.11:41508 (size: 3.4 KB, free: 391.3 MB)
  692. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.25:40063 (size: 3.4 KB, free: 410.3 MB)
  693. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.35:57975 (size: 3.4 KB, free: 391.4 MB)
  694. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.22:49473 (size: 3.4 KB, free: 410.4 MB)
  695. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.34:48478 (size: 3.4 KB, free: 391.1 MB)
  696. 16/02/14 20:33:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.0.19:51776 (size: 3.4 KB, free: 410.4 MB)
  697. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 30.0 in stage 1.0 (TID 80, 10.0.0.10, PROCESS_LOCAL, 2208 bytes)
  698. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 63) in 382 ms on 10.0.0.10 (1/60)
  699. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 21.0 in stage 1.0 (TID 81, 10.0.0.21, PROCESS_LOCAL, 2208 bytes)
  700. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 79) in 361 ms on 10.0.0.21 (2/60)
  701. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 65) in 438 ms on 10.0.0.26 (3/60)
  702. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 20.0 in stage 1.0 (TID 82, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  703. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 32.0 in stage 1.0 (TID 83, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  704. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 42.0 in stage 1.0 (TID 84, 10.0.0.18, PROCESS_LOCAL, 2208 bytes)
  705. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 24.0 in stage 1.0 (TID 85, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  706. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 64) in 516 ms on 10.0.0.30 (4/60)
  707. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 69) in 503 ms on 10.0.0.38 (5/60)
  708. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 70) in 503 ms on 10.0.0.18 (6/60)
  709. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 71) in 505 ms on 10.0.0.27 (7/60)
  710. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 34.0 in stage 1.0 (TID 86, 10.0.0.27, PROCESS_LOCAL, 2208 bytes)
  711. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 31.0 in stage 1.0 (TID 87, 10.0.0.7, PROCESS_LOCAL, 2208 bytes)
  712. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 57.0 in stage 1.0 (TID 88, 10.0.0.10, PROCESS_LOCAL, 2208 bytes)
  713. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 23.0 in stage 1.0 (TID 89, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  714. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 22.0 in stage 1.0 (TID 90, 10.0.0.39, PROCESS_LOCAL, 2208 bytes)
  715. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 67) in 583 ms on 10.0.0.39 (8/60)
  716. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 66) in 600 ms on 10.0.0.4 (9/60)
  717. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 72) in 571 ms on 10.0.0.7 (10/60)
  718. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 30.0 in stage 1.0 (TID 80) in 233 ms on 10.0.0.10 (11/60)
  719. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 32.0 in stage 1.0 (TID 83) in 158 ms on 10.0.0.38 (12/60)
  720. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 44.0 in stage 1.0 (TID 91, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  721. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 26.0 in stage 1.0 (TID 92, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  722. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 20.0 in stage 1.0 (TID 82) in 191 ms on 10.0.0.26 (13/60)
  723. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 21.0 in stage 1.0 (TID 81) in 215 ms on 10.0.0.21 (14/60)
  724. 16/02/14 20:33:37 INFO TaskSetManager: Starting task 48.0 in stage 1.0 (TID 93, 10.0.0.21, PROCESS_LOCAL, 2208 bytes)
  725. 16/02/14 20:33:37 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 78) in 592 ms on 10.0.0.11 (15/60)
  726. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 38.0 in stage 1.0 (TID 94, 10.0.0.11, PROCESS_LOCAL, 2208 bytes)
  727. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 74) in 630 ms on 10.0.0.22 (16/60)
  728. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 37.0 in stage 1.0 (TID 95, 10.0.0.22, PROCESS_LOCAL, 2208 bytes)
  729. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 36.0 in stage 1.0 (TID 96, 10.0.0.32, PROCESS_LOCAL, 2208 bytes)
  730. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 73) in 650 ms on 10.0.0.32 (17/60)
  731. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 39.0 in stage 1.0 (TID 97, 10.0.0.25, PROCESS_LOCAL, 2208 bytes)
  732. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 41.0 in stage 1.0 (TID 98, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  733. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 62) in 714 ms on 10.0.0.37 (18/60)
  734. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 76) in 659 ms on 10.0.0.25 (19/60)
  735. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 29.0 in stage 1.0 (TID 99, 10.0.0.35, PROCESS_LOCAL, 2208 bytes)
  736. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 68) in 714 ms on 10.0.0.35 (20/60)
  737. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 42.0 in stage 1.0 (TID 84) in 292 ms on 10.0.0.18 (21/60)
  738. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 25.0 in stage 1.0 (TID 100, 10.0.0.28, PROCESS_LOCAL, 2208 bytes)
  739. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 33.0 in stage 1.0 (TID 101, 10.0.0.19, PROCESS_LOCAL, 2208 bytes)
  740. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 61) in 774 ms on 10.0.0.28 (22/60)
  741. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 75) in 727 ms on 10.0.0.19 (23/60)
  742. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 40.0 in stage 1.0 (TID 102, 10.0.0.20, PROCESS_LOCAL, 2208 bytes)
  743. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 60) in 811 ms on 10.0.0.20 (24/60)
  744. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 23.0 in stage 1.0 (TID 89) in 227 ms on 10.0.0.4 (25/60)
  745. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 27.0 in stage 1.0 (TID 103, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  746. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 50.0 in stage 1.0 (TID 104, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  747. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 24.0 in stage 1.0 (TID 85) in 339 ms on 10.0.0.30 (26/60)
  748. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 34.0 in stage 1.0 (TID 86) in 280 ms on 10.0.0.27 (27/60)
  749. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 52.0 in stage 1.0 (TID 105, 10.0.0.38, PROCESS_LOCAL, 2208 bytes)
  750. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 57.0 in stage 1.0 (TID 88) in 264 ms on 10.0.0.10 (28/60)
  751. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 44.0 in stage 1.0 (TID 91) in 211 ms on 10.0.0.38 (29/60)
  752. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 31.0 in stage 1.0 (TID 87) in 296 ms on 10.0.0.7 (30/60)
  753. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 58.0 in stage 1.0 (TID 106, 10.0.0.7, PROCESS_LOCAL, 2208 bytes)
  754. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 22.0 in stage 1.0 (TID 90) in 286 ms on 10.0.0.39 (31/60)
  755. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 45.0 in stage 1.0 (TID 107, 10.0.0.39, PROCESS_LOCAL, 2208 bytes)
  756. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 49.0 in stage 1.0 (TID 108, 10.0.0.11, PROCESS_LOCAL, 2208 bytes)
  757. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 47.0 in stage 1.0 (TID 109, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  758. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 41.0 in stage 1.0 (TID 98) in 196 ms on 10.0.0.37 (32/60)
  759. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 37.0 in stage 1.0 (TID 95) in 226 ms on 10.0.0.22 (33/60)
  760. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 38.0 in stage 1.0 (TID 94) in 238 ms on 10.0.0.11 (34/60)
  761. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 48.0 in stage 1.0 (TID 93) in 259 ms on 10.0.0.21 (35/60)
  762. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 36.0 in stage 1.0 (TID 96) in 254 ms on 10.0.0.32 (36/60)
  763. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 39.0 in stage 1.0 (TID 97) in 251 ms on 10.0.0.25 (37/60)
  764. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 43.0 in stage 1.0 (TID 110, 10.0.0.35, PROCESS_LOCAL, 2208 bytes)
  765. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 29.0 in stage 1.0 (TID 99) in 234 ms on 10.0.0.35 (38/60)
  766. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 55.0 in stage 1.0 (TID 111, 10.0.0.30, PROCESS_LOCAL, 2208 bytes)
  767. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 51.0 in stage 1.0 (TID 112, 10.0.0.28, PROCESS_LOCAL, 2208 bytes)
  768. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 50.0 in stage 1.0 (TID 104) in 176 ms on 10.0.0.30 (39/60)
  769. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 25.0 in stage 1.0 (TID 100) in 236 ms on 10.0.0.28 (40/60)
  770. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 33.0 in stage 1.0 (TID 101) in 226 ms on 10.0.0.19 (41/60)
  771. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 54.0 in stage 1.0 (TID 113, 10.0.0.4, PROCESS_LOCAL, 2208 bytes)
  772. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 28.0 in stage 1.0 (TID 114, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  773. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 53.0 in stage 1.0 (TID 115, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  774. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 26.0 in stage 1.0 (TID 92) in 456 ms on 10.0.0.26 (42/60)
  775. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 52.0 in stage 1.0 (TID 105) in 263 ms on 10.0.0.38 (43/60)
  776. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 58.0 in stage 1.0 (TID 106) in 236 ms on 10.0.0.7 (44/60)
  777. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 40.0 in stage 1.0 (TID 102) in 304 ms on 10.0.0.20 (45/60)
  778. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 45.0 in stage 1.0 (TID 107) in 227 ms on 10.0.0.39 (46/60)
  779. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 27.0 in stage 1.0 (TID 103) in 298 ms on 10.0.0.4 (47/60)
  780. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 49.0 in stage 1.0 (TID 108) in 221 ms on 10.0.0.11 (48/60)
  781. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 54.0 in stage 1.0 (TID 113) in 129 ms on 10.0.0.4 (49/60)
  782. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 43.0 in stage 1.0 (TID 110) in 184 ms on 10.0.0.35 (50/60)
  783. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 51.0 in stage 1.0 (TID 112) in 147 ms on 10.0.0.28 (51/60)
  784. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 47.0 in stage 1.0 (TID 109) in 241 ms on 10.0.0.37 (52/60)
  785. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 55.0 in stage 1.0 (TID 111) in 158 ms on 10.0.0.30 (53/60)
  786. 16/02/14 20:33:38 INFO TaskSetManager: Marking task 4 in stage 1.0 (on 10.0.0.34) as speculatable because it ran more than 429 ms
  787. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 59.0 in stage 1.0 (TID 116, 10.0.0.37, PROCESS_LOCAL, 2208 bytes)
  788. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 53.0 in stage 1.0 (TID 115) in 107 ms on 10.0.0.37 (54/60)
  789. 16/02/14 20:33:38 INFO TaskSetManager: Starting task 56.0 in stage 1.0 (TID 117, 10.0.0.26, PROCESS_LOCAL, 2208 bytes)
  790. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 59.0 in stage 1.0 (TID 116) in 68 ms on 10.0.0.37 (55/60)
  791. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 28.0 in stage 1.0 (TID 114) in 224 ms on 10.0.0.26 (56/60)
  792. 16/02/14 20:33:38 INFO TaskSetManager: Finished task 56.0 in stage 1.0 (TID 117) in 93 ms on 10.0.0.26 (57/60)
  793. 16/02/14 20:33:44 INFO TaskSetManager: Starting task 35.0 in stage 1.0 (TID 118, 10.0.0.34, PROCESS_LOCAL, 2208 bytes)
  794. 16/02/14 20:33:44 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 77) in 7022 ms on 10.0.0.34 (58/60)
  795. 16/02/14 20:33:44 INFO TaskSetManager: Finished task 35.0 in stage 1.0 (TID 118) in 74 ms on 10.0.0.34 (59/60)
  796. 16/02/14 20:33:44 INFO TaskSetManager: Starting task 46.0 in stage 1.0 (TID 119, 10.0.0.34, PROCESS_LOCAL, 2208 bytes)
  797. 16/02/14 20:33:44 INFO DAGScheduler: ResultStage 1 (reduce at <console>:31) finished in 7.258 s
  798. 16/02/14 20:33:44 INFO DAGScheduler: Job 1 finished: reduce at <console>:31, took 7.369896 s
  799. 16/02/14 20:33:44 INFO TaskSetManager: Finished task 46.0 in stage 1.0 (TID 119) in 107 ms on 10.0.0.34 (60/60)
  800. 16/02/14 20:33:44 INFO YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
  801. 16/02/14 20:33:47 INFO SparkContext: Invoking stop() from shutdown hook
  802. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}
  803. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null}
  804. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null}
  805. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null}
  806. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null}
  807. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
  808. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
  809. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
  810. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
  811. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
  812. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
  813. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
  814. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
  815. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
  816. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
  817. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
  818. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
  819. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
  820. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
  821. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
  822. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
  823. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
  824. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
  825. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
  826. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
  827. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
  828. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
  829. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
  830. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
  831. 16/02/14 20:33:47 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
  832. 16/02/14 20:33:47 INFO SparkUI: Stopped Spark web UI at http://10.0.0.13:4041
  833. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=DAGScheduler.job.activeJobs, value=0
  834. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=DAGScheduler.job.allJobs, value=2
  835. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=DAGScheduler.stage.failedStages, value=0
  836. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=DAGScheduler.stage.runningStages, value=0
  837. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=DAGScheduler.stage.waitingStages, value=0
  838. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.BlockManager.disk.diskSpaceUsed_MB, value=0
  839. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.BlockManager.memory.maxMem_MB, value=11135
  840. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.BlockManager.memory.memUsed_MB, value=2779
  841. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.BlockManager.memory.remainingMem_MB, value=8356
  842. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.PS-MarkSweep.count, value=0
  843. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.PS-MarkSweep.time, value=0
  844. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.PS-Scavenge.count, value=6
  845. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.PS-Scavenge.time, value=1279
  846. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.heap.committed, value=880803840
  847. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.heap.init, value=1073741824
  848. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.heap.max, value=955252736
  849. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.heap.usage, value=0.3253750010719676
  850. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.heap.used, value=310922344
  851. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.non-heap.committed, value=115802112
  852. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.non-heap.init, value=24576000
  853. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.non-heap.max, value=318767104
  854. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.non-heap.usage, value=0.36178613963880035
  855. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.non-heap.used, value=115325520
  856. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.Code-Cache.committed, value=8847360
  857. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.Code-Cache.init, value=2555904
  858. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.Code-Cache.max, value=50331648
  859. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.Code-Cache.usage, value=0.1734441121419271
  860. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.Code-Cache.used, value=8729728
  861. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Eden-Space.committed, value=120061952
  862. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Eden-Space.init, value=268959744
  863. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Eden-Space.max, value=194510848
  864. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Eden-Space.usage, value=0.22939409528459823
  865. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Eden-Space.used, value=44619640
  866. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Old-Gen.committed, value=716177408
  867. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Old-Gen.init, value=716177408
  868. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Old-Gen.max, value=716177408
  869. 16/02/14 20:33:47 INFO MetricsSystemImpl: Stopping azure-file-system metrics system...
  870. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Old-Gen.usage, value=0.31046262771807515
  871. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Old-Gen.used, value=222346320
  872. 16/02/14 20:33:47 INFO MetricsSinkAdapter: azurefs2 thread interrupted.
  873. 16/02/14 20:33:47 INFO MetricsSystemImpl: azure-file-system metrics system stopped.
  874. 16/02/14 20:33:47 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.
  875. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Perm-Gen.committed, value=106954752
  876. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Perm-Gen.init, value=22020096
  877. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Perm-Gen.max, value=268435456
  878. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Perm-Gen.usage, value=0.39710575342178345
  879. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Perm-Gen.used, value=106597264
  880. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Survivor-Space.committed, value=44564480
  881. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Survivor-Space.init, value=44564480
  882. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Survivor-Space.max, value=44564480
  883. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Survivor-Space.usage, value=0.9995968089384191
  884. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.pools.PS-Survivor-Space.used, value=44546512
  885. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.total.committed, value=996605952
  886. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.total.init, value=1098317824
  887. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.total.max, value=1274019840
  888. 16/02/14 20:33:47 INFO metrics: type=GAUGE, name=application_1455282642107_0043.driver.jvm.total.used, value=429098168
  889. 16/02/14 20:33:47 INFO metrics: type=TIMER, name=DAGScheduler.messageProcessingTime, count=262, min=0.031, max=396.617279, mean=4.061550773343201, stddev=35.36671052133605, median=0.0817, p75=0.15289999999999998, p95=0.942298, p98=10.239571999999999, p99=100.562823, p999=396.617279, mean_rate=0.3160181405875423, m1=1.793229801009668, m5=0.49721466057064745, m15=0.2203868991582715, rate_unit=events/second, duration_unit=milliseconds
  890. 16/02/14 20:33:47 INFO DAGScheduler: Stopping DAGScheduler
  891. 16/02/14 20:33:47 INFO YarnClientSchedulerBackend: Shutting down all executors
  892. 16/02/14 20:33:47 INFO YarnClientSchedulerBackend: Interrupting monitor thread
  893. 16/02/14 20:33:47 INFO YarnClientSchedulerBackend: Asking each executor to shut down
  894. 16/02/14 20:33:47 INFO YarnExtensionServices: Stopping org.apache.spark.scheduler.cluster.YarnExtensionServices@39c8b90
  895. 16/02/14 20:33:47 INFO YarnClientSchedulerBackend: Stopped
  896. 16/02/14 20:33:48 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
  897. 16/02/14 20:33:48 INFO MemoryStore: MemoryStore cleared
  898. 16/02/14 20:33:48 INFO BlockManager: BlockManager stopped
  899. 16/02/14 20:33:48 INFO BlockManagerMaster: BlockManagerMaster stopped
  900. 16/02/14 20:33:48 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  901. 16/02/14 20:33:48 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
  902. 16/02/14 20:33:48 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
  903. 16/02/14 20:33:48 INFO SparkContext: Successfully stopped SparkContext
  904. 16/02/14 20:33:48 INFO ShutdownHookManager: Shutdown hook called
  905. 16/02/14 20:33:48 INFO ShutdownHookManager: Deleting directory /var/tmp/spark/spark-e6adb8dd-f870-4455-8b6a-abcc9653fed0
  906. 16/02/14 20:33:49 INFO ShutdownHookManager: Deleting directory /var/tmp/spark/spark-fb3a3311-7aaf-43d4-a9db-6dfb51f97422
  907. 16/02/14 20:33:49 INFO ShutdownHookManager: Deleting directory /var/tmp/spark/spark-fd3ea7dc-5e0c-4607-b5e5-a63e56851acf
  908. 16/02/14 20:33:49 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
  909. 16/02/14 20:33:49 INFO MetricsSystemImpl: Stopping azure-file-system metrics system...
  910. 16/02/14 20:33:49 INFO MetricsSinkAdapter: azurefs2 thread interrupted.
  911. 16/02/14 20:33:49 INFO MetricsSystemImpl: azure-file-system metrics system stopped.
  912. 16/02/14 20:33:49 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!