API tools faq
paste
Advertisement
fcnorman

icu-mortality-yarn-logs

fcnorman
Mar 9th, 2018
149
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 239.71 KB | None | 0 0
raw download clone embed print report
  1. Container: container_1520619475936_0001_02_000001 on 608788f18ea1_39033
  2. LogAggregationType: AGGREGATED
  3. =======================================================================
  4. LogType:directory.info
  5. LogLastModifiedTime:Fri Mar 09 18:55:10 +0000 2018
  6. LogLength:21689
  7. LogContents:
  8. ls -l:
  9. total 36
  10. lrwxrwxrwx 1 hduser hadoop 89 Mar 9 18:54 __app__.jar -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar
  11. lrwxrwxrwx 1 hduser hadoop 80 Mar 9 18:54 __spark_conf__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip
  12. lrwxrwxrwx 1 hduser hadoop 99 Mar 9 18:54 __spark_libs__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip
  13. -rw-r--r-- 1 hduser hadoop 69 Mar 9 18:54 container_tokens
  14. -rwx------ 1 hduser hadoop 733 Mar 9 18:54 default_container_executor.sh
  15. -rwx------ 1 hduser hadoop 678 Mar 9 18:54 default_container_executor_session.sh
  16. -rwx------ 1 hduser hadoop 5896 Mar 9 18:54 launch_container.sh
  17. drwx--x--- 2 hduser hadoop 4096 Mar 9 18:54 tmp
  18. find -L . -maxdepth 5 -ls:
  19. 1675507 4 drwx--x--- 3 hduser hadoop 4096 Mar 9 18:54 .
  20. 7089826 4 -rwx------ 1 hduser hadoop 733 Mar 9 18:54 ./default_container_executor.sh
  21. 7089822 8 -rwx------ 1 hduser hadoop 5896 Mar 9 18:54 ./launch_container.sh
  22. 1675509 4 drwx--x--- 2 hduser hadoop 4096 Mar 9 18:54 ./tmp
  23. 7089618 517916 -r-x------ 1 hduser hadoop 530343861 Mar 9 18:54 ./__app__.jar
  24. 7089827 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:54 ./.default_container_executor.sh.crc
  25. 1670085 12 drwx------ 2 hduser hadoop 12288 Mar 9 18:53 ./__spark_libs__
  26. 7089636 88 -r-x------ 1 hduser hadoop 87192 Mar 9 18:53 ./__spark_libs__/spire-macros_2.11-0.13.0.jar
  27. 7089637 5612 -r-x------ 1 hduser hadoop 5744974 Mar 9 18:53 ./__spark_libs__/scala-library-2.11.8.jar
  28. 7089638 20 -r-x------ 1 hduser hadoop 19827 Mar 9 18:53 ./__spark_libs__/opencsv-2.3.jar
  29. 7089639 16 -r-x------ 1 hduser hadoop 13911 Mar 9 18:53 ./__spark_libs__/generex-1.0.1.jar
  30. 7089640 364 -r-x------ 1 hduser hadoop 370119 Mar 9 18:53 ./__spark_libs__/lz4-java-1.4.0.jar
  31. 7089641 256 -r-x------ 1 hduser hadoop 258919 Mar 9 18:53 ./__spark_libs__/jackson-core-2.6.7.jar
  32. 7089642 684 -r-x------ 1 hduser hadoop 698375 Mar 9 18:53 ./__spark_libs__/jersey-common-2.22.2.jar
  33. 7089643 472 -r-x------ 1 hduser hadoop 479881 Mar 9 18:53 ./__spark_libs__/commons-lang3-3.5.jar
  34. 7089644 6852 -r-x------ 1 hduser hadoop 7015233 Mar 9 18:53 ./__spark_libs__/kubernetes-model-2.0.0.jar
  35. 7089645 176 -r-x------ 1 hduser hadoop 178947 Mar 9 18:53 ./__spark_libs__/hk2-api-2.4.0-b34.jar
  36. 7089646 7172 -r-x------ 1 hduser hadoop 7343426 Mar 9 18:53 ./__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
  37. 7089648 504 -r-x------ 1 hduser hadoop 515645 Mar 9 18:53 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
  38. 7089649 44 -r-x------ 1 hduser hadoop 42858 Mar 9 18:53 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar
  39. 7089650 60 -r-x------ 1 hduser hadoop 60483 Mar 9 18:53 ./__spark_libs__/chill-java-0.8.4.jar
  40. 7089651 132 -r-x------ 1 hduser hadoop 134595 Mar 9 18:53 ./__spark_libs__/commons-crypto-1.0.0.jar
  41. 7089652 128 -r-x------ 1 hduser hadoop 130802 Mar 9 18:53 ./__spark_libs__/aircompressor-0.8.jar
  42. 7089653 264 -r-x------ 1 hduser hadoop 269295 Mar 9 18:53 ./__spark_libs__/snakeyaml-1.15.jar
  43. 7089654 1168 -r-x------ 1 hduser hadoop 1194003 Mar 9 18:53 ./__spark_libs__/arpack_combined_all-0.1.jar
  44. 7089655 14760 -r-x------ 1 hduser hadoop 15113382 Mar 9 18:53 ./__spark_libs__/breeze_2.11-0.13.2.jar
  45. 7089656 36 -r-x------ 1 hduser hadoop 35518 Mar 9 18:53 ./__spark_libs__/zjsonpatch-0.3.0.jar
  46. 7089657 120 -r-x------ 1 hduser hadoop 120321 Mar 9 18:53 ./__spark_libs__/spark-repl_2.11-2.3.0.jar
  47. 7089658 36 -r-x------ 1 hduser hadoop 34603 Mar 9 18:53 ./__spark_libs__/machinist_2.11-0.6.1.jar
  48. 7089659 28 -r-x------ 1 hduser hadoop 26366 Mar 9 18:53 ./__spark_libs__/javax.annotation-api-1.2.jar
  49. 7089660 180 -r-x------ 1 hduser hadoop 180736 Mar 9 18:53 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
  50. 7089661 788 -r-x------ 1 hduser hadoop 802818 Mar 9 18:53 ./__spark_libs__/scalap-2.11.8.jar
  51. 7089662 1504 -r-x------ 1 hduser hadoop 1537966 Mar 9 18:53 ./__spark_libs__/hadoop-mapreduce-client-core-2.6.5.jar
  52. 7089663 1024 -r-x------ 1 hduser hadoop 1048116 Mar 9 18:53 ./__spark_libs__/parquet-jackson-1.8.2.jar
  53. 7089664 172 -r-x------ 1 hduser hadoop 174351 Mar 9 18:53 ./__spark_libs__/stream-2.7.0.jar
  54. 7089665 68 -r-x------ 1 hduser hadoop 66270 Mar 9 18:53 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar
  55. 7089666 1632 -r-x------ 1 hduser hadoop 1671083 Mar 9 18:53 ./__spark_libs__/hppc-0.7.2.jar
  56. 7089667 180 -r-x------ 1 hduser hadoop 183832 Mar 9 18:53 ./__spark_libs__/spark-mllib-local_2.11-2.3.0.jar
  57. 7089668 656 -r-x------ 1 hduser hadoop 671138 Mar 9 18:53 ./__spark_libs__/scala-xml_2.11-1.0.5.jar
  58. 7089669 200 -r-x------ 1 hduser hadoop 201928 Mar 9 18:53 ./__spark_libs__/RoaringBitmap-0.5.11.jar
  59. 7089670 96 -r-x------ 1 hduser hadoop 94796 Mar 9 18:53 ./__spark_libs__/pyrolite-4.13.jar
  60. 7089671 3692 -r-x------ 1 hduser hadoop 3780056 Mar 9 18:53 ./__spark_libs__/netty-all-4.1.17.Final.jar
  61. 7089672 316 -r-x------ 1 hduser hadoop 320444 Mar 9 18:53 ./__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
  62. 7089673 1024 -r-x------ 1 hduser hadoop 1045744 Mar 9 18:53 ./__spark_libs__/leveldbjni-all-1.8.jar
  63. 7089674 376 -r-x------ 1 hduser hadoop 381098 Mar 9 18:53 ./__spark_libs__/spark-kubernetes_2.11-2.3.0.jar
  64. 7089675 32 -r-x------ 1 hduser hadoop 32612 Mar 9 18:53 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
  65. 7089676 636 -r-x------ 1 hduser hadoop 650236 Mar 9 18:53 ./__spark_libs__/spark-yarn_2.11-2.3.0.jar
  66. 7089677 48 -r-x------ 1 hduser hadoop 46986 Mar 9 18:53 ./__spark_libs__/jackson-annotations-2.6.7.jar
  67. 7089678 700 -r-x------ 1 hduser hadoop 714194 Mar 9 18:53 ./__spark_libs__/javassist-3.18.1-GA.jar
  68. 7089679 248 -r-x------ 1 hduser hadoop 250377 Mar 9 18:53 ./__spark_libs__/parquet-hadoop-1.8.2.jar
  69. 7089680 64 -r-x------ 1 hduser hadoop 64847 Mar 9 18:53 ./__spark_libs__/spark-network-shuffle_2.11-2.3.0.jar
  70. 7089681 116 -r-x------ 1 hduser hadoop 116859 Mar 9 18:53 ./__spark_libs__/py4j-0.10.6.jar
  71. 7089682 76 -r-x------ 1 hduser hadoop 75835 Mar 9 18:53 ./__spark_libs__/spark-launcher_2.11-2.3.0.jar
  72. 7089683 20 -r-x------ 1 hduser hadoop 16430 Mar 9 18:53 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar
  73. 7089684 52 -r-x------ 1 hduser hadoop 52076 Mar 9 18:53 ./__spark_libs__/arrow-format-0.8.0.jar
  74. 7089685 84 -r-x------ 1 hduser hadoop 82421 Mar 9 18:53 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar
  75. 7089686 740 -r-x------ 1 hduser hadoop 757423 Mar 9 18:53 ./__spark_libs__/orc-mapreduce-1.4.1-nohive.jar
  76. 7089687 1300 -r-x------ 1 hduser hadoop 1330219 Mar 9 18:53 ./__spark_libs__/netty-3.9.9.Final.jar
  77. 7089688 120 -r-x------ 1 hduser hadoop 118973 Mar 9 18:53 ./__spark_libs__/hk2-utils-2.4.0-b34.jar
  78. 7089689 408 -r-x------ 1 hduser hadoop 416731 Mar 9 18:53 ./__spark_libs__/kubernetes-client-3.0.0.jar
  79. 7089690 96 -r-x------ 1 hduser hadoop 95806 Mar 9 18:53 ./__spark_libs__/javax.servlet-api-3.1.0.jar
  80. 7089691 936 -r-x------ 1 hduser hadoop 956984 Mar 9 18:53 ./__spark_libs__/parquet-column-1.8.2.jar
  81. 7089692 1412 -r-x------ 1 hduser hadoop 1441994 Mar 9 18:53 ./__spark_libs__/orc-core-1.4.1-nohive.jar
  82. 7089693 576 -r-x------ 1 hduser hadoop 589462 Mar 9 18:53 ./__spark_libs__/json4s-core_2.11-3.2.11.jar
  83. 7089694 2328 -r-x------ 1 hduser hadoop 2382165 Mar 9 18:53 ./__spark_libs__/spark-network-common_2.11-2.3.0.jar
  84. 7089695 12 -r-x------ 1 hduser hadoop 8268 Mar 9 18:53 ./__spark_libs__/logging-interceptor-3.8.1.jar
  85. 7089696 40 -r-x------ 1 hduser hadoop 40341 Mar 9 18:53 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar
  86. 7089697 52 -r-x------ 1 hduser hadoop 53007 Mar 9 18:53 ./__spark_libs__/spark-kvstore_2.11-2.3.0.jar
  87. 7089698 8 -r-x------ 1 hduser hadoop 5950 Mar 9 18:53 ./__spark_libs__/javax.inject-2.4.0-b34.jar
  88. 7089699 376 -r-x------ 1 hduser hadoop 384150 Mar 9 18:53 ./__spark_libs__/univocity-parsers-2.5.9.jar
  89. 7089700 7560 -r-x------ 1 hduser hadoop 7738527 Mar 9 18:53 ./__spark_libs__/spark-mllib_2.11-2.3.0.jar
  90. 7089701 116 -r-x------ 1 hduser hadoop 115534 Mar 9 18:53 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar
  91. 7089702 72 -r-x------ 1 hduser hadoop 72733 Mar 9 18:53 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar
  92. 7089703 748 -r-x------ 1 hduser hadoop 764569 Mar 9 18:53 ./__spark_libs__/jtransforms-2.4.0.jar
  93. 7089704 12 -r-x------ 1 hduser hadoop 10166 Mar 9 18:53 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
  94. 7089705 36 -r-x------ 1 hduser hadoop 34654 Mar 9 18:53 ./__spark_libs__/paranamer-2.8.jar
  95. 7089706 8464 -r-x------ 1 hduser hadoop 8664987 Mar 9 18:53 ./__spark_libs__/spark-sql_2.11-2.3.0.jar
  96. 7089707 164 -r-x------ 1 hduser hadoop 164422 Mar 9 18:53 ./__spark_libs__/core-1.1.2.jar
  97. 7089708 44 -r-x------ 1 hduser hadoop 42032 Mar 9 18:53 ./__spark_libs__/parquet-common-1.8.2.jar
  98. 7089709 36 -r-x------ 1 hduser hadoop 33015 Mar 9 18:53 ./__spark_libs__/jsr305-1.3.9.jar
  99. 7089710 80 -r-x------ 1 hduser hadoop 81811 Mar 9 18:53 ./__spark_libs__/okio-1.13.0.jar
  100. 7089711 328 -r-x------ 1 hduser hadoop 334662 Mar 9 18:53 ./__spark_libs__/antlr4-runtime-4.7.jar
  101. 7089712 392 -r-x------ 1 hduser hadoop 398122 Mar 9 18:53 ./__spark_libs__/okhttp-3.8.1.jar
  102. 7089713 352 -r-x------ 1 hduser hadoop 358390 Mar 9 18:53 ./__spark_libs__/kryo-shaded-3.0.3.jar
  103. 7089714 80 -r-x------ 1 hduser hadoop 79150 Mar 9 18:53 ./__spark_libs__/arrow-memory-0.8.0.jar
  104. 7089715 15128 -r-x------ 1 hduser hadoop 15487351 Mar 9 18:53 ./__spark_libs__/scala-compiler-2.11.8.jar
  105. 7089716 164 -r-x------ 1 hduser hadoop 167421 Mar 9 18:53 ./__spark_libs__/jersey-client-2.22.2.jar
  106. 7089717 384 -r-x------ 1 hduser hadoop 390733 Mar 9 18:53 ./__spark_libs__/parquet-format-2.3.1.jar
  107. 7089718 64 -r-x------ 1 hduser hadoop 63777 Mar 9 18:53 ./__spark_libs__/validation-api-1.1.0.Final.jar
  108. 7089719 4468 -r-x------ 1 hduser hadoop 4573750 Mar 9 18:53 ./__spark_libs__/scala-reflect-2.11.8.jar
  109. 7089720 932 -r-x------ 1 hduser hadoop 951701 Mar 9 18:53 ./__spark_libs__/jersey-server-2.22.2.jar
  110. 7089721 48 -r-x------ 1 hduser hadoop 48464 Mar 9 18:53 ./__spark_libs__/spark-unsafe_2.11-2.3.0.jar
  111. 7089722 184 -r-x------ 1 hduser hadoop 187882 Mar 9 18:53 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar
  112. 7089723 3444 -r-x------ 1 hduser hadoop 3522616 Mar 9 18:53 ./__spark_libs__/shapeless_2.11-2.3.2.jar
  113. 7089724 8 -r-x------ 1 hduser hadoop 5711 Mar 9 18:53 ./__spark_libs__/minlog-1.3.0.jar
  114. 7089725 20 -r-x------ 1 hduser hadoop 18098 Mar 9 18:53 ./__spark_libs__/jersey-container-servlet-2.22.2.jar
  115. 7089726 40 -r-x------ 1 hduser hadoop 37887 Mar 9 18:53 ./__spark_libs__/commons-compiler-3.0.8.jar
  116. 7089727 44 -r-x------ 1 hduser hadoop 41755 Mar 9 18:53 ./__spark_libs__/objenesis-2.1.jar
  117. 7089728 280 -r-x------ 1 hduser hadoop 286071 Mar 9 18:53 ./__spark_libs__/parquet-encoding-1.8.2.jar
  118. 7089729 220 -r-x------ 1 hduser hadoop 224167 Mar 9 18:53 ./__spark_libs__/chill_2.11-0.8.4.jar
  119. 7089730 780 -r-x------ 1 hduser hadoop 796326 Mar 9 18:53 ./__spark_libs__/janino-3.0.8.jar
  120. 7089731 1244 -r-x------ 1 hduser hadoop 1270377 Mar 9 18:53 ./__spark_libs__/arrow-vector-0.8.0.jar
  121. 7089732 24 -r-x------ 1 hduser hadoop 21247 Mar 9 18:53 ./__spark_libs__/metrics-graphite-3.1.5.jar
  122. 7089733 1988 -r-x------ 1 hduser hadoop 2035066 Mar 9 18:53 ./__spark_libs__/commons-math3-3.4.1.jar
  123. 7089734 32 -r-x------ 1 hduser hadoop 30057 Mar 9 18:53 ./__spark_libs__/spark-sketch_2.11-2.3.0.jar
  124. 7089735 2120 -r-x------ 1 hduser hadoop 2169825 Mar 9 18:53 ./__spark_libs__/spark-streaming_2.11-2.3.0.jar
  125. 7089736 9888 -r-x------ 1 hduser hadoop 10121868 Mar 9 18:53 ./__spark_libs__/spire_2.11-0.13.0.jar
  126. 7089737 8 -r-x------ 1 hduser hadoop 4596 Mar 9 18:53 ./__spark_libs__/jul-to-slf4j-1.7.16.jar
  127. 7089738 20 -r-x------ 1 hduser hadoop 20235 Mar 9 18:53 ./__spark_libs__/osgi-resource-locator-1.0.1.jar
  128. 7089739 4 -r-x------ 1 hduser hadoop 3142 Mar 9 18:53 ./__spark_libs__/macro-compat_2.11-1.1.1.jar
  129. 7089740 280 -r-x------ 1 hduser hadoop 284220 Mar 9 18:53 ./__spark_libs__/commons-lang-2.6.jar
  130. 7089741 1256 -r-x------ 1 hduser hadoop 1282424 Mar 9 18:53 ./__spark_libs__/ivy-2.4.0.jar
  131. 7089742 80 -r-x------ 1 hduser hadoop 79845 Mar 9 18:53 ./__spark_libs__/compress-lzf-1.0.3.jar
  132. 7089743 16 -r-x------ 1 hduser hadoop 15824 Mar 9 18:53 ./__spark_libs__/metrics-json-3.1.5.jar
  133. 7089744 12680 -r-x------ 1 hduser hadoop 12982716 Mar 9 18:53 ./__spark_libs__/spark-core_2.11-2.3.0.jar
  134. 7089745 180 -r-x------ 1 hduser hadoop 181271 Mar 9 18:53 ./__spark_libs__/hk2-locator-2.4.0-b34.jar
  135. 7089746 416 -r-x------ 1 hduser hadoop 423753 Mar 9 18:53 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
  136. 7089747 1140 -r-x------ 1 hduser hadoop 1165323 Mar 9 18:53 ./__spark_libs__/jackson-databind-2.6.7.1.jar
  137. 7089748 208 -r-x------ 1 hduser hadoop 212453 Mar 9 18:53 ./__spark_libs__/commons-net-2.2.jar
  138. 7089749 64 -r-x------ 1 hduser hadoop 65261 Mar 9 18:53 ./__spark_libs__/oro-2.0.8.jar
  139. 7089750 144 -r-x------ 1 hduser hadoop 144660 Mar 9 18:53 ./__spark_libs__/xbean-asm5-shaded-4.4.jar
  140. 7089751 8776 -r-x------ 1 hduser hadoop 8984422 Mar 9 18:53 ./__spark_libs__/spark-catalyst_2.11-2.3.0.jar
  141. 7089752 40 -r-x------ 1 hduser hadoop 39283 Mar 9 18:53 ./__spark_libs__/metrics-jvm-3.1.5.jar
  142. 7089753 280 -r-x------ 1 hduser hadoop 284184 Mar 9 18:53 ./__spark_libs__/commons-codec-1.10.jar
  143. 7089754 176 -r-x------ 1 hduser hadoop 176285 Mar 9 18:53 ./__spark_libs__/automaton-1.11-8.jar
  144. 7089755 2280 -r-x------ 1 hduser hadoop 2333186 Mar 9 18:53 ./__spark_libs__/zstd-jni-1.3.2-2.jar
  145. 7089756 648 -r-x------ 1 hduser hadoop 663328 Mar 9 18:53 ./__spark_libs__/spark-mesos_2.11-2.3.0.jar
  146. 7089757 696 -r-x------ 1 hduser hadoop 708667 Mar 9 18:53 ./__spark_libs__/spark-graphx_2.11-2.3.0.jar
  147. 7089758 16 -r-x------ 1 hduser hadoop 15456 Mar 9 18:53 ./__spark_libs__/spark-tags_2.11-2.3.0.jar
  148. 7089759 616 -r-x------ 1 hduser hadoop 627814 Mar 9 18:53 ./__spark_libs__/joda-time-2.9.3.jar
  149. 7089760 120 -r-x------ 1 hduser hadoop 120465 Mar 9 18:53 ./__spark_libs__/metrics-core-3.1.5.jar
  150. 7089761 16 -r-x------ 1 hduser hadoop 14766 Mar 9 18:53 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
  151. 7089762 952 -r-x------ 1 hduser hadoop 971310 Mar 9 18:53 ./__spark_libs__/jersey-guava-2.22.2.jar
  152. 7089823 4 -rw-r--r-- 1 hduser hadoop 56 Mar 9 18:54 ./.launch_container.sh.crc
  153. 7089820 4 -rw-r--r-- 1 hduser hadoop 69 Mar 9 18:54 ./container_tokens
  154. 7089825 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:54 ./.default_container_executor_session.sh.crc
  155. 7089824 4 -rwx------ 1 hduser hadoop 678 Mar 9 18:54 ./default_container_executor_session.sh
  156. 1663502 4 drwx------ 3 hduser hadoop 4096 Mar 9 18:53 ./__spark_conf__
  157. 7089811 4 -r-x------ 1 hduser hadoop 1192 Mar 9 18:53 ./__spark_conf__/__spark_conf__.properties
  158. 1663505 4 drwx------ 2 hduser hadoop 4096 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__
  159. 7089794 4 -r-x------ 1 hduser hadoop 2316 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example
  160. 7089798 4 -r-x------ 1 hduser hadoop 891 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/slaves
  161. 7089809 4 -r-x------ 1 hduser hadoop 682 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-site.xml
  162. 7089803 4 -r-x------ 1 hduser hadoop 996 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/docker.properties.template
  163. 7089790 4 -r-x------ 1 hduser hadoop 1764 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-env.sh
  164. 7089783 4 -r-x------ 1 hduser hadoop 3518 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-acls.xml
  165. 7089808 8 -r-x------ 1 hduser hadoop 4113 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template
  166. 7089800 4 -r-x------ 1 hduser hadoop 21 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-signature.secret
  167. 7089799 4 -r-x------ 1 hduser hadoop 1484 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-env.sh
  168. 7089785 4 -r-x------ 1 hduser hadoop 482 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-env.sh
  169. 7089774 4 -r-x------ 1 hduser hadoop 3670 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd
  170. 7089770 4 -r-x------ 1 hduser hadoop 2016 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.spark.properties
  171. 7089779 8 -r-x------ 1 hduser hadoop 4221 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-env.sh.template
  172. 7089769 4 -r-x------ 1 hduser hadoop 137 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
  173. 7089793 4 -r-x------ 1 hduser hadoop 3414 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-user-functions.sh.example
  174. 7089781 4 -r-x------ 1 hduser hadoop 2250 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd
  175. 7089788 4 -r-x------ 1 hduser hadoop 589 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/container-executor.cfg
  176. 7089778 4 -r-x------ 1 hduser hadoop 890 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/workers
  177. 7089807 4 -r-x------ 1 hduser hadoop 2697 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example
  178. 7089771 4 -r-x------ 1 hduser hadoop 1657 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-log4j.properties
  179. 7089791 8 -r-x------ 1 hduser hadoop 7801 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/metrics.properties.template
  180. 7089789 4 -r-x------ 1 hduser hadoop 1781 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf
  181. 7089777 4 -r-x------ 1 hduser hadoop 225 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/core-site.xml
  182. 7089772 4 -r-x------ 1 hduser hadoop 3323 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties
  183. 7089782 8 -r-x------ 1 hduser hadoop 7861 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml
  184. 7089805 4 -r-x------ 1 hduser hadoop 1292 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf.template
  185. 7089792 4 -r-x------ 1 hduser hadoop 2642 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/user_ec_policies.xml.template
  186. 7089804 4 -r-x------ 1 hduser hadoop 951 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd
  187. 7089801 4 -r-x------ 1 hduser hadoop 620 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-site.xml
  188. 7089806 4 -r-x------ 1 hduser hadoop 3589 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j2.xml
  189. 7089796 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.properties.template
  190. 7089776 4 -r-x------ 1 hduser hadoop 865 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/slaves.template
  191. 7089787 4 -r-x------ 1 hduser hadoop 809 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
  192. 7089768 16 -r-x------ 1 hduser hadoop 16270 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh
  193. 7089784 4 -r-x------ 1 hduser hadoop 1335 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/configuration.xsl
  194. 7089773 8 -r-x------ 1 hduser hadoop 8048 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
  195. 7089767 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.properties
  196. 7089795 4 -r-x------ 1 hduser hadoop 1351 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-env.sh
  197. 7089780 4 -r-x------ 1 hduser hadoop 1747 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties
  198. 7089797 12 -r-x------ 1 hduser hadoop 10206 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml
  199. 7089802 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/metrics.properties
  200. 7089786 8 -r-x------ 1 hduser hadoop 5502 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-env.sh
  201. 7089775 4 -r-x------ 1 hduser hadoop 1105 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.template
  202. 7089764 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/log4j.properties
  203. 7089810 172 -r-x------ 1 hduser hadoop 173257 Mar 9 18:53 ./__spark_conf__/__spark_hadoop_conf__.xml
  204. 7089765 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:53 ./__spark_conf__/metrics.properties
  205. 7089821 4 -rw-r--r-- 1 hduser hadoop 12 Mar 9 18:54 ./.container_tokens.crc
  206. broken symlinks(find -L . -maxdepth 5 -type l -ls):
  207.  
  208. End of LogType:directory.info
  209. *******************************************************************************
  210.  
  211. Container: container_1520619475936_0001_02_000001 on 608788f18ea1_39033
  212. LogAggregationType: AGGREGATED
  213. =======================================================================
  214. LogType:launch_container.sh
  215. LogLastModifiedTime:Fri Mar 09 18:55:10 +0000 2018
  216. LogLength:5896
  217. LogContents:
  218. #!/bin/bash
  219.  
  220. set -o pipefail -e
  221. export PRELAUNCH_OUT="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/prelaunch.out"
  222. exec >"${PRELAUNCH_OUT}"
  223. export PRELAUNCH_ERR="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/prelaunch.err"
  224. exec 2>"${PRELAUNCH_ERR}"
  225. echo "Setting up env variables"
  226. export SPARK_YARN_STAGING_DIR="hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001"
  227. export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
  228. export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
  229. export JAVA_HOME="/usr/java/default"
  230. export APP_SUBMIT_TIME_ENV="1520621561751"
  231. export NM_HOST="608788f18ea1"
  232. export HADOOP_HDFS_HOME="/usr/local/hadoop"
  233. export LOGNAME="hduser"
  234. export JVM_PID="$$"
  235. export PWD="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_02_000001"
  236. export HADOOP_COMMON_HOME="/usr/local/hadoop"
  237. export PYTHONHASHSEED="0"
  238. export LOCAL_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001"
  239. export APPLICATION_WEB_PROXY_BASE="/proxy/application_1520619475936_0001"
  240. export NM_HTTP_PORT="18042"
  241. export SPARK_DIST_CLASSPATH="/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*"
  242. export LOG_DIRS="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001"
  243. export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  244. "
  245. export NM_PORT="39033"
  246. export USER="hduser"
  247. export HADOOP_YARN_HOME="/usr/local/hadoop"
  248. export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$PWD/__spark_conf__/__hadoop_conf__"
  249. export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_02_000001/container_tokens"
  250. export SPARK_USER="hduser"
  251. export LOCAL_USER_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/"
  252. export HADOOP_HOME="/usr/local/hadoop"
  253. export HOME="/home/"
  254. export CONTAINER_ID="container_1520619475936_0001_02_000001"
  255. export MALLOC_ARENA_MAX="4"
  256. echo "Setting up job resources"
  257. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip" "__spark_libs__"
  258. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar" "__app__.jar"
  259. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip" "__spark_conf__"
  260. echo "Copying debugging information"
  261. # Creating copy of launch script
  262. cp "launch_container.sh" "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/launch_container.sh"
  263. chmod 640 "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/launch_container.sh"
  264. # Determining directory contents
  265. echo "ls -l:" 1>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/directory.info"
  266. ls -l 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/directory.info"
  267. echo "find -L . -maxdepth 5 -ls:" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/directory.info"
  268. find -L . -maxdepth 5 -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/directory.info"
  269. echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/directory.info"
  270. find -L . -maxdepth 5 -type l -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/directory.info"
  271. echo "Launching container"
  272. exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx5120m -Djava.io.tmpdir=$PWD/tmp '-XX:+PrintGCDetails' '-Dlog4j.configuration=file:/usr/local/hadoop/etc/hadoop/log4j.spark.properties' -Dspark.yarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy' --jar file:/home/hduser/icu-mortality-1.0.0-all.jar --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/stdout 2> /usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000001/stderr"
  273.  
  274. End of LogType:launch_container.sh
  275. ************************************************************************************
  276.  
  277.  
  278. End of LogType:prelaunch.err
  279. ******************************************************************************
  280.  
  281. Container: container_1520619475936_0001_02_000001 on 608788f18ea1_39033
  282. LogAggregationType: AGGREGATED
  283. =======================================================================
  284. LogType:prelaunch.out
  285. LogLastModifiedTime:Fri Mar 09 18:55:10 +0000 2018
  286. LogLength:100
  287. LogContents:
  288. Setting up env variables
  289. Setting up job resources
  290. Copying debugging information
  291. Launching container
  292.  
  293. End of LogType:prelaunch.out
  294. ******************************************************************************
  295.  
  296. Container: container_1520619475936_0001_02_000001 on 608788f18ea1_39033
  297. LogAggregationType: AGGREGATED
  298. =======================================================================
  299. LogType:stderr
  300. LogLastModifiedTime:Fri Mar 09 18:55:10 +0000 2018
  301. LogLength:24204
  302. LogContents:
  303. log4j:WARN custom level class [ Define the root logger] not found.
  304. 18/03/09 18:54:07 INFO SignalUtils: Registered signal handler for TERM
  305. 18/03/09 18:54:07 INFO SignalUtils: Registered signal handler for HUP
  306. 18/03/09 18:54:07 INFO SignalUtils: Registered signal handler for INT
  307. 18/03/09 18:54:08 INFO SecurityManager: Changing view acls to: hduser
  308. 18/03/09 18:54:08 INFO SecurityManager: Changing modify acls to: hduser
  309. 18/03/09 18:54:08 INFO SecurityManager: Changing view acls groups to:
  310. 18/03/09 18:54:08 INFO SecurityManager: Changing modify acls groups to:
  311. 18/03/09 18:54:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  312. 18/03/09 18:54:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  313. 18/03/09 18:54:08 INFO ApplicationMaster: Preparing Local resources
  314. 18/03/09 18:54:11 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1520619475936_0001_000002
  315. 18/03/09 18:54:11 INFO ApplicationMaster: Starting the user application in a separate Thread
  316. 18/03/09 18:54:11 INFO ApplicationMaster: Waiting for spark context initialization...
  317. ERROR StatusLogger Unrecognized format specifier [d]
  318. ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
  319. ERROR StatusLogger Unrecognized format specifier [thread]
  320. ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
  321. ERROR StatusLogger Unrecognized format specifier [level]
  322. ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
  323. ERROR StatusLogger Unrecognized format specifier [logger]
  324. ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
  325. ERROR StatusLogger Unrecognized format specifier [msg]
  326. ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
  327. ERROR StatusLogger Unrecognized format specifier [n]
  328. ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
  329. ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
  330. ERROR StatusLogger Unrecognized format specifier [d]
  331. ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
  332. ERROR StatusLogger Unrecognized format specifier [thread]
  333. ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
  334. ERROR StatusLogger Unrecognized format specifier [level]
  335. ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
  336. ERROR StatusLogger Unrecognized format specifier [logger]
  337. ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
  338. ERROR StatusLogger Unrecognized format specifier [msg]
  339. ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
  340. ERROR StatusLogger Unrecognized format specifier [n]
  341. ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
  342. 18/03/09 18:54:12 INFO SparkContext: Running Spark version 2.3.0
  343. 18/03/09 18:54:12 WARN SparkConf: spark.master yarn-cluster is deprecated in Spark 2.0+, please instead use "yarn" with specified deploy mode.
  344. 18/03/09 18:54:12 INFO SparkContext: Submitted application: ICU Mortality Predictor
  345. 18/03/09 18:54:12 INFO SecurityManager: Changing view acls to: hduser
  346. 18/03/09 18:54:12 INFO SecurityManager: Changing modify acls to: hduser
  347. 18/03/09 18:54:12 INFO SecurityManager: Changing view acls groups to:
  348. 18/03/09 18:54:12 INFO SecurityManager: Changing modify acls groups to:
  349. 18/03/09 18:54:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  350. 18/03/09 18:54:13 INFO SparkEnv: Registering MapOutputTracker
  351. 18/03/09 18:54:13 INFO SparkEnv: Registering BlockManagerMaster
  352. 18/03/09 18:54:13 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  353. 18/03/09 18:54:13 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  354. 18/03/09 18:54:13 INFO DiskBlockManager: Created local directory at /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/blockmgr-960153d5-a74c-418d-9dba-3d9d717bd9b2
  355. 18/03/09 18:54:13 INFO MemoryStore: MemoryStore started with capacity 4.5 GB
  356. 18/03/09 18:54:13 INFO SparkEnv: Registering OutputCommitCoordinator
  357. 18/03/09 18:54:14 INFO log: Logging initialized @7373ms
  358. 18/03/09 18:54:14 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  359. 18/03/09 18:54:14 INFO Server: jetty-9.3.z-SNAPSHOT
  360. 18/03/09 18:54:14 INFO Server: Started @7658ms
  361. 18/03/09 18:54:14 INFO AbstractConnector: Started ServerConnector@4de6e829{HTTP/1.1,[http/1.1]}{0.0.0.0:36441}
  362. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2705df2c{/jobs,null,AVAILABLE,@Spark}
  363. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@349dd026{/jobs/json,null,AVAILABLE,@Spark}
  364. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2455e52f{/jobs/job,null,AVAILABLE,@Spark}
  365. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60bb75b3{/jobs/job/json,null,AVAILABLE,@Spark}
  366. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6b4c8816{/stages,null,AVAILABLE,@Spark}
  367. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1cf086b8{/stages/json,null,AVAILABLE,@Spark}
  368. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4a2ac742{/stages/stage,null,AVAILABLE,@Spark}
  369. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@66d98369{/stages/stage/json,null,AVAILABLE,@Spark}
  370. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7ad54827{/stages/pool,null,AVAILABLE,@Spark}
  371. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@323017d9{/stages/pool/json,null,AVAILABLE,@Spark}
  372. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6c10d87f{/storage,null,AVAILABLE,@Spark}
  373. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4678dc5f{/storage/json,null,AVAILABLE,@Spark}
  374. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@163c20c7{/storage/rdd,null,AVAILABLE,@Spark}
  375. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@24007306{/storage/rdd/json,null,AVAILABLE,@Spark}
  376. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4e3994d9{/environment,null,AVAILABLE,@Spark}
  377. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4ea32365{/environment/json,null,AVAILABLE,@Spark}
  378. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@396e086e{/executors,null,AVAILABLE,@Spark}
  379. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1e51c406{/executors/json,null,AVAILABLE,@Spark}
  380. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1c4c0449{/executors/threadDump,null,AVAILABLE,@Spark}
  381. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4ba3a4ae{/executors/threadDump/json,null,AVAILABLE,@Spark}
  382. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@247e8cc{/static,null,AVAILABLE,@Spark}
  383. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@ade5{/,null,AVAILABLE,@Spark}
  384. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6281a686{/api,null,AVAILABLE,@Spark}
  385. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@731a6c8f{/jobs/job/kill,null,AVAILABLE,@Spark}
  386. 18/03/09 18:54:14 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@529d7db5{/stages/stage/kill,null,AVAILABLE,@Spark}
  387. 18/03/09 18:54:14 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://608788f18ea1:36441
  388. 18/03/09 18:54:15 INFO YarnClusterScheduler: Created YarnClusterScheduler
  389. 18/03/09 18:54:15 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1520619475936_0001 and attemptId Some(appattempt_1520619475936_0001_000002)
  390. 18/03/09 18:54:15 INFO NettyBlockTransferService: Server created on 608788f18ea1:38735
  391. 18/03/09 18:54:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  392. 18/03/09 18:54:15 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 608788f18ea1, 38735, None)
  393. 18/03/09 18:54:15 INFO BlockManagerMasterEndpoint: Registering block manager 608788f18ea1:38735 with 4.5 GB RAM, BlockManagerId(driver, 608788f18ea1, 38735, None)
  394. 18/03/09 18:54:15 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 608788f18ea1, 38735, None)
  395. 18/03/09 18:54:15 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 608788f18ea1, 38735, None)
  396. 18/03/09 18:54:15 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@219e1118{/metrics/json,null,AVAILABLE,@Spark}
  397. 18/03/09 18:54:16 INFO EventLoggingListener: Logging events to hdfs://master:9000/user/hduser/input/application_1520619475936_0001_2
  398. 18/03/09 18:54:16 INFO ApplicationMaster:
  399. ===============================================================================
  400. YARN executor launch context:
  401. env:
  402. CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>/usr/local/hadoop/etc/hadoop<CPS>/usr/local/hadoop/share/hadoop/common/lib/*<CPS>/usr/local/hadoop/share/hadoop/common/*<CPS>/usr/local/hadoop/share/hadoop/hdfs<CPS>/usr/local/hadoop/share/hadoop/hdfs/lib/*<CPS>/usr/local/hadoop/share/hadoop/hdfs/*<CPS>/usr/local/hadoop/share/hadoop/mapreduce/*<CPS>/usr/local/hadoop/share/hadoop/yarn<CPS>/usr/local/hadoop/share/hadoop/yarn/lib/*<CPS>/usr/local/hadoop/share/hadoop/yarn/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
  403. SPARK_DIST_CLASSPATH -> /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*
  404. SPARK_YARN_STAGING_DIR -> *********(redacted)
  405. SPARK_USER -> *********(redacted)
  406.  
  407. command:
  408. {{JAVA_HOME}}/bin/java \
  409. -server \
  410. -Xmx1024m \
  411. '-XX:MaxGCPauseMillis=950' \
  412. -Djava.io.tmpdir={{PWD}}/tmp \
  413. -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
  414. -XX:OnOutOfMemoryError='kill %p' \
  415. org.apache.spark.executor.CoarseGrainedExecutorBackend \
  416. --driver-url \
  417. spark://CoarseGrainedScheduler@608788f18ea1:45035 \
  418. --executor-id \
  419. <executorId> \
  420. --hostname \
  421. <hostname> \
  422. --cores \
  423. 1 \
  424. --app-id \
  425. application_1520619475936_0001 \
  426. --user-class-path \
  427. file:$PWD/__app__.jar \
  428. 1><LOG_DIR>/stdout \
  429. 2><LOG_DIR>/stderr
  430.  
  431. resources:
  432. __app__.jar -> resource { scheme: "hdfs" host: "master" port: 9000 file: "/user/hduser/.sparkStaging/application_1520619475936_0001/icu-mortality-1.0.0-all.jar" } size: 530343861 timestamp: 1520621558212 type: FILE visibility: PRIVATE
  433. __spark_libs__ -> resource { scheme: "hdfs" host: "master" port: 9000 file: "/user/hduser/.sparkStaging/application_1520619475936_0001/__spark_libs__1007958283373928587.zip" } size: 155925110 timestamp: 1520621529995 type: ARCHIVE visibility: PRIVATE
  434. __spark_conf__ -> resource { scheme: "hdfs" host: "master" port: 9000 file: "/user/hduser/.sparkStaging/application_1520619475936_0001/__spark_conf__.zip" } size: 303564 timestamp: 1520621559848 type: ARCHIVE visibility: PRIVATE
  435.  
  436. ===============================================================================
  437. 18/03/09 18:54:16 INFO RMProxy: Connecting to ResourceManager at master/172.22.0.7:18030
  438. 18/03/09 18:54:16 INFO YarnRMClient: Registering the ApplicationMaster
  439. 18/03/09 18:54:17 INFO Configuration: resource-types.xml not found
  440. 18/03/09 18:54:17 INFO ResourceUtils: Unable to find 'resource-types.xml'.
  441. 18/03/09 18:54:17 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@608788f18ea1:45035)
  442. 18/03/09 18:54:17 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
  443. 18/03/09 18:54:17 INFO YarnAllocator: Submitted 2 unlocalized container requests.
  444. 18/03/09 18:54:17 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
  445. 18/03/09 18:54:18 INFO YarnAllocator: Launching container container_1520619475936_0001_02_000002 on host bcaf3c62a630 for executor with ID 1
  446. 18/03/09 18:54:18 INFO YarnAllocator: Launching container container_1520619475936_0001_02_000003 on host bcaf3c62a630 for executor with ID 2
  447. 18/03/09 18:54:18 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
  448. 18/03/09 18:54:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.22.0.3:33988) with ID 1
  449. 18/03/09 18:54:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.22.0.3:33990) with ID 2
  450. 18/03/09 18:54:26 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
  451. 18/03/09 18:54:26 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
  452. 18/03/09 18:54:26 INFO BlockManagerMasterEndpoint: Registering block manager bcaf3c62a630:36729 with 2.4 GB RAM, BlockManagerId(1, bcaf3c62a630, 36729, None)
  453. 18/03/09 18:54:26 INFO BlockManagerMasterEndpoint: Registering block manager bcaf3c62a630:38431 with 2.4 GB RAM, BlockManagerId(2, bcaf3c62a630, 38431, None)
  454. 18/03/09 18:54:31 INFO Nd4jBackend: Loaded [CpuBackend] backend
  455. 18/03/09 18:54:35 INFO NativeOpsHolder: Number of threads used for NativeOps: 1
  456. 18/03/09 18:54:37 ERROR ApplicationMaster: User class threw exception: java.lang.ExceptionInInitializerError
  457. java.lang.ExceptionInInitializerError
  458. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.setMaxThreads(CpuBlas.java:117)
  459. at org.nd4j.nativeblas.Nd4jBlas.<init>(Nd4jBlas.java:35)
  460. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.<init>(CpuBlas.java:12)
  461. at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createBlas(CpuNDArrayFactory.java:89)
  462. at org.nd4j.linalg.factory.BaseNDArrayFactory.blas(BaseNDArrayFactory.java:67)
  463. at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getEnvironmentInformation(NativeOpExecutioner.java:1176)
  464. at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.printEnvironmentInformation(DefaultOpExecutioner.java:562)
  465. at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:6210)
  466. at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6087)
  467. at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:201)
  468. at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:777)
  469. at org.deeplearning4j.icumortality.SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.entryPoint(SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.java:102)
  470. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.mainEntryPoint(SparkMainForLocalSparkClusterDeploy.java:67)
  471. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.main(SparkMainForLocalSparkClusterDeploy.java:23)
  472. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  473. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  474. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  475. at java.lang.reflect.Method.invoke(Method.java:498)
  476. at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
  477. Caused by: java.nio.channels.OverlappingFileLockException
  478. at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
  479. at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
  480. at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
  481. at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
  482. at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:429)
  483. at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:944)
  484. at org.bytedeco.javacpp.Loader.load(Loader.java:765)
  485. at org.bytedeco.javacpp.Loader.load(Loader.java:684)
  486. at org.bytedeco.javacpp.openblas.<clinit>(openblas.java:10)
  487. ... 19 more
  488. 18/03/09 18:54:37 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.ExceptionInInitializerError
  489. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.setMaxThreads(CpuBlas.java:117)
  490. at org.nd4j.nativeblas.Nd4jBlas.<init>(Nd4jBlas.java:35)
  491. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.<init>(CpuBlas.java:12)
  492. at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createBlas(CpuNDArrayFactory.java:89)
  493. at org.nd4j.linalg.factory.BaseNDArrayFactory.blas(BaseNDArrayFactory.java:67)
  494. at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getEnvironmentInformation(NativeOpExecutioner.java:1176)
  495. at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.printEnvironmentInformation(DefaultOpExecutioner.java:562)
  496. at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:6210)
  497. at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6087)
  498. at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:201)
  499. at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:777)
  500. at org.deeplearning4j.icumortality.SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.entryPoint(SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.java:102)
  501. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.mainEntryPoint(SparkMainForLocalSparkClusterDeploy.java:67)
  502. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.main(SparkMainForLocalSparkClusterDeploy.java:23)
  503. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  504. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  505. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  506. at java.lang.reflect.Method.invoke(Method.java:498)
  507. at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
  508. Caused by: java.nio.channels.OverlappingFileLockException
  509. at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
  510. at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
  511. at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
  512. at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
  513. at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:429)
  514. at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:944)
  515. at org.bytedeco.javacpp.Loader.load(Loader.java:765)
  516. at org.bytedeco.javacpp.Loader.load(Loader.java:684)
  517. at org.bytedeco.javacpp.openblas.<clinit>(openblas.java:10)
  518. ... 19 more
  519. )
  520. 18/03/09 18:54:37 INFO SparkContext: Invoking stop() from shutdown hook
  521. 18/03/09 18:54:37 INFO AbstractConnector: Stopped Spark@4de6e829{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
  522. 18/03/09 18:54:37 INFO SparkUI: Stopped Spark web UI at http://608788f18ea1:36441
  523. 18/03/09 18:54:37 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
  524. 18/03/09 18:54:37 INFO YarnClusterSchedulerBackend: Shutting down all executors
  525. 18/03/09 18:54:37 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
  526. 18/03/09 18:54:37 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
  527. (serviceOption=None,
  528. services=List(),
  529. started=false)
  530. 18/03/09 18:54:37 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
  531. 18/03/09 18:54:37 INFO MemoryStore: MemoryStore cleared
  532. 18/03/09 18:54:37 INFO BlockManager: BlockManager stopped
  533. 18/03/09 18:54:37 INFO BlockManagerMaster: BlockManagerMaster stopped
  534. 18/03/09 18:54:37 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  535. 18/03/09 18:54:37 INFO SparkContext: Successfully stopped SparkContext
  536. 18/03/09 18:54:37 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.ExceptionInInitializerError
  537. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.setMaxThreads(CpuBlas.java:117)
  538. at org.nd4j.nativeblas.Nd4jBlas.<init>(Nd4jBlas.java:35)
  539. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.<init>(CpuBlas.java:12)
  540. at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createBlas(CpuNDArrayFactory.java:89)
  541. at org.nd4j.linalg.factory.BaseNDArrayFactory.blas(BaseNDArrayFactory.java:67)
  542. at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getEnvironmentInformation(NativeOpExecutioner.java:1176)
  543. at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.printEnvironmentInformation(DefaultOpExecutioner.java:562)
  544. at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:6210)
  545. at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6087)
  546. at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:201)
  547. at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:777)
  548. at org.deeplearning4j.icumortality.SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.entryPoint(SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.java:102)
  549. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.mainEntryPoint(SparkMainForLocalSparkClusterDeploy.java:67)
  550. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.main(SparkMainForLocalSparkClusterDeploy.java:23)
  551. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  552. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  553. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  554. at java.lang.reflect.Method.invoke(Method.java:498)
  555. at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
  556. Caused by: java.nio.channels.OverlappingFileLockException
  557. at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
  558. at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
  559. at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
  560. at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
  561. at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:429)
  562. at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:944)
  563. at org.bytedeco.javacpp.Loader.load(Loader.java:765)
  564. at org.bytedeco.javacpp.Loader.load(Loader.java:684)
  565. at org.bytedeco.javacpp.openblas.<clinit>(openblas.java:10)
  566. ... 19 more
  567. )
  568. 18/03/09 18:54:37 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
  569. 18/03/09 18:54:37 INFO ApplicationMaster: Deleting staging directory hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001
  570. 18/03/09 18:54:37 INFO ShutdownHookManager: Shutdown hook called
  571. 18/03/09 18:54:37 INFO ShutdownHookManager: Deleting directory /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/spark-c760fb67-e468-471e-8dd6-66640b857960
  572.  
  573. End of LogType:stderr
  574. ***********************************************************************
  575.  
  576. Container: container_1520619475936_0001_02_000001 on 608788f18ea1_39033
  577. LogAggregationType: AGGREGATED
  578. =======================================================================
  579. LogType:stdout
  580. LogLastModifiedTime:Fri Mar 09 18:55:10 +0000 2018
  581. LogLength:2616
  582. LogContents:
  583. [GC (Allocation Failure) [PSYoungGen: 64512K->8861K(75264K)] 64512K->8877K(247296K), 0.0943325 secs] [Times: user=0.28 sys=0.01, real=0.10 secs]
  584. [GC (Allocation Failure) [PSYoungGen: 73373K->7465K(139776K)] 73389K->7489K(311808K), 0.0104129 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
  585. [GC (Metadata GC Threshold) [PSYoungGen: 81722K->8279K(139776K)] 81746K->8311K(311808K), 0.0530689 secs] [Times: user=0.12 sys=0.00, real=0.05 secs]
  586. [Full GC (Metadata GC Threshold) [PSYoungGen: 8279K->0K(139776K)] [ParOldGen: 32K->7942K(86016K)] 8311K->7942K(225792K), [Metaspace: 20897K->20897K(1067008K)], 0.0382324 secs] [Times: user=0.14 sys=0.01, real=0.04 secs]
  587. [GC (Allocation Failure) [PSYoungGen: 129024K->4930K(231936K)] 136966K->12872K(317952K), 0.0115700 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
  588. [GC (Metadata GC Threshold) [PSYoungGen: 152329K->10735K(263168K)] 160272K->19275K(349184K), 0.2404408 secs] [Times: user=0.49 sys=0.05, real=0.24 secs]
  589. [Full GC (Metadata GC Threshold) [PSYoungGen: 10735K->0K(263168K)] [ParOldGen: 8540K->14694K(121856K)] 19275K->14694K(385024K), [Metaspace: 35148K->35148K(1079296K)], 0.1879988 secs] [Times: user=0.73 sys=0.05, real=0.19 secs]
  590. [GC (Allocation Failure) [PSYoungGen: 252416K->12268K(416256K)] 267110K->30830K(538112K), 0.2162021 secs] [Times: user=0.36 sys=0.10, real=0.22 secs]
  591. [GC (Allocation Failure) [PSYoungGen: 416236K->16378K(420352K)] 434798K->136356K(542208K), 1.0469694 secs] [Times: user=1.86 sys=3.20, real=1.04 secs]
  592. [Full GC (Ergonomics) [PSYoungGen: 16378K->3061K(420352K)] [ParOldGen: 119977K->121768K(300544K)] 136356K->124830K(720896K), [Metaspace: 52288K->52287K(1095680K)], 0.4157283 secs] [Times: user=1.60 sys=0.18, real=0.42 secs]
  593. [GC (Allocation Failure) [PSYoungGen: 407029K->69600K(723968K)] 528798K->211900K(1024512K), 0.6901218 secs] [Times: user=1.03 sys=2.55, real=0.69 secs]
  594. Heap
  595. PSYoungGen total 723968K, used 448666K [0x0000000755580000, 0x000000078cf00000, 0x00000007c0000000)
  596. eden space 654336K, 57% used [0x0000000755580000,0x000000076c7ae8c8,0x000000077d480000)
  597. from space 69632K, 99% used [0x0000000782f00000,0x00000007872f8010,0x0000000787300000)
  598. to space 92672K, 0% used [0x000000077d480000,0x000000077d480000,0x0000000782f00000)
  599. ParOldGen total 300544K, used 142300K [0x0000000680000000, 0x0000000692580000, 0x0000000755580000)
  600. object space 300544K, 47% used [0x0000000680000000,0x0000000688af7368,0x0000000692580000)
  601. Metaspace used 58723K, capacity 59148K, committed 59432K, reserved 1101824K
  602. class space used 7519K, capacity 7637K, committed 7720K, reserved 1048576K
  603.  
  604. End of LogType:stdout
  605. ***********************************************************************
  606.  
  607. Container: container_1520619475936_0001_01_000001 on bcaf3c62a630_41471
  608. LogAggregationType: AGGREGATED
  609. =======================================================================
  610. LogType:directory.info
  611. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  612. LogLength:21689
  613. LogContents:
  614. ls -l:
  615. total 36
  616. lrwxrwxrwx 1 hduser hadoop 89 Mar 9 18:53 __app__.jar -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar
  617. lrwxrwxrwx 1 hduser hadoop 80 Mar 9 18:53 __spark_conf__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip
  618. lrwxrwxrwx 1 hduser hadoop 99 Mar 9 18:53 __spark_libs__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip
  619. -rw-r--r-- 1 hduser hadoop 69 Mar 9 18:53 container_tokens
  620. -rwx------ 1 hduser hadoop 733 Mar 9 18:53 default_container_executor.sh
  621. -rwx------ 1 hduser hadoop 678 Mar 9 18:53 default_container_executor_session.sh
  622. -rwx------ 1 hduser hadoop 5896 Mar 9 18:53 launch_container.sh
  623. drwx--x--- 2 hduser hadoop 4096 Mar 9 18:53 tmp
  624. find -L . -maxdepth 5 -ls:
  625. 1660803 4 drwx--x--- 3 hduser hadoop 4096 Mar 9 18:53 .
  626. 7089355 4 -rwx------ 1 hduser hadoop 733 Mar 9 18:53 ./default_container_executor.sh
  627. 7089351 8 -rwx------ 1 hduser hadoop 5896 Mar 9 18:53 ./launch_container.sh
  628. 1660805 4 drwx--x--- 2 hduser hadoop 4096 Mar 9 18:53 ./tmp
  629. 7089162 517916 -r-x------ 1 hduser hadoop 530343861 Mar 9 18:53 ./__app__.jar
  630. 7089356 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:53 ./.default_container_executor.sh.crc
  631. 1656886 12 drwx------ 2 hduser hadoop 12288 Mar 9 18:52 ./__spark_libs__
  632. 7089166 88 -r-x------ 1 hduser hadoop 87192 Mar 9 18:52 ./__spark_libs__/spire-macros_2.11-0.13.0.jar
  633. 7089167 5612 -r-x------ 1 hduser hadoop 5744974 Mar 9 18:52 ./__spark_libs__/scala-library-2.11.8.jar
  634. 7089168 20 -r-x------ 1 hduser hadoop 19827 Mar 9 18:52 ./__spark_libs__/opencsv-2.3.jar
  635. 7089169 16 -r-x------ 1 hduser hadoop 13911 Mar 9 18:52 ./__spark_libs__/generex-1.0.1.jar
  636. 7089170 364 -r-x------ 1 hduser hadoop 370119 Mar 9 18:52 ./__spark_libs__/lz4-java-1.4.0.jar
  637. 7089171 256 -r-x------ 1 hduser hadoop 258919 Mar 9 18:52 ./__spark_libs__/jackson-core-2.6.7.jar
  638. 7089172 684 -r-x------ 1 hduser hadoop 698375 Mar 9 18:52 ./__spark_libs__/jersey-common-2.22.2.jar
  639. 7089173 472 -r-x------ 1 hduser hadoop 479881 Mar 9 18:52 ./__spark_libs__/commons-lang3-3.5.jar
  640. 7089174 6852 -r-x------ 1 hduser hadoop 7015233 Mar 9 18:52 ./__spark_libs__/kubernetes-model-2.0.0.jar
  641. 7089175 176 -r-x------ 1 hduser hadoop 178947 Mar 9 18:52 ./__spark_libs__/hk2-api-2.4.0-b34.jar
  642. 7089176 7172 -r-x------ 1 hduser hadoop 7343426 Mar 9 18:52 ./__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
  643. 7089177 504 -r-x------ 1 hduser hadoop 515645 Mar 9 18:52 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
  644. 7089178 44 -r-x------ 1 hduser hadoop 42858 Mar 9 18:52 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar
  645. 7089179 60 -r-x------ 1 hduser hadoop 60483 Mar 9 18:52 ./__spark_libs__/chill-java-0.8.4.jar
  646. 7089180 132 -r-x------ 1 hduser hadoop 134595 Mar 9 18:52 ./__spark_libs__/commons-crypto-1.0.0.jar
  647. 7089181 128 -r-x------ 1 hduser hadoop 130802 Mar 9 18:52 ./__spark_libs__/aircompressor-0.8.jar
  648. 7089182 264 -r-x------ 1 hduser hadoop 269295 Mar 9 18:52 ./__spark_libs__/snakeyaml-1.15.jar
  649. 7089183 1168 -r-x------ 1 hduser hadoop 1194003 Mar 9 18:52 ./__spark_libs__/arpack_combined_all-0.1.jar
  650. 7089184 14760 -r-x------ 1 hduser hadoop 15113382 Mar 9 18:52 ./__spark_libs__/breeze_2.11-0.13.2.jar
  651. 7089185 36 -r-x------ 1 hduser hadoop 35518 Mar 9 18:52 ./__spark_libs__/zjsonpatch-0.3.0.jar
  652. 7089186 120 -r-x------ 1 hduser hadoop 120321 Mar 9 18:52 ./__spark_libs__/spark-repl_2.11-2.3.0.jar
  653. 7089187 36 -r-x------ 1 hduser hadoop 34603 Mar 9 18:52 ./__spark_libs__/machinist_2.11-0.6.1.jar
  654. 7089188 28 -r-x------ 1 hduser hadoop 26366 Mar 9 18:52 ./__spark_libs__/javax.annotation-api-1.2.jar
  655. 7089189 180 -r-x------ 1 hduser hadoop 180736 Mar 9 18:52 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
  656. 7089190 788 -r-x------ 1 hduser hadoop 802818 Mar 9 18:52 ./__spark_libs__/scalap-2.11.8.jar
  657. 7089191 1504 -r-x------ 1 hduser hadoop 1537966 Mar 9 18:52 ./__spark_libs__/hadoop-mapreduce-client-core-2.6.5.jar
  658. 7089192 1024 -r-x------ 1 hduser hadoop 1048116 Mar 9 18:52 ./__spark_libs__/parquet-jackson-1.8.2.jar
  659. 7089193 172 -r-x------ 1 hduser hadoop 174351 Mar 9 18:52 ./__spark_libs__/stream-2.7.0.jar
  660. 7089194 68 -r-x------ 1 hduser hadoop 66270 Mar 9 18:52 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar
  661. 7089195 1632 -r-x------ 1 hduser hadoop 1671083 Mar 9 18:52 ./__spark_libs__/hppc-0.7.2.jar
  662. 7089196 180 -r-x------ 1 hduser hadoop 183832 Mar 9 18:52 ./__spark_libs__/spark-mllib-local_2.11-2.3.0.jar
  663. 7089197 656 -r-x------ 1 hduser hadoop 671138 Mar 9 18:52 ./__spark_libs__/scala-xml_2.11-1.0.5.jar
  664. 7089198 200 -r-x------ 1 hduser hadoop 201928 Mar 9 18:52 ./__spark_libs__/RoaringBitmap-0.5.11.jar
  665. 7089199 96 -r-x------ 1 hduser hadoop 94796 Mar 9 18:52 ./__spark_libs__/pyrolite-4.13.jar
  666. 7089200 3692 -r-x------ 1 hduser hadoop 3780056 Mar 9 18:52 ./__spark_libs__/netty-all-4.1.17.Final.jar
  667. 7089201 316 -r-x------ 1 hduser hadoop 320444 Mar 9 18:52 ./__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
  668. 7089202 1024 -r-x------ 1 hduser hadoop 1045744 Mar 9 18:52 ./__spark_libs__/leveldbjni-all-1.8.jar
  669. 7089203 376 -r-x------ 1 hduser hadoop 381098 Mar 9 18:52 ./__spark_libs__/spark-kubernetes_2.11-2.3.0.jar
  670. 7089204 32 -r-x------ 1 hduser hadoop 32612 Mar 9 18:52 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
  671. 7089205 636 -r-x------ 1 hduser hadoop 650236 Mar 9 18:52 ./__spark_libs__/spark-yarn_2.11-2.3.0.jar
  672. 7089206 48 -r-x------ 1 hduser hadoop 46986 Mar 9 18:52 ./__spark_libs__/jackson-annotations-2.6.7.jar
  673. 7089207 700 -r-x------ 1 hduser hadoop 714194 Mar 9 18:52 ./__spark_libs__/javassist-3.18.1-GA.jar
  674. 7089208 248 -r-x------ 1 hduser hadoop 250377 Mar 9 18:52 ./__spark_libs__/parquet-hadoop-1.8.2.jar
  675. 7089209 64 -r-x------ 1 hduser hadoop 64847 Mar 9 18:52 ./__spark_libs__/spark-network-shuffle_2.11-2.3.0.jar
  676. 7089210 116 -r-x------ 1 hduser hadoop 116859 Mar 9 18:52 ./__spark_libs__/py4j-0.10.6.jar
  677. 7089211 76 -r-x------ 1 hduser hadoop 75835 Mar 9 18:52 ./__spark_libs__/spark-launcher_2.11-2.3.0.jar
  678. 7089212 20 -r-x------ 1 hduser hadoop 16430 Mar 9 18:52 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar
  679. 7089213 52 -r-x------ 1 hduser hadoop 52076 Mar 9 18:52 ./__spark_libs__/arrow-format-0.8.0.jar
  680. 7089214 84 -r-x------ 1 hduser hadoop 82421 Mar 9 18:52 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar
  681. 7089215 740 -r-x------ 1 hduser hadoop 757423 Mar 9 18:52 ./__spark_libs__/orc-mapreduce-1.4.1-nohive.jar
  682. 7089216 1300 -r-x------ 1 hduser hadoop 1330219 Mar 9 18:52 ./__spark_libs__/netty-3.9.9.Final.jar
  683. 7089217 120 -r-x------ 1 hduser hadoop 118973 Mar 9 18:52 ./__spark_libs__/hk2-utils-2.4.0-b34.jar
  684. 7089218 408 -r-x------ 1 hduser hadoop 416731 Mar 9 18:52 ./__spark_libs__/kubernetes-client-3.0.0.jar
  685. 7089219 96 -r-x------ 1 hduser hadoop 95806 Mar 9 18:52 ./__spark_libs__/javax.servlet-api-3.1.0.jar
  686. 7089220 936 -r-x------ 1 hduser hadoop 956984 Mar 9 18:52 ./__spark_libs__/parquet-column-1.8.2.jar
  687. 7089221 1412 -r-x------ 1 hduser hadoop 1441994 Mar 9 18:52 ./__spark_libs__/orc-core-1.4.1-nohive.jar
  688. 7089222 576 -r-x------ 1 hduser hadoop 589462 Mar 9 18:52 ./__spark_libs__/json4s-core_2.11-3.2.11.jar
  689. 7089223 2328 -r-x------ 1 hduser hadoop 2382165 Mar 9 18:52 ./__spark_libs__/spark-network-common_2.11-2.3.0.jar
  690. 7089224 12 -r-x------ 1 hduser hadoop 8268 Mar 9 18:52 ./__spark_libs__/logging-interceptor-3.8.1.jar
  691. 7089225 40 -r-x------ 1 hduser hadoop 40341 Mar 9 18:52 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar
  692. 7089226 52 -r-x------ 1 hduser hadoop 53007 Mar 9 18:52 ./__spark_libs__/spark-kvstore_2.11-2.3.0.jar
  693. 7089227 8 -r-x------ 1 hduser hadoop 5950 Mar 9 18:52 ./__spark_libs__/javax.inject-2.4.0-b34.jar
  694. 7089228 376 -r-x------ 1 hduser hadoop 384150 Mar 9 18:52 ./__spark_libs__/univocity-parsers-2.5.9.jar
  695. 7089229 7560 -r-x------ 1 hduser hadoop 7738527 Mar 9 18:52 ./__spark_libs__/spark-mllib_2.11-2.3.0.jar
  696. 7089230 116 -r-x------ 1 hduser hadoop 115534 Mar 9 18:52 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar
  697. 7089231 72 -r-x------ 1 hduser hadoop 72733 Mar 9 18:52 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar
  698. 7089232 748 -r-x------ 1 hduser hadoop 764569 Mar 9 18:52 ./__spark_libs__/jtransforms-2.4.0.jar
  699. 7089233 12 -r-x------ 1 hduser hadoop 10166 Mar 9 18:52 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
  700. 7089234 36 -r-x------ 1 hduser hadoop 34654 Mar 9 18:52 ./__spark_libs__/paranamer-2.8.jar
  701. 7089235 8464 -r-x------ 1 hduser hadoop 8664987 Mar 9 18:52 ./__spark_libs__/spark-sql_2.11-2.3.0.jar
  702. 7089236 164 -r-x------ 1 hduser hadoop 164422 Mar 9 18:52 ./__spark_libs__/core-1.1.2.jar
  703. 7089237 44 -r-x------ 1 hduser hadoop 42032 Mar 9 18:52 ./__spark_libs__/parquet-common-1.8.2.jar
  704. 7089238 36 -r-x------ 1 hduser hadoop 33015 Mar 9 18:52 ./__spark_libs__/jsr305-1.3.9.jar
  705. 7089239 80 -r-x------ 1 hduser hadoop 81811 Mar 9 18:52 ./__spark_libs__/okio-1.13.0.jar
  706. 7089240 328 -r-x------ 1 hduser hadoop 334662 Mar 9 18:52 ./__spark_libs__/antlr4-runtime-4.7.jar
  707. 7089241 392 -r-x------ 1 hduser hadoop 398122 Mar 9 18:52 ./__spark_libs__/okhttp-3.8.1.jar
  708. 7089242 352 -r-x------ 1 hduser hadoop 358390 Mar 9 18:52 ./__spark_libs__/kryo-shaded-3.0.3.jar
  709. 7089243 80 -r-x------ 1 hduser hadoop 79150 Mar 9 18:52 ./__spark_libs__/arrow-memory-0.8.0.jar
  710. 7089244 15128 -r-x------ 1 hduser hadoop 15487351 Mar 9 18:52 ./__spark_libs__/scala-compiler-2.11.8.jar
  711. 7089245 164 -r-x------ 1 hduser hadoop 167421 Mar 9 18:52 ./__spark_libs__/jersey-client-2.22.2.jar
  712. 7089246 384 -r-x------ 1 hduser hadoop 390733 Mar 9 18:52 ./__spark_libs__/parquet-format-2.3.1.jar
  713. 7089247 64 -r-x------ 1 hduser hadoop 63777 Mar 9 18:52 ./__spark_libs__/validation-api-1.1.0.Final.jar
  714. 7089248 4468 -r-x------ 1 hduser hadoop 4573750 Mar 9 18:52 ./__spark_libs__/scala-reflect-2.11.8.jar
  715. 7089249 932 -r-x------ 1 hduser hadoop 951701 Mar 9 18:52 ./__spark_libs__/jersey-server-2.22.2.jar
  716. 7089250 48 -r-x------ 1 hduser hadoop 48464 Mar 9 18:52 ./__spark_libs__/spark-unsafe_2.11-2.3.0.jar
  717. 7089251 184 -r-x------ 1 hduser hadoop 187882 Mar 9 18:52 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar
  718. 7089252 3444 -r-x------ 1 hduser hadoop 3522616 Mar 9 18:52 ./__spark_libs__/shapeless_2.11-2.3.2.jar
  719. 7089253 8 -r-x------ 1 hduser hadoop 5711 Mar 9 18:52 ./__spark_libs__/minlog-1.3.0.jar
  720. 7089254 20 -r-x------ 1 hduser hadoop 18098 Mar 9 18:52 ./__spark_libs__/jersey-container-servlet-2.22.2.jar
  721. 7089255 40 -r-x------ 1 hduser hadoop 37887 Mar 9 18:52 ./__spark_libs__/commons-compiler-3.0.8.jar
  722. 7089256 44 -r-x------ 1 hduser hadoop 41755 Mar 9 18:52 ./__spark_libs__/objenesis-2.1.jar
  723. 7089257 280 -r-x------ 1 hduser hadoop 286071 Mar 9 18:52 ./__spark_libs__/parquet-encoding-1.8.2.jar
  724. 7089258 220 -r-x------ 1 hduser hadoop 224167 Mar 9 18:52 ./__spark_libs__/chill_2.11-0.8.4.jar
  725. 7089259 780 -r-x------ 1 hduser hadoop 796326 Mar 9 18:52 ./__spark_libs__/janino-3.0.8.jar
  726. 7089260 1244 -r-x------ 1 hduser hadoop 1270377 Mar 9 18:52 ./__spark_libs__/arrow-vector-0.8.0.jar
  727. 7089261 24 -r-x------ 1 hduser hadoop 21247 Mar 9 18:52 ./__spark_libs__/metrics-graphite-3.1.5.jar
  728. 7089262 1988 -r-x------ 1 hduser hadoop 2035066 Mar 9 18:52 ./__spark_libs__/commons-math3-3.4.1.jar
  729. 7089263 32 -r-x------ 1 hduser hadoop 30057 Mar 9 18:52 ./__spark_libs__/spark-sketch_2.11-2.3.0.jar
  730. 7089264 2120 -r-x------ 1 hduser hadoop 2169825 Mar 9 18:52 ./__spark_libs__/spark-streaming_2.11-2.3.0.jar
  731. 7089265 9888 -r-x------ 1 hduser hadoop 10121868 Mar 9 18:52 ./__spark_libs__/spire_2.11-0.13.0.jar
  732. 7089266 8 -r-x------ 1 hduser hadoop 4596 Mar 9 18:52 ./__spark_libs__/jul-to-slf4j-1.7.16.jar
  733. 7089267 20 -r-x------ 1 hduser hadoop 20235 Mar 9 18:52 ./__spark_libs__/osgi-resource-locator-1.0.1.jar
  734. 7089268 4 -r-x------ 1 hduser hadoop 3142 Mar 9 18:52 ./__spark_libs__/macro-compat_2.11-1.1.1.jar
  735. 7089269 280 -r-x------ 1 hduser hadoop 284220 Mar 9 18:52 ./__spark_libs__/commons-lang-2.6.jar
  736. 7089270 1256 -r-x------ 1 hduser hadoop 1282424 Mar 9 18:52 ./__spark_libs__/ivy-2.4.0.jar
  737. 7089271 80 -r-x------ 1 hduser hadoop 79845 Mar 9 18:52 ./__spark_libs__/compress-lzf-1.0.3.jar
  738. 7089272 16 -r-x------ 1 hduser hadoop 15824 Mar 9 18:52 ./__spark_libs__/metrics-json-3.1.5.jar
  739. 7089273 12680 -r-x------ 1 hduser hadoop 12982716 Mar 9 18:52 ./__spark_libs__/spark-core_2.11-2.3.0.jar
  740. 7089274 180 -r-x------ 1 hduser hadoop 181271 Mar 9 18:52 ./__spark_libs__/hk2-locator-2.4.0-b34.jar
  741. 7089275 416 -r-x------ 1 hduser hadoop 423753 Mar 9 18:52 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
  742. 7089276 1140 -r-x------ 1 hduser hadoop 1165323 Mar 9 18:52 ./__spark_libs__/jackson-databind-2.6.7.1.jar
  743. 7089277 208 -r-x------ 1 hduser hadoop 212453 Mar 9 18:52 ./__spark_libs__/commons-net-2.2.jar
  744. 7089278 64 -r-x------ 1 hduser hadoop 65261 Mar 9 18:52 ./__spark_libs__/oro-2.0.8.jar
  745. 7089279 144 -r-x------ 1 hduser hadoop 144660 Mar 9 18:52 ./__spark_libs__/xbean-asm5-shaded-4.4.jar
  746. 7089280 8776 -r-x------ 1 hduser hadoop 8984422 Mar 9 18:52 ./__spark_libs__/spark-catalyst_2.11-2.3.0.jar
  747. 7089281 40 -r-x------ 1 hduser hadoop 39283 Mar 9 18:52 ./__spark_libs__/metrics-jvm-3.1.5.jar
  748. 7089282 280 -r-x------ 1 hduser hadoop 284184 Mar 9 18:52 ./__spark_libs__/commons-codec-1.10.jar
  749. 7089283 176 -r-x------ 1 hduser hadoop 176285 Mar 9 18:52 ./__spark_libs__/automaton-1.11-8.jar
  750. 7089284 2280 -r-x------ 1 hduser hadoop 2333186 Mar 9 18:52 ./__spark_libs__/zstd-jni-1.3.2-2.jar
  751. 7089285 648 -r-x------ 1 hduser hadoop 663328 Mar 9 18:52 ./__spark_libs__/spark-mesos_2.11-2.3.0.jar
  752. 7089286 696 -r-x------ 1 hduser hadoop 708667 Mar 9 18:52 ./__spark_libs__/spark-graphx_2.11-2.3.0.jar
  753. 7089287 16 -r-x------ 1 hduser hadoop 15456 Mar 9 18:52 ./__spark_libs__/spark-tags_2.11-2.3.0.jar
  754. 7089288 616 -r-x------ 1 hduser hadoop 627814 Mar 9 18:52 ./__spark_libs__/joda-time-2.9.3.jar
  755. 7089289 120 -r-x------ 1 hduser hadoop 120465 Mar 9 18:52 ./__spark_libs__/metrics-core-3.1.5.jar
  756. 7089290 16 -r-x------ 1 hduser hadoop 14766 Mar 9 18:52 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
  757. 7089291 952 -r-x------ 1 hduser hadoop 971310 Mar 9 18:52 ./__spark_libs__/jersey-guava-2.22.2.jar
  758. 7089352 4 -rw-r--r-- 1 hduser hadoop 56 Mar 9 18:53 ./.launch_container.sh.crc
  759. 7089349 4 -rw-r--r-- 1 hduser hadoop 69 Mar 9 18:53 ./container_tokens
  760. 7089354 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:53 ./.default_container_executor_session.sh.crc
  761. 7089353 4 -rwx------ 1 hduser hadoop 678 Mar 9 18:53 ./default_container_executor_session.sh
  762. 1653456 4 drwx------ 3 hduser hadoop 4096 Mar 9 18:52 ./__spark_conf__
  763. 7089340 4 -r-x------ 1 hduser hadoop 1192 Mar 9 18:52 ./__spark_conf__/__spark_conf__.properties
  764. 1653459 4 drwx------ 2 hduser hadoop 4096 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__
  765. 7089323 4 -r-x------ 1 hduser hadoop 2316 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example
  766. 7089327 4 -r-x------ 1 hduser hadoop 891 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/slaves
  767. 7089338 4 -r-x------ 1 hduser hadoop 682 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-site.xml
  768. 7089332 4 -r-x------ 1 hduser hadoop 996 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/docker.properties.template
  769. 7089319 4 -r-x------ 1 hduser hadoop 1764 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-env.sh
  770. 7089312 4 -r-x------ 1 hduser hadoop 3518 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-acls.xml
  771. 7089337 8 -r-x------ 1 hduser hadoop 4113 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template
  772. 7089329 4 -r-x------ 1 hduser hadoop 21 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-signature.secret
  773. 7089328 4 -r-x------ 1 hduser hadoop 1484 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-env.sh
  774. 7089314 4 -r-x------ 1 hduser hadoop 482 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-env.sh
  775. 7089303 4 -r-x------ 1 hduser hadoop 3670 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd
  776. 7089299 4 -r-x------ 1 hduser hadoop 2016 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.spark.properties
  777. 7089308 8 -r-x------ 1 hduser hadoop 4221 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-env.sh.template
  778. 7089298 4 -r-x------ 1 hduser hadoop 137 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
  779. 7089322 4 -r-x------ 1 hduser hadoop 3414 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-user-functions.sh.example
  780. 7089310 4 -r-x------ 1 hduser hadoop 2250 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd
  781. 7089317 4 -r-x------ 1 hduser hadoop 589 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/container-executor.cfg
  782. 7089307 4 -r-x------ 1 hduser hadoop 890 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/workers
  783. 7089336 4 -r-x------ 1 hduser hadoop 2697 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example
  784. 7089300 4 -r-x------ 1 hduser hadoop 1657 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-log4j.properties
  785. 7089320 8 -r-x------ 1 hduser hadoop 7801 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/metrics.properties.template
  786. 7089318 4 -r-x------ 1 hduser hadoop 1781 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf
  787. 7089306 4 -r-x------ 1 hduser hadoop 225 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/core-site.xml
  788. 7089301 4 -r-x------ 1 hduser hadoop 3323 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties
  789. 7089311 8 -r-x------ 1 hduser hadoop 7861 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml
  790. 7089334 4 -r-x------ 1 hduser hadoop 1292 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf.template
  791. 7089321 4 -r-x------ 1 hduser hadoop 2642 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/user_ec_policies.xml.template
  792. 7089333 4 -r-x------ 1 hduser hadoop 951 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd
  793. 7089330 4 -r-x------ 1 hduser hadoop 620 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-site.xml
  794. 7089335 4 -r-x------ 1 hduser hadoop 3589 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j2.xml
  795. 7089325 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.properties.template
  796. 7089305 4 -r-x------ 1 hduser hadoop 865 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/slaves.template
  797. 7089316 4 -r-x------ 1 hduser hadoop 809 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
  798. 7089297 16 -r-x------ 1 hduser hadoop 16270 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh
  799. 7089313 4 -r-x------ 1 hduser hadoop 1335 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/configuration.xsl
  800. 7089302 8 -r-x------ 1 hduser hadoop 8048 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
  801. 7089296 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.properties
  802. 7089324 4 -r-x------ 1 hduser hadoop 1351 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-env.sh
  803. 7089309 4 -r-x------ 1 hduser hadoop 1747 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties
  804. 7089326 12 -r-x------ 1 hduser hadoop 10206 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml
  805. 7089331 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/metrics.properties
  806. 7089315 8 -r-x------ 1 hduser hadoop 5502 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-env.sh
  807. 7089304 4 -r-x------ 1 hduser hadoop 1105 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.template
  808. 7089293 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/log4j.properties
  809. 7089339 172 -r-x------ 1 hduser hadoop 173257 Mar 9 18:52 ./__spark_conf__/__spark_hadoop_conf__.xml
  810. 7089294 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:52 ./__spark_conf__/metrics.properties
  811. 7089350 4 -rw-r--r-- 1 hduser hadoop 12 Mar 9 18:53 ./.container_tokens.crc
  812. broken symlinks(find -L . -maxdepth 5 -type l -ls):
  813.  
  814. End of LogType:directory.info
  815. *******************************************************************************
  816.  
  817. Container: container_1520619475936_0001_01_000001 on bcaf3c62a630_41471
  818. LogAggregationType: AGGREGATED
  819. =======================================================================
  820. LogType:launch_container.sh
  821. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  822. LogLength:5896
  823. LogContents:
  824. #!/bin/bash
  825.  
  826. set -o pipefail -e
  827. export PRELAUNCH_OUT="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/prelaunch.out"
  828. exec >"${PRELAUNCH_OUT}"
  829. export PRELAUNCH_ERR="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/prelaunch.err"
  830. exec 2>"${PRELAUNCH_ERR}"
  831. echo "Setting up env variables"
  832. export SPARK_YARN_STAGING_DIR="hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001"
  833. export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
  834. export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
  835. export JAVA_HOME="/usr/java/default"
  836. export APP_SUBMIT_TIME_ENV="1520621561751"
  837. export NM_HOST="bcaf3c62a630"
  838. export HADOOP_HDFS_HOME="/usr/local/hadoop"
  839. export LOGNAME="hduser"
  840. export JVM_PID="$$"
  841. export PWD="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_01_000001"
  842. export HADOOP_COMMON_HOME="/usr/local/hadoop"
  843. export PYTHONHASHSEED="0"
  844. export LOCAL_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001"
  845. export APPLICATION_WEB_PROXY_BASE="/proxy/application_1520619475936_0001"
  846. export NM_HTTP_PORT="18042"
  847. export SPARK_DIST_CLASSPATH="/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*"
  848. export LOG_DIRS="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001"
  849. export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  850. "
  851. export NM_PORT="41471"
  852. export USER="hduser"
  853. export HADOOP_YARN_HOME="/usr/local/hadoop"
  854. export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$PWD/__spark_conf__/__hadoop_conf__"
  855. export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_01_000001/container_tokens"
  856. export SPARK_USER="hduser"
  857. export LOCAL_USER_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/"
  858. export HADOOP_HOME="/usr/local/hadoop"
  859. export HOME="/home/"
  860. export CONTAINER_ID="container_1520619475936_0001_01_000001"
  861. export MALLOC_ARENA_MAX="4"
  862. echo "Setting up job resources"
  863. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip" "__spark_libs__"
  864. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar" "__app__.jar"
  865. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip" "__spark_conf__"
  866. echo "Copying debugging information"
  867. # Creating copy of launch script
  868. cp "launch_container.sh" "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/launch_container.sh"
  869. chmod 640 "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/launch_container.sh"
  870. # Determining directory contents
  871. echo "ls -l:" 1>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/directory.info"
  872. ls -l 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/directory.info"
  873. echo "find -L . -maxdepth 5 -ls:" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/directory.info"
  874. find -L . -maxdepth 5 -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/directory.info"
  875. echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/directory.info"
  876. find -L . -maxdepth 5 -type l -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/directory.info"
  877. echo "Launching container"
  878. exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx5120m -Djava.io.tmpdir=$PWD/tmp '-XX:+PrintGCDetails' '-Dlog4j.configuration=file:/usr/local/hadoop/etc/hadoop/log4j.spark.properties' -Dspark.yarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy' --jar file:/home/hduser/icu-mortality-1.0.0-all.jar --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/stdout 2> /usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000001/stderr"
  879.  
  880. End of LogType:launch_container.sh
  881. ************************************************************************************
  882.  
  883.  
  884. End of LogType:prelaunch.err
  885. ******************************************************************************
  886.  
  887. Container: container_1520619475936_0001_01_000001 on bcaf3c62a630_41471
  888. LogAggregationType: AGGREGATED
  889. =======================================================================
  890. LogType:prelaunch.out
  891. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  892. LogLength:100
  893. LogContents:
  894. Setting up env variables
  895. Setting up job resources
  896. Copying debugging information
  897. Launching container
  898.  
  899. End of LogType:prelaunch.out
  900. ******************************************************************************
  901.  
  902. Container: container_1520619475936_0001_01_000001 on bcaf3c62a630_41471
  903. LogAggregationType: AGGREGATED
  904. =======================================================================
  905. LogType:stderr
  906. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  907. LogLength:21499
  908. LogContents:
  909. log4j:WARN custom level class [ Define the root logger] not found.
  910. 18/03/09 18:53:09 INFO SignalUtils: Registered signal handler for TERM
  911. 18/03/09 18:53:09 INFO SignalUtils: Registered signal handler for HUP
  912. 18/03/09 18:53:09 INFO SignalUtils: Registered signal handler for INT
  913. 18/03/09 18:53:09 INFO SecurityManager: Changing view acls to: hduser
  914. 18/03/09 18:53:09 INFO SecurityManager: Changing modify acls to: hduser
  915. 18/03/09 18:53:09 INFO SecurityManager: Changing view acls groups to:
  916. 18/03/09 18:53:09 INFO SecurityManager: Changing modify acls groups to:
  917. 18/03/09 18:53:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  918. 18/03/09 18:53:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  919. 18/03/09 18:53:10 INFO ApplicationMaster: Preparing Local resources
  920. 18/03/09 18:53:11 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1520619475936_0001_000001
  921. 18/03/09 18:53:12 INFO ApplicationMaster: Starting the user application in a separate Thread
  922. 18/03/09 18:53:12 INFO ApplicationMaster: Waiting for spark context initialization...
  923. ERROR StatusLogger Unrecognized format specifier [d]
  924. ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
  925. ERROR StatusLogger Unrecognized format specifier [thread]
  926. ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
  927. ERROR StatusLogger Unrecognized format specifier [level]
  928. ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
  929. ERROR StatusLogger Unrecognized format specifier [logger]
  930. ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
  931. ERROR StatusLogger Unrecognized format specifier [msg]
  932. ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
  933. ERROR StatusLogger Unrecognized format specifier [n]
  934. ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
  935. ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
  936. ERROR StatusLogger Unrecognized format specifier [d]
  937. ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
  938. ERROR StatusLogger Unrecognized format specifier [thread]
  939. ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
  940. ERROR StatusLogger Unrecognized format specifier [level]
  941. ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
  942. ERROR StatusLogger Unrecognized format specifier [logger]
  943. ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
  944. ERROR StatusLogger Unrecognized format specifier [msg]
  945. ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
  946. ERROR StatusLogger Unrecognized format specifier [n]
  947. ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
  948. 18/03/09 18:53:14 INFO SparkContext: Running Spark version 2.3.0
  949. 18/03/09 18:53:15 WARN SparkConf: spark.master yarn-cluster is deprecated in Spark 2.0+, please instead use "yarn" with specified deploy mode.
  950. 18/03/09 18:53:15 INFO SparkContext: Submitted application: ICU Mortality Predictor
  951. 18/03/09 18:53:15 INFO SecurityManager: Changing view acls to: hduser
  952. 18/03/09 18:53:15 INFO SecurityManager: Changing modify acls to: hduser
  953. 18/03/09 18:53:15 INFO SecurityManager: Changing view acls groups to:
  954. 18/03/09 18:53:15 INFO SecurityManager: Changing modify acls groups to:
  955. 18/03/09 18:53:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  956. 18/03/09 18:53:16 INFO SparkEnv: Registering MapOutputTracker
  957. 18/03/09 18:53:16 INFO SparkEnv: Registering BlockManagerMaster
  958. 18/03/09 18:53:16 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  959. 18/03/09 18:53:16 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  960. 18/03/09 18:53:16 INFO DiskBlockManager: Created local directory at /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/blockmgr-88ce1f6a-5340-496f-96dc-804649ef6334
  961. 18/03/09 18:53:16 INFO MemoryStore: MemoryStore started with capacity 4.5 GB
  962. 18/03/09 18:53:16 INFO SparkEnv: Registering OutputCommitCoordinator
  963. 18/03/09 18:53:16 INFO log: Logging initialized @8949ms
  964. 18/03/09 18:53:16 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
  965. 18/03/09 18:53:16 INFO Server: jetty-9.3.z-SNAPSHOT
  966. 18/03/09 18:53:16 INFO Server: Started @9084ms
  967. 18/03/09 18:53:16 INFO AbstractConnector: Started ServerConnector@4c972453{HTTP/1.1,[http/1.1]}{0.0.0.0:41907}
  968. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@66f42bf3{/jobs,null,AVAILABLE,@Spark}
  969. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@18b8da1d{/jobs/json,null,AVAILABLE,@Spark}
  970. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4a39e2d5{/jobs/job,null,AVAILABLE,@Spark}
  971. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@27a26ec5{/jobs/job/json,null,AVAILABLE,@Spark}
  972. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@22881fae{/stages,null,AVAILABLE,@Spark}
  973. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6bb6b759{/stages/json,null,AVAILABLE,@Spark}
  974. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@33249b1c{/stages/stage,null,AVAILABLE,@Spark}
  975. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3b017a4e{/stages/stage/json,null,AVAILABLE,@Spark}
  976. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ceaea49{/stages/pool,null,AVAILABLE,@Spark}
  977. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b7613f8{/stages/pool/json,null,AVAILABLE,@Spark}
  978. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@76767ad1{/storage,null,AVAILABLE,@Spark}
  979. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4615bd5b{/storage/json,null,AVAILABLE,@Spark}
  980. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3d56af72{/storage/rdd,null,AVAILABLE,@Spark}
  981. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60482f3{/storage/rdd/json,null,AVAILABLE,@Spark}
  982. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@43e1d010{/environment,null,AVAILABLE,@Spark}
  983. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@28f54252{/environment/json,null,AVAILABLE,@Spark}
  984. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@20d04501{/executors,null,AVAILABLE,@Spark}
  985. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2e158f2{/executors/json,null,AVAILABLE,@Spark}
  986. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4fcd79ee{/executors/threadDump,null,AVAILABLE,@Spark}
  987. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4d285597{/executors/threadDump/json,null,AVAILABLE,@Spark}
  988. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6ff2f97e{/static,null,AVAILABLE,@Spark}
  989. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a754b54{/,null,AVAILABLE,@Spark}
  990. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@275037fd{/api,null,AVAILABLE,@Spark}
  991. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7a6c5c77{/jobs/job/kill,null,AVAILABLE,@Spark}
  992. 18/03/09 18:53:16 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1354ccf0{/stages/stage/kill,null,AVAILABLE,@Spark}
  993. 18/03/09 18:53:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://bcaf3c62a630:41907
  994. 18/03/09 18:53:17 INFO YarnClusterScheduler: Created YarnClusterScheduler
  995. 18/03/09 18:53:17 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1520619475936_0001 and attemptId Some(appattempt_1520619475936_0001_000001)
  996. 18/03/09 18:53:17 INFO NettyBlockTransferService: Server created on bcaf3c62a630:42057
  997. 18/03/09 18:53:17 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  998. 18/03/09 18:53:17 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, bcaf3c62a630, 42057, None)
  999. 18/03/09 18:53:17 INFO BlockManagerMasterEndpoint: Registering block manager bcaf3c62a630:42057 with 4.5 GB RAM, BlockManagerId(driver, bcaf3c62a630, 42057, None)
  1000. 18/03/09 18:53:17 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, bcaf3c62a630, 42057, None)
  1001. 18/03/09 18:53:17 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, bcaf3c62a630, 42057, None)
  1002. 18/03/09 18:53:17 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@14b2678a{/metrics/json,null,AVAILABLE,@Spark}
  1003. 18/03/09 18:53:17 INFO EventLoggingListener: Logging events to hdfs://master:9000/user/hduser/input/application_1520619475936_0001_1
  1004. 18/03/09 18:53:17 INFO ApplicationMaster:
  1005. ===============================================================================
  1006. YARN executor launch context:
  1007. env:
  1008. CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>/usr/local/hadoop/etc/hadoop<CPS>/usr/local/hadoop/share/hadoop/common/lib/*<CPS>/usr/local/hadoop/share/hadoop/common/*<CPS>/usr/local/hadoop/share/hadoop/hdfs<CPS>/usr/local/hadoop/share/hadoop/hdfs/lib/*<CPS>/usr/local/hadoop/share/hadoop/hdfs/*<CPS>/usr/local/hadoop/share/hadoop/mapreduce/*<CPS>/usr/local/hadoop/share/hadoop/yarn<CPS>/usr/local/hadoop/share/hadoop/yarn/lib/*<CPS>/usr/local/hadoop/share/hadoop/yarn/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
  1009. SPARK_DIST_CLASSPATH -> /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*
  1010. SPARK_YARN_STAGING_DIR -> *********(redacted)
  1011. SPARK_USER -> *********(redacted)
  1012.  
  1013. command:
  1014. {{JAVA_HOME}}/bin/java \
  1015. -server \
  1016. -Xmx1024m \
  1017. '-XX:MaxGCPauseMillis=950' \
  1018. -Djava.io.tmpdir={{PWD}}/tmp \
  1019. -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
  1020. -XX:OnOutOfMemoryError='kill %p' \
  1021. org.apache.spark.executor.CoarseGrainedExecutorBackend \
  1022. --driver-url \
  1023. spark://CoarseGrainedScheduler@bcaf3c62a630:36031 \
  1024. --executor-id \
  1025. <executorId> \
  1026. --hostname \
  1027. <hostname> \
  1028. --cores \
  1029. 1 \
  1030. --app-id \
  1031. application_1520619475936_0001 \
  1032. --user-class-path \
  1033. file:$PWD/__app__.jar \
  1034. 1><LOG_DIR>/stdout \
  1035. 2><LOG_DIR>/stderr
  1036.  
  1037. resources:
  1038. __app__.jar -> resource { scheme: "hdfs" host: "master" port: 9000 file: "/user/hduser/.sparkStaging/application_1520619475936_0001/icu-mortality-1.0.0-all.jar" } size: 530343861 timestamp: 1520621558212 type: FILE visibility: PRIVATE
  1039. __spark_libs__ -> resource { scheme: "hdfs" host: "master" port: 9000 file: "/user/hduser/.sparkStaging/application_1520619475936_0001/__spark_libs__1007958283373928587.zip" } size: 155925110 timestamp: 1520621529995 type: ARCHIVE visibility: PRIVATE
  1040. __spark_conf__ -> resource { scheme: "hdfs" host: "master" port: 9000 file: "/user/hduser/.sparkStaging/application_1520619475936_0001/__spark_conf__.zip" } size: 303564 timestamp: 1520621559848 type: ARCHIVE visibility: PRIVATE
  1041.  
  1042. ===============================================================================
  1043. 18/03/09 18:53:17 INFO RMProxy: Connecting to ResourceManager at master/172.22.0.7:18030
  1044. 18/03/09 18:53:18 INFO YarnRMClient: Registering the ApplicationMaster
  1045. 18/03/09 18:53:18 INFO Configuration: resource-types.xml not found
  1046. 18/03/09 18:53:18 INFO ResourceUtils: Unable to find 'resource-types.xml'.
  1047. 18/03/09 18:53:18 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@bcaf3c62a630:36031)
  1048. 18/03/09 18:53:18 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
  1049. 18/03/09 18:53:18 INFO YarnAllocator: Submitted 2 unlocalized container requests.
  1050. 18/03/09 18:53:19 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
  1051. 18/03/09 18:53:19 INFO YarnAllocator: Launching container container_1520619475936_0001_01_000002 on host bf263d1bde4d for executor with ID 1
  1052. 18/03/09 18:53:19 INFO YarnAllocator: Launching container container_1520619475936_0001_01_000003 on host bf263d1bde4d for executor with ID 2
  1053. 18/03/09 18:53:19 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
  1054. 18/03/09 18:53:39 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.22.0.6:33822) with ID 1
  1055. 18/03/09 18:53:39 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.22.0.6:33824) with ID 2
  1056. 18/03/09 18:53:39 INFO BlockManagerMasterEndpoint: Registering block manager bf263d1bde4d:39531 with 2.4 GB RAM, BlockManagerId(2, bf263d1bde4d, 39531, None)
  1057. 18/03/09 18:53:39 INFO BlockManagerMasterEndpoint: Registering block manager bf263d1bde4d:38407 with 2.4 GB RAM, BlockManagerId(1, bf263d1bde4d, 38407, None)
  1058. 18/03/09 18:53:39 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
  1059. 18/03/09 18:53:39 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
  1060. 18/03/09 18:53:42 INFO Nd4jBackend: Loaded [CpuBackend] backend
  1061. 18/03/09 18:53:47 INFO NativeOpsHolder: Number of threads used for NativeOps: 1
  1062. 18/03/09 18:53:48 ERROR ApplicationMaster: User class threw exception: java.lang.ExceptionInInitializerError
  1063. java.lang.ExceptionInInitializerError
  1064. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.setMaxThreads(CpuBlas.java:117)
  1065. at org.nd4j.nativeblas.Nd4jBlas.<init>(Nd4jBlas.java:35)
  1066. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.<init>(CpuBlas.java:12)
  1067. at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createBlas(CpuNDArrayFactory.java:89)
  1068. at org.nd4j.linalg.factory.BaseNDArrayFactory.blas(BaseNDArrayFactory.java:67)
  1069. at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getEnvironmentInformation(NativeOpExecutioner.java:1176)
  1070. at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.printEnvironmentInformation(DefaultOpExecutioner.java:562)
  1071. at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:6210)
  1072. at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6087)
  1073. at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:201)
  1074. at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:777)
  1075. at org.deeplearning4j.icumortality.SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.entryPoint(SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.java:102)
  1076. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.mainEntryPoint(SparkMainForLocalSparkClusterDeploy.java:67)
  1077. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.main(SparkMainForLocalSparkClusterDeploy.java:23)
  1078. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1079. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1080. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1081. at java.lang.reflect.Method.invoke(Method.java:498)
  1082. at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
  1083. Caused by: java.nio.channels.OverlappingFileLockException
  1084. at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
  1085. at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
  1086. at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
  1087. at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
  1088. at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:429)
  1089. at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:944)
  1090. at org.bytedeco.javacpp.Loader.load(Loader.java:765)
  1091. at org.bytedeco.javacpp.Loader.load(Loader.java:684)
  1092. at org.bytedeco.javacpp.openblas.<clinit>(openblas.java:10)
  1093. ... 19 more
  1094. 18/03/09 18:53:48 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.ExceptionInInitializerError
  1095. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.setMaxThreads(CpuBlas.java:117)
  1096. at org.nd4j.nativeblas.Nd4jBlas.<init>(Nd4jBlas.java:35)
  1097. at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.<init>(CpuBlas.java:12)
  1098. at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createBlas(CpuNDArrayFactory.java:89)
  1099. at org.nd4j.linalg.factory.BaseNDArrayFactory.blas(BaseNDArrayFactory.java:67)
  1100. at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getEnvironmentInformation(NativeOpExecutioner.java:1176)
  1101. at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.printEnvironmentInformation(DefaultOpExecutioner.java:562)
  1102. at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:6210)
  1103. at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6087)
  1104. at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:201)
  1105. at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:777)
  1106. at org.deeplearning4j.icumortality.SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.entryPoint(SparkLSTMClinicalTimeSeriesClassificationExampleHDFS.java:102)
  1107. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.mainEntryPoint(SparkMainForLocalSparkClusterDeploy.java:67)
  1108. at org.deeplearning4j.rnn.SparkMainForLocalSparkClusterDeploy.main(SparkMainForLocalSparkClusterDeploy.java:23)
  1109. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1110. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1111. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1112. at java.lang.reflect.Method.invoke(Method.java:498)
  1113. at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
  1114. Caused by: java.nio.channels.OverlappingFileLockException
  1115. at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
  1116. at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
  1117. at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
  1118. at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
  1119. at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:429)
  1120. at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:944)
  1121. at org.bytedeco.javacpp.Loader.load(Loader.java:765)
  1122. at org.bytedeco.javacpp.Loader.load(Loader.java:684)
  1123. at org.bytedeco.javacpp.openblas.<clinit>(openblas.java:10)
  1124. ... 19 more
  1125. )
  1126. 18/03/09 18:53:48 INFO SparkContext: Invoking stop() from shutdown hook
  1127. 18/03/09 18:53:48 INFO AbstractConnector: Stopped Spark@4c972453{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
  1128. 18/03/09 18:53:48 INFO SparkUI: Stopped Spark web UI at http://bcaf3c62a630:41907
  1129. 18/03/09 18:53:48 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
  1130. 18/03/09 18:53:48 INFO YarnClusterSchedulerBackend: Shutting down all executors
  1131. 18/03/09 18:53:48 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
  1132. 18/03/09 18:53:48 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
  1133. (serviceOption=None,
  1134. services=List(),
  1135. started=false)
  1136. 18/03/09 18:53:48 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
  1137. 18/03/09 18:53:48 INFO MemoryStore: MemoryStore cleared
  1138. 18/03/09 18:53:48 INFO BlockManager: BlockManager stopped
  1139. 18/03/09 18:53:48 INFO BlockManagerMaster: BlockManagerMaster stopped
  1140. 18/03/09 18:53:48 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  1141. 18/03/09 18:53:48 INFO SparkContext: Successfully stopped SparkContext
  1142. 18/03/09 18:53:48 INFO ShutdownHookManager: Shutdown hook called
  1143. 18/03/09 18:53:48 INFO ShutdownHookManager: Deleting directory /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/spark-33e28136-3187-42d8-b984-a10aa032da9c
  1144.  
  1145. End of LogType:stderr
  1146. ***********************************************************************
  1147.  
  1148. Container: container_1520619475936_0001_01_000001 on bcaf3c62a630_41471
  1149. LogAggregationType: AGGREGATED
  1150. =======================================================================
  1151. LogType:stdout
  1152. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1153. LogLength:2901
  1154. LogContents:
  1155. [GC (Allocation Failure) [PSYoungGen: 64512K->8870K(75264K)] 64512K->8878K(247296K), 0.0165927 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
  1156. [GC (Allocation Failure) [PSYoungGen: 73382K->7466K(75264K)] 73390K->7482K(247296K), 0.0275285 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
  1157. [GC (Allocation Failure) [PSYoungGen: 71978K->7929K(75264K)] 71994K->7953K(247296K), 0.0216622 secs] [Times: user=0.04 sys=0.01, real=0.02 secs]
  1158. [GC (Metadata GC Threshold) [PSYoungGen: 16761K->4556K(139776K)] 16785K->4580K(311808K), 0.0141460 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
  1159. [Full GC (Metadata GC Threshold) [PSYoungGen: 4556K->0K(139776K)] [ParOldGen: 24K->4477K(75264K)] 4580K->4477K(215040K), [Metaspace: 20872K->20872K(1067008K)], 0.0455738 secs] [Times: user=0.20 sys=0.00, real=0.04 secs]
  1160. [GC (Allocation Failure) [PSYoungGen: 129024K->5017K(139776K)] 133501K->9494K(215040K), 0.0131629 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
  1161. [GC (Allocation Failure) [PSYoungGen: 134041K->10033K(250880K)] 138518K->14518K(326144K), 0.0160260 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
  1162. [GC (Metadata GC Threshold) [PSYoungGen: 22163K->5230K(253440K)] 26648K->9723K(328704K), 0.0099738 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
  1163. [Full GC (Metadata GC Threshold) [PSYoungGen: 5230K->0K(253440K)] [ParOldGen: 4493K->8829K(89600K)] 9723K->8829K(343040K), [Metaspace: 35130K->35130K(1079296K)], 0.0599117 secs] [Times: user=0.17 sys=0.02, real=0.06 secs]
  1164. [GC (Allocation Failure) [PSYoungGen: 242176K->11263K(455680K)] 251005K->24313K(545280K), 0.0408380 secs] [Times: user=0.14 sys=0.03, real=0.04 secs]
  1165. [GC (Allocation Failure) [PSYoungGen: 455679K->14826K(459264K)] 468729K->142533K(587264K), 1.2294700 secs] [Times: user=1.21 sys=3.02, real=1.23 secs]
  1166. [Full GC (Ergonomics) [PSYoungGen: 14826K->8623K(459264K)] [ParOldGen: 127707K->127775K(266240K)] 142533K->136399K(725504K), [Metaspace: 52321K->52316K(1095680K)], 0.6152671 secs] [Times: user=1.81 sys=0.14, real=0.62 secs]
  1167. [GC (Allocation Failure) [PSYoungGen: 453039K->62944K(709632K)] 580815K->211886K(975872K), 0.0599539 secs] [Times: user=0.25 sys=0.06, real=0.06 secs]
  1168. Heap
  1169. PSYoungGen total 709632K, used 375306K [0x0000000755580000, 0x0000000787080000, 0x00000007c0000000)
  1170. eden space 646656K, 48% used [0x0000000755580000,0x000000076868a720,0x000000077cd00000)
  1171. from space 62976K, 99% used [0x0000000781d00000,0x0000000785a78250,0x0000000785a80000)
  1172. to space 81920K, 0% used [0x000000077cd00000,0x000000077cd00000,0x0000000781d00000)
  1173. ParOldGen total 266240K, used 148941K [0x0000000680000000, 0x0000000690400000, 0x0000000755580000)
  1174. object space 266240K, 55% used [0x0000000680000000,0x00000006891735d0,0x0000000690400000)
  1175. Metaspace used 58619K, capacity 58972K, committed 59176K, reserved 1101824K
  1176. class space used 7503K, capacity 7605K, committed 7720K, reserved 1048576K
  1177.  
  1178. End of LogType:stdout
  1179. ***********************************************************************
  1180.  
  1181. Container: container_1520619475936_0001_02_000002 on bcaf3c62a630_41471
  1182. LogAggregationType: AGGREGATED
  1183. =======================================================================
  1184. LogType:directory.info
  1185. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1186. LogLength:21689
  1187. LogContents:
  1188. ls -l:
  1189. total 36
  1190. lrwxrwxrwx 1 hduser hadoop 89 Mar 9 18:54 __app__.jar -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar
  1191. lrwxrwxrwx 1 hduser hadoop 80 Mar 9 18:54 __spark_conf__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip
  1192. lrwxrwxrwx 1 hduser hadoop 99 Mar 9 18:54 __spark_libs__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip
  1193. -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:54 container_tokens
  1194. -rwx------ 1 hduser hadoop 733 Mar 9 18:54 default_container_executor.sh
  1195. -rwx------ 1 hduser hadoop 678 Mar 9 18:54 default_container_executor_session.sh
  1196. -rwx------ 1 hduser hadoop 6052 Mar 9 18:54 launch_container.sh
  1197. drwx--x--- 2 hduser hadoop 4096 Mar 9 18:54 tmp
  1198. find -L . -maxdepth 5 -ls:
  1199. 1674208 4 drwx--x--- 3 hduser hadoop 4096 Mar 9 18:54 .
  1200. 7089874 4 -rwx------ 1 hduser hadoop 733 Mar 9 18:54 ./default_container_executor.sh
  1201. 7089867 8 -rwx------ 1 hduser hadoop 6052 Mar 9 18:54 ./launch_container.sh
  1202. 1674210 4 drwx--x--- 2 hduser hadoop 4096 Mar 9 18:54 ./tmp
  1203. 7089162 517920 -r-x------ 1 hduser hadoop 530343861 Mar 9 18:53 ./__app__.jar
  1204. 7089875 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:54 ./.default_container_executor.sh.crc
  1205. 1656886 12 drwx------ 2 hduser hadoop 12288 Mar 9 18:52 ./__spark_libs__
  1206. 7089166 88 -r-x------ 1 hduser hadoop 87192 Mar 9 18:52 ./__spark_libs__/spire-macros_2.11-0.13.0.jar
  1207. 7089167 5612 -r-x------ 1 hduser hadoop 5744974 Mar 9 18:52 ./__spark_libs__/scala-library-2.11.8.jar
  1208. 7089168 20 -r-x------ 1 hduser hadoop 19827 Mar 9 18:52 ./__spark_libs__/opencsv-2.3.jar
  1209. 7089169 16 -r-x------ 1 hduser hadoop 13911 Mar 9 18:52 ./__spark_libs__/generex-1.0.1.jar
  1210. 7089170 364 -r-x------ 1 hduser hadoop 370119 Mar 9 18:52 ./__spark_libs__/lz4-java-1.4.0.jar
  1211. 7089171 256 -r-x------ 1 hduser hadoop 258919 Mar 9 18:52 ./__spark_libs__/jackson-core-2.6.7.jar
  1212. 7089172 684 -r-x------ 1 hduser hadoop 698375 Mar 9 18:52 ./__spark_libs__/jersey-common-2.22.2.jar
  1213. 7089173 472 -r-x------ 1 hduser hadoop 479881 Mar 9 18:52 ./__spark_libs__/commons-lang3-3.5.jar
  1214. 7089174 6852 -r-x------ 1 hduser hadoop 7015233 Mar 9 18:52 ./__spark_libs__/kubernetes-model-2.0.0.jar
  1215. 7089175 176 -r-x------ 1 hduser hadoop 178947 Mar 9 18:52 ./__spark_libs__/hk2-api-2.4.0-b34.jar
  1216. 7089176 7172 -r-x------ 1 hduser hadoop 7343426 Mar 9 18:52 ./__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
  1217. 7089177 504 -r-x------ 1 hduser hadoop 515645 Mar 9 18:52 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
  1218. 7089178 44 -r-x------ 1 hduser hadoop 42858 Mar 9 18:52 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar
  1219. 7089179 60 -r-x------ 1 hduser hadoop 60483 Mar 9 18:52 ./__spark_libs__/chill-java-0.8.4.jar
  1220. 7089180 132 -r-x------ 1 hduser hadoop 134595 Mar 9 18:52 ./__spark_libs__/commons-crypto-1.0.0.jar
  1221. 7089181 128 -r-x------ 1 hduser hadoop 130802 Mar 9 18:52 ./__spark_libs__/aircompressor-0.8.jar
  1222. 7089182 264 -r-x------ 1 hduser hadoop 269295 Mar 9 18:52 ./__spark_libs__/snakeyaml-1.15.jar
  1223. 7089183 1168 -r-x------ 1 hduser hadoop 1194003 Mar 9 18:52 ./__spark_libs__/arpack_combined_all-0.1.jar
  1224. 7089184 14760 -r-x------ 1 hduser hadoop 15113382 Mar 9 18:52 ./__spark_libs__/breeze_2.11-0.13.2.jar
  1225. 7089185 36 -r-x------ 1 hduser hadoop 35518 Mar 9 18:52 ./__spark_libs__/zjsonpatch-0.3.0.jar
  1226. 7089186 120 -r-x------ 1 hduser hadoop 120321 Mar 9 18:52 ./__spark_libs__/spark-repl_2.11-2.3.0.jar
  1227. 7089187 36 -r-x------ 1 hduser hadoop 34603 Mar 9 18:52 ./__spark_libs__/machinist_2.11-0.6.1.jar
  1228. 7089188 28 -r-x------ 1 hduser hadoop 26366 Mar 9 18:52 ./__spark_libs__/javax.annotation-api-1.2.jar
  1229. 7089189 180 -r-x------ 1 hduser hadoop 180736 Mar 9 18:52 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
  1230. 7089190 788 -r-x------ 1 hduser hadoop 802818 Mar 9 18:52 ./__spark_libs__/scalap-2.11.8.jar
  1231. 7089191 1504 -r-x------ 1 hduser hadoop 1537966 Mar 9 18:52 ./__spark_libs__/hadoop-mapreduce-client-core-2.6.5.jar
  1232. 7089192 1024 -r-x------ 1 hduser hadoop 1048116 Mar 9 18:52 ./__spark_libs__/parquet-jackson-1.8.2.jar
  1233. 7089193 172 -r-x------ 1 hduser hadoop 174351 Mar 9 18:52 ./__spark_libs__/stream-2.7.0.jar
  1234. 7089194 68 -r-x------ 1 hduser hadoop 66270 Mar 9 18:52 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar
  1235. 7089195 1632 -r-x------ 1 hduser hadoop 1671083 Mar 9 18:52 ./__spark_libs__/hppc-0.7.2.jar
  1236. 7089196 180 -r-x------ 1 hduser hadoop 183832 Mar 9 18:52 ./__spark_libs__/spark-mllib-local_2.11-2.3.0.jar
  1237. 7089197 656 -r-x------ 1 hduser hadoop 671138 Mar 9 18:52 ./__spark_libs__/scala-xml_2.11-1.0.5.jar
  1238. 7089198 200 -r-x------ 1 hduser hadoop 201928 Mar 9 18:52 ./__spark_libs__/RoaringBitmap-0.5.11.jar
  1239. 7089199 96 -r-x------ 1 hduser hadoop 94796 Mar 9 18:52 ./__spark_libs__/pyrolite-4.13.jar
  1240. 7089200 3692 -r-x------ 1 hduser hadoop 3780056 Mar 9 18:52 ./__spark_libs__/netty-all-4.1.17.Final.jar
  1241. 7089201 316 -r-x------ 1 hduser hadoop 320444 Mar 9 18:52 ./__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
  1242. 7089202 1024 -r-x------ 1 hduser hadoop 1045744 Mar 9 18:52 ./__spark_libs__/leveldbjni-all-1.8.jar
  1243. 7089203 376 -r-x------ 1 hduser hadoop 381098 Mar 9 18:52 ./__spark_libs__/spark-kubernetes_2.11-2.3.0.jar
  1244. 7089204 32 -r-x------ 1 hduser hadoop 32612 Mar 9 18:52 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
  1245. 7089205 636 -r-x------ 1 hduser hadoop 650236 Mar 9 18:52 ./__spark_libs__/spark-yarn_2.11-2.3.0.jar
  1246. 7089206 48 -r-x------ 1 hduser hadoop 46986 Mar 9 18:52 ./__spark_libs__/jackson-annotations-2.6.7.jar
  1247. 7089207 700 -r-x------ 1 hduser hadoop 714194 Mar 9 18:52 ./__spark_libs__/javassist-3.18.1-GA.jar
  1248. 7089208 248 -r-x------ 1 hduser hadoop 250377 Mar 9 18:52 ./__spark_libs__/parquet-hadoop-1.8.2.jar
  1249. 7089209 64 -r-x------ 1 hduser hadoop 64847 Mar 9 18:52 ./__spark_libs__/spark-network-shuffle_2.11-2.3.0.jar
  1250. 7089210 116 -r-x------ 1 hduser hadoop 116859 Mar 9 18:52 ./__spark_libs__/py4j-0.10.6.jar
  1251. 7089211 76 -r-x------ 1 hduser hadoop 75835 Mar 9 18:52 ./__spark_libs__/spark-launcher_2.11-2.3.0.jar
  1252. 7089212 20 -r-x------ 1 hduser hadoop 16430 Mar 9 18:52 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar
  1253. 7089213 52 -r-x------ 1 hduser hadoop 52076 Mar 9 18:52 ./__spark_libs__/arrow-format-0.8.0.jar
  1254. 7089214 84 -r-x------ 1 hduser hadoop 82421 Mar 9 18:52 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar
  1255. 7089215 740 -r-x------ 1 hduser hadoop 757423 Mar 9 18:52 ./__spark_libs__/orc-mapreduce-1.4.1-nohive.jar
  1256. 7089216 1300 -r-x------ 1 hduser hadoop 1330219 Mar 9 18:52 ./__spark_libs__/netty-3.9.9.Final.jar
  1257. 7089217 120 -r-x------ 1 hduser hadoop 118973 Mar 9 18:52 ./__spark_libs__/hk2-utils-2.4.0-b34.jar
  1258. 7089218 408 -r-x------ 1 hduser hadoop 416731 Mar 9 18:52 ./__spark_libs__/kubernetes-client-3.0.0.jar
  1259. 7089219 96 -r-x------ 1 hduser hadoop 95806 Mar 9 18:52 ./__spark_libs__/javax.servlet-api-3.1.0.jar
  1260. 7089220 936 -r-x------ 1 hduser hadoop 956984 Mar 9 18:52 ./__spark_libs__/parquet-column-1.8.2.jar
  1261. 7089221 1412 -r-x------ 1 hduser hadoop 1441994 Mar 9 18:52 ./__spark_libs__/orc-core-1.4.1-nohive.jar
  1262. 7089222 576 -r-x------ 1 hduser hadoop 589462 Mar 9 18:52 ./__spark_libs__/json4s-core_2.11-3.2.11.jar
  1263. 7089223 2328 -r-x------ 1 hduser hadoop 2382165 Mar 9 18:52 ./__spark_libs__/spark-network-common_2.11-2.3.0.jar
  1264. 7089224 12 -r-x------ 1 hduser hadoop 8268 Mar 9 18:52 ./__spark_libs__/logging-interceptor-3.8.1.jar
  1265. 7089225 40 -r-x------ 1 hduser hadoop 40341 Mar 9 18:52 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar
  1266. 7089226 52 -r-x------ 1 hduser hadoop 53007 Mar 9 18:52 ./__spark_libs__/spark-kvstore_2.11-2.3.0.jar
  1267. 7089227 8 -r-x------ 1 hduser hadoop 5950 Mar 9 18:52 ./__spark_libs__/javax.inject-2.4.0-b34.jar
  1268. 7089228 376 -r-x------ 1 hduser hadoop 384150 Mar 9 18:52 ./__spark_libs__/univocity-parsers-2.5.9.jar
  1269. 7089229 7560 -r-x------ 1 hduser hadoop 7738527 Mar 9 18:52 ./__spark_libs__/spark-mllib_2.11-2.3.0.jar
  1270. 7089230 116 -r-x------ 1 hduser hadoop 115534 Mar 9 18:52 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar
  1271. 7089231 72 -r-x------ 1 hduser hadoop 72733 Mar 9 18:52 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar
  1272. 7089232 748 -r-x------ 1 hduser hadoop 764569 Mar 9 18:52 ./__spark_libs__/jtransforms-2.4.0.jar
  1273. 7089233 12 -r-x------ 1 hduser hadoop 10166 Mar 9 18:52 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
  1274. 7089234 36 -r-x------ 1 hduser hadoop 34654 Mar 9 18:52 ./__spark_libs__/paranamer-2.8.jar
  1275. 7089235 8464 -r-x------ 1 hduser hadoop 8664987 Mar 9 18:52 ./__spark_libs__/spark-sql_2.11-2.3.0.jar
  1276. 7089236 164 -r-x------ 1 hduser hadoop 164422 Mar 9 18:52 ./__spark_libs__/core-1.1.2.jar
  1277. 7089237 44 -r-x------ 1 hduser hadoop 42032 Mar 9 18:52 ./__spark_libs__/parquet-common-1.8.2.jar
  1278. 7089238 36 -r-x------ 1 hduser hadoop 33015 Mar 9 18:52 ./__spark_libs__/jsr305-1.3.9.jar
  1279. 7089239 80 -r-x------ 1 hduser hadoop 81811 Mar 9 18:52 ./__spark_libs__/okio-1.13.0.jar
  1280. 7089240 328 -r-x------ 1 hduser hadoop 334662 Mar 9 18:52 ./__spark_libs__/antlr4-runtime-4.7.jar
  1281. 7089241 392 -r-x------ 1 hduser hadoop 398122 Mar 9 18:52 ./__spark_libs__/okhttp-3.8.1.jar
  1282. 7089242 352 -r-x------ 1 hduser hadoop 358390 Mar 9 18:52 ./__spark_libs__/kryo-shaded-3.0.3.jar
  1283. 7089243 80 -r-x------ 1 hduser hadoop 79150 Mar 9 18:52 ./__spark_libs__/arrow-memory-0.8.0.jar
  1284. 7089244 15128 -r-x------ 1 hduser hadoop 15487351 Mar 9 18:52 ./__spark_libs__/scala-compiler-2.11.8.jar
  1285. 7089245 164 -r-x------ 1 hduser hadoop 167421 Mar 9 18:52 ./__spark_libs__/jersey-client-2.22.2.jar
  1286. 7089246 384 -r-x------ 1 hduser hadoop 390733 Mar 9 18:52 ./__spark_libs__/parquet-format-2.3.1.jar
  1287. 7089247 64 -r-x------ 1 hduser hadoop 63777 Mar 9 18:52 ./__spark_libs__/validation-api-1.1.0.Final.jar
  1288. 7089248 4468 -r-x------ 1 hduser hadoop 4573750 Mar 9 18:52 ./__spark_libs__/scala-reflect-2.11.8.jar
  1289. 7089249 932 -r-x------ 1 hduser hadoop 951701 Mar 9 18:52 ./__spark_libs__/jersey-server-2.22.2.jar
  1290. 7089250 48 -r-x------ 1 hduser hadoop 48464 Mar 9 18:52 ./__spark_libs__/spark-unsafe_2.11-2.3.0.jar
  1291. 7089251 184 -r-x------ 1 hduser hadoop 187882 Mar 9 18:52 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar
  1292. 7089252 3444 -r-x------ 1 hduser hadoop 3522616 Mar 9 18:52 ./__spark_libs__/shapeless_2.11-2.3.2.jar
  1293. 7089253 8 -r-x------ 1 hduser hadoop 5711 Mar 9 18:52 ./__spark_libs__/minlog-1.3.0.jar
  1294. 7089254 20 -r-x------ 1 hduser hadoop 18098 Mar 9 18:52 ./__spark_libs__/jersey-container-servlet-2.22.2.jar
  1295. 7089255 40 -r-x------ 1 hduser hadoop 37887 Mar 9 18:52 ./__spark_libs__/commons-compiler-3.0.8.jar
  1296. 7089256 44 -r-x------ 1 hduser hadoop 41755 Mar 9 18:52 ./__spark_libs__/objenesis-2.1.jar
  1297. 7089257 280 -r-x------ 1 hduser hadoop 286071 Mar 9 18:52 ./__spark_libs__/parquet-encoding-1.8.2.jar
  1298. 7089258 220 -r-x------ 1 hduser hadoop 224167 Mar 9 18:52 ./__spark_libs__/chill_2.11-0.8.4.jar
  1299. 7089259 780 -r-x------ 1 hduser hadoop 796326 Mar 9 18:52 ./__spark_libs__/janino-3.0.8.jar
  1300. 7089260 1244 -r-x------ 1 hduser hadoop 1270377 Mar 9 18:52 ./__spark_libs__/arrow-vector-0.8.0.jar
  1301. 7089261 24 -r-x------ 1 hduser hadoop 21247 Mar 9 18:52 ./__spark_libs__/metrics-graphite-3.1.5.jar
  1302. 7089262 1988 -r-x------ 1 hduser hadoop 2035066 Mar 9 18:52 ./__spark_libs__/commons-math3-3.4.1.jar
  1303. 7089263 32 -r-x------ 1 hduser hadoop 30057 Mar 9 18:52 ./__spark_libs__/spark-sketch_2.11-2.3.0.jar
  1304. 7089264 2120 -r-x------ 1 hduser hadoop 2169825 Mar 9 18:52 ./__spark_libs__/spark-streaming_2.11-2.3.0.jar
  1305. 7089265 9888 -r-x------ 1 hduser hadoop 10121868 Mar 9 18:52 ./__spark_libs__/spire_2.11-0.13.0.jar
  1306. 7089266 8 -r-x------ 1 hduser hadoop 4596 Mar 9 18:52 ./__spark_libs__/jul-to-slf4j-1.7.16.jar
  1307. 7089267 20 -r-x------ 1 hduser hadoop 20235 Mar 9 18:52 ./__spark_libs__/osgi-resource-locator-1.0.1.jar
  1308. 7089268 4 -r-x------ 1 hduser hadoop 3142 Mar 9 18:52 ./__spark_libs__/macro-compat_2.11-1.1.1.jar
  1309. 7089269 280 -r-x------ 1 hduser hadoop 284220 Mar 9 18:52 ./__spark_libs__/commons-lang-2.6.jar
  1310. 7089270 1256 -r-x------ 1 hduser hadoop 1282424 Mar 9 18:52 ./__spark_libs__/ivy-2.4.0.jar
  1311. 7089271 80 -r-x------ 1 hduser hadoop 79845 Mar 9 18:52 ./__spark_libs__/compress-lzf-1.0.3.jar
  1312. 7089272 16 -r-x------ 1 hduser hadoop 15824 Mar 9 18:52 ./__spark_libs__/metrics-json-3.1.5.jar
  1313. 7089273 12680 -r-x------ 1 hduser hadoop 12982716 Mar 9 18:52 ./__spark_libs__/spark-core_2.11-2.3.0.jar
  1314. 7089274 180 -r-x------ 1 hduser hadoop 181271 Mar 9 18:52 ./__spark_libs__/hk2-locator-2.4.0-b34.jar
  1315. 7089275 416 -r-x------ 1 hduser hadoop 423753 Mar 9 18:52 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
  1316. 7089276 1140 -r-x------ 1 hduser hadoop 1165323 Mar 9 18:52 ./__spark_libs__/jackson-databind-2.6.7.1.jar
  1317. 7089277 208 -r-x------ 1 hduser hadoop 212453 Mar 9 18:52 ./__spark_libs__/commons-net-2.2.jar
  1318. 7089278 64 -r-x------ 1 hduser hadoop 65261 Mar 9 18:52 ./__spark_libs__/oro-2.0.8.jar
  1319. 7089279 144 -r-x------ 1 hduser hadoop 144660 Mar 9 18:52 ./__spark_libs__/xbean-asm5-shaded-4.4.jar
  1320. 7089280 8776 -r-x------ 1 hduser hadoop 8984422 Mar 9 18:52 ./__spark_libs__/spark-catalyst_2.11-2.3.0.jar
  1321. 7089281 40 -r-x------ 1 hduser hadoop 39283 Mar 9 18:52 ./__spark_libs__/metrics-jvm-3.1.5.jar
  1322. 7089282 280 -r-x------ 1 hduser hadoop 284184 Mar 9 18:52 ./__spark_libs__/commons-codec-1.10.jar
  1323. 7089283 176 -r-x------ 1 hduser hadoop 176285 Mar 9 18:52 ./__spark_libs__/automaton-1.11-8.jar
  1324. 7089284 2280 -r-x------ 1 hduser hadoop 2333186 Mar 9 18:52 ./__spark_libs__/zstd-jni-1.3.2-2.jar
  1325. 7089285 648 -r-x------ 1 hduser hadoop 663328 Mar 9 18:52 ./__spark_libs__/spark-mesos_2.11-2.3.0.jar
  1326. 7089286 696 -r-x------ 1 hduser hadoop 708667 Mar 9 18:52 ./__spark_libs__/spark-graphx_2.11-2.3.0.jar
  1327. 7089287 16 -r-x------ 1 hduser hadoop 15456 Mar 9 18:52 ./__spark_libs__/spark-tags_2.11-2.3.0.jar
  1328. 7089288 616 -r-x------ 1 hduser hadoop 627814 Mar 9 18:52 ./__spark_libs__/joda-time-2.9.3.jar
  1329. 7089289 120 -r-x------ 1 hduser hadoop 120465 Mar 9 18:52 ./__spark_libs__/metrics-core-3.1.5.jar
  1330. 7089290 16 -r-x------ 1 hduser hadoop 14766 Mar 9 18:52 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
  1331. 7089291 952 -r-x------ 1 hduser hadoop 971310 Mar 9 18:52 ./__spark_libs__/jersey-guava-2.22.2.jar
  1332. 7089868 4 -rw-r--r-- 1 hduser hadoop 56 Mar 9 18:54 ./.launch_container.sh.crc
  1333. 7089863 4 -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:54 ./container_tokens
  1334. 7089872 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:54 ./.default_container_executor_session.sh.crc
  1335. 7089870 4 -rwx------ 1 hduser hadoop 678 Mar 9 18:54 ./default_container_executor_session.sh
  1336. 1653456 4 drwx------ 3 hduser hadoop 4096 Mar 9 18:52 ./__spark_conf__
  1337. 7089340 4 -r-x------ 1 hduser hadoop 1192 Mar 9 18:52 ./__spark_conf__/__spark_conf__.properties
  1338. 1653459 4 drwx------ 2 hduser hadoop 4096 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__
  1339. 7089323 4 -r-x------ 1 hduser hadoop 2316 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example
  1340. 7089327 4 -r-x------ 1 hduser hadoop 891 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/slaves
  1341. 7089338 4 -r-x------ 1 hduser hadoop 682 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-site.xml
  1342. 7089332 4 -r-x------ 1 hduser hadoop 996 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/docker.properties.template
  1343. 7089319 4 -r-x------ 1 hduser hadoop 1764 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-env.sh
  1344. 7089312 4 -r-x------ 1 hduser hadoop 3518 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-acls.xml
  1345. 7089337 8 -r-x------ 1 hduser hadoop 4113 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template
  1346. 7089329 4 -r-x------ 1 hduser hadoop 21 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-signature.secret
  1347. 7089328 4 -r-x------ 1 hduser hadoop 1484 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-env.sh
  1348. 7089314 4 -r-x------ 1 hduser hadoop 482 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-env.sh
  1349. 7089303 4 -r-x------ 1 hduser hadoop 3670 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd
  1350. 7089299 4 -r-x------ 1 hduser hadoop 2016 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.spark.properties
  1351. 7089308 8 -r-x------ 1 hduser hadoop 4221 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-env.sh.template
  1352. 7089298 4 -r-x------ 1 hduser hadoop 137 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
  1353. 7089322 4 -r-x------ 1 hduser hadoop 3414 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-user-functions.sh.example
  1354. 7089310 4 -r-x------ 1 hduser hadoop 2250 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd
  1355. 7089317 4 -r-x------ 1 hduser hadoop 589 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/container-executor.cfg
  1356. 7089307 4 -r-x------ 1 hduser hadoop 890 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/workers
  1357. 7089336 4 -r-x------ 1 hduser hadoop 2697 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example
  1358. 7089300 4 -r-x------ 1 hduser hadoop 1657 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-log4j.properties
  1359. 7089320 8 -r-x------ 1 hduser hadoop 7801 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/metrics.properties.template
  1360. 7089318 4 -r-x------ 1 hduser hadoop 1781 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf
  1361. 7089306 4 -r-x------ 1 hduser hadoop 225 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/core-site.xml
  1362. 7089301 4 -r-x------ 1 hduser hadoop 3323 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties
  1363. 7089311 8 -r-x------ 1 hduser hadoop 7861 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml
  1364. 7089334 4 -r-x------ 1 hduser hadoop 1292 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf.template
  1365. 7089321 4 -r-x------ 1 hduser hadoop 2642 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/user_ec_policies.xml.template
  1366. 7089333 4 -r-x------ 1 hduser hadoop 951 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd
  1367. 7089330 4 -r-x------ 1 hduser hadoop 620 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-site.xml
  1368. 7089335 4 -r-x------ 1 hduser hadoop 3589 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j2.xml
  1369. 7089325 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.properties.template
  1370. 7089305 4 -r-x------ 1 hduser hadoop 865 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/slaves.template
  1371. 7089316 4 -r-x------ 1 hduser hadoop 809 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
  1372. 7089297 16 -r-x------ 1 hduser hadoop 16270 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh
  1373. 7089313 4 -r-x------ 1 hduser hadoop 1335 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/configuration.xsl
  1374. 7089302 8 -r-x------ 1 hduser hadoop 8048 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
  1375. 7089296 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.properties
  1376. 7089324 4 -r-x------ 1 hduser hadoop 1351 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-env.sh
  1377. 7089309 4 -r-x------ 1 hduser hadoop 1747 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties
  1378. 7089326 12 -r-x------ 1 hduser hadoop 10206 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml
  1379. 7089331 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/metrics.properties
  1380. 7089315 8 -r-x------ 1 hduser hadoop 5502 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-env.sh
  1381. 7089304 4 -r-x------ 1 hduser hadoop 1105 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.template
  1382. 7089293 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/log4j.properties
  1383. 7089339 172 -r-x------ 1 hduser hadoop 173257 Mar 9 18:52 ./__spark_conf__/__spark_hadoop_conf__.xml
  1384. 7089294 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:52 ./__spark_conf__/metrics.properties
  1385. 7089865 4 -rw-r--r-- 1 hduser hadoop 12 Mar 9 18:54 ./.container_tokens.crc
  1386. broken symlinks(find -L . -maxdepth 5 -type l -ls):
  1387.  
  1388. End of LogType:directory.info
  1389. *******************************************************************************
  1390.  
  1391. Container: container_1520619475936_0001_02_000002 on bcaf3c62a630_41471
  1392. LogAggregationType: AGGREGATED
  1393. =======================================================================
  1394. LogType:launch_container.sh
  1395. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1396. LogLength:6052
  1397. LogContents:
  1398. #!/bin/bash
  1399.  
  1400. set -o pipefail -e
  1401. export PRELAUNCH_OUT="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/prelaunch.out"
  1402. exec >"${PRELAUNCH_OUT}"
  1403. export PRELAUNCH_ERR="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/prelaunch.err"
  1404. exec 2>"${PRELAUNCH_ERR}"
  1405. echo "Setting up env variables"
  1406. export SPARK_YARN_STAGING_DIR="hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001"
  1407. export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
  1408. export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
  1409. export JAVA_HOME="/usr/java/default"
  1410. export SPARK_LOG_URL_STDOUT="http://bcaf3c62a630:18042/node/containerlogs/container_1520619475936_0001_02_000002/hduser/stdout?start=-4096"
  1411. export NM_HOST="bcaf3c62a630"
  1412. export HADOOP_HDFS_HOME="/usr/local/hadoop"
  1413. export LOGNAME="hduser"
  1414. export JVM_PID="$$"
  1415. export PWD="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_02_000002"
  1416. export HADOOP_COMMON_HOME="/usr/local/hadoop"
  1417. export LOCAL_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001"
  1418. export NM_HTTP_PORT="18042"
  1419. export SPARK_DIST_CLASSPATH="/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*"
  1420. export LOG_DIRS="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002"
  1421. export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  1422. "
  1423. export NM_PORT="41471"
  1424. export USER="hduser"
  1425. export HADOOP_YARN_HOME="/usr/local/hadoop"
  1426. export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$PWD/__spark_conf__/__hadoop_conf__"
  1427. export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_02_000002/container_tokens"
  1428. export SPARK_USER="hduser"
  1429. export LOCAL_USER_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/"
  1430. export HADOOP_HOME="/usr/local/hadoop"
  1431. export SPARK_LOG_URL_STDERR="http://bcaf3c62a630:18042/node/containerlogs/container_1520619475936_0001_02_000002/hduser/stderr?start=-4096"
  1432. export HOME="/home/"
  1433. export CONTAINER_ID="container_1520619475936_0001_02_000002"
  1434. export MALLOC_ARENA_MAX="4"
  1435. echo "Setting up job resources"
  1436. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip" "__spark_libs__"
  1437. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar" "__app__.jar"
  1438. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip" "__spark_conf__"
  1439. echo "Copying debugging information"
  1440. # Creating copy of launch script
  1441. cp "launch_container.sh" "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/launch_container.sh"
  1442. chmod 640 "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/launch_container.sh"
  1443. # Determining directory contents
  1444. echo "ls -l:" 1>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/directory.info"
  1445. ls -l 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/directory.info"
  1446. echo "find -L . -maxdepth 5 -ls:" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/directory.info"
  1447. find -L . -maxdepth 5 -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/directory.info"
  1448. echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/directory.info"
  1449. find -L . -maxdepth 5 -type l -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/directory.info"
  1450. echo "Launching container"
  1451. exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx1024m '-XX:MaxGCPauseMillis=950' -Djava.io.tmpdir=$PWD/tmp '-Dspark.driver.port=45035' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@608788f18ea1:45035 --executor-id 1 --hostname bcaf3c62a630 --cores 1 --app-id application_1520619475936_0001 --user-class-path file:$PWD/__app__.jar 1>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/stdout 2>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000002/stderr"
  1452.  
  1453. End of LogType:launch_container.sh
  1454. ************************************************************************************
  1455.  
  1456.  
  1457. End of LogType:prelaunch.err
  1458. ******************************************************************************
  1459.  
  1460. Container: container_1520619475936_0001_02_000002 on bcaf3c62a630_41471
  1461. LogAggregationType: AGGREGATED
  1462. =======================================================================
  1463. LogType:prelaunch.out
  1464. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1465. LogLength:100
  1466. LogContents:
  1467. Setting up env variables
  1468. Setting up job resources
  1469. Copying debugging information
  1470. Launching container
  1471.  
  1472. End of LogType:prelaunch.out
  1473. ******************************************************************************
  1474.  
  1475. Container: container_1520619475936_0001_02_000002 on bcaf3c62a630_41471
  1476. LogAggregationType: AGGREGATED
  1477. =======================================================================
  1478. LogType:stderr
  1479. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1480. LogLength:3371
  1481. LogContents:
  1482. 18/03/09 18:54:21 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 13136@bcaf3c62a630
  1483. 18/03/09 18:54:21 INFO SignalUtils: Registered signal handler for TERM
  1484. 18/03/09 18:54:21 INFO SignalUtils: Registered signal handler for HUP
  1485. 18/03/09 18:54:21 INFO SignalUtils: Registered signal handler for INT
  1486. 18/03/09 18:54:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  1487. 18/03/09 18:54:23 INFO SecurityManager: Changing view acls to: hduser
  1488. 18/03/09 18:54:23 INFO SecurityManager: Changing modify acls to: hduser
  1489. 18/03/09 18:54:23 INFO SecurityManager: Changing view acls groups to:
  1490. 18/03/09 18:54:23 INFO SecurityManager: Changing modify acls groups to:
  1491. 18/03/09 18:54:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  1492. 18/03/09 18:54:23 INFO TransportClientFactory: Successfully created connection to 608788f18ea1/172.22.0.4:45035 after 142 ms (0 ms spent in bootstraps)
  1493. 18/03/09 18:54:24 INFO SecurityManager: Changing view acls to: hduser
  1494. 18/03/09 18:54:24 INFO SecurityManager: Changing modify acls to: hduser
  1495. 18/03/09 18:54:24 INFO SecurityManager: Changing view acls groups to:
  1496. 18/03/09 18:54:24 INFO SecurityManager: Changing modify acls groups to:
  1497. 18/03/09 18:54:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  1498. 18/03/09 18:54:24 INFO TransportClientFactory: Successfully created connection to 608788f18ea1/172.22.0.4:45035 after 16 ms (0 ms spent in bootstraps)
  1499. 18/03/09 18:54:24 INFO DiskBlockManager: Created local directory at /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/blockmgr-1915ccd2-5cab-4ebf-bf9e-62bf0bb8b3e2
  1500. 18/03/09 18:54:24 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
  1501. 18/03/09 18:54:26 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@608788f18ea1:45035
  1502. 18/03/09 18:54:26 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  1503. 18/03/09 18:54:26 INFO Executor: Starting executor ID 1 on host bcaf3c62a630
  1504. 18/03/09 18:54:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36729.
  1505. 18/03/09 18:54:26 INFO NettyBlockTransferService: Server created on bcaf3c62a630:36729
  1506. 18/03/09 18:54:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  1507. 18/03/09 18:54:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, bcaf3c62a630, 36729, None)
  1508. 18/03/09 18:54:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, bcaf3c62a630, 36729, None)
  1509. 18/03/09 18:54:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, bcaf3c62a630, 36729, None)
  1510. 18/03/09 18:54:37 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
  1511. 18/03/09 18:54:37 INFO MemoryStore: MemoryStore cleared
  1512. 18/03/09 18:54:37 INFO BlockManager: BlockManager stopped
  1513. 18/03/09 18:54:37 INFO ShutdownHookManager: Shutdown hook called
  1514.  
  1515. End of LogType:stderr
  1516. ***********************************************************************
  1517.  
  1518.  
  1519. End of LogType:stdout
  1520. ***********************************************************************
  1521.  
  1522. Container: container_1520619475936_0001_02_000003 on bcaf3c62a630_41471
  1523. LogAggregationType: AGGREGATED
  1524. =======================================================================
  1525. LogType:directory.info
  1526. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1527. LogLength:21689
  1528. LogContents:
  1529. ls -l:
  1530. total 36
  1531. lrwxrwxrwx 1 hduser hadoop 89 Mar 9 18:54 __app__.jar -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar
  1532. lrwxrwxrwx 1 hduser hadoop 80 Mar 9 18:54 __spark_conf__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip
  1533. lrwxrwxrwx 1 hduser hadoop 99 Mar 9 18:54 __spark_libs__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip
  1534. -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:54 container_tokens
  1535. -rwx------ 1 hduser hadoop 733 Mar 9 18:54 default_container_executor.sh
  1536. -rwx------ 1 hduser hadoop 678 Mar 9 18:54 default_container_executor_session.sh
  1537. -rwx------ 1 hduser hadoop 6052 Mar 9 18:54 launch_container.sh
  1538. drwx--x--- 2 hduser hadoop 4096 Mar 9 18:54 tmp
  1539. find -L . -maxdepth 5 -ls:
  1540. 1676608 4 drwx--x--- 3 hduser hadoop 4096 Mar 9 18:54 .
  1541. 7089877 4 -rwx------ 1 hduser hadoop 733 Mar 9 18:54 ./default_container_executor.sh
  1542. 7089869 8 -rwx------ 1 hduser hadoop 6052 Mar 9 18:54 ./launch_container.sh
  1543. 1676610 4 drwx--x--- 2 hduser hadoop 4096 Mar 9 18:54 ./tmp
  1544. 7089162 517920 -r-x------ 1 hduser hadoop 530343861 Mar 9 18:53 ./__app__.jar
  1545. 7089878 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:54 ./.default_container_executor.sh.crc
  1546. 1656886 12 drwx------ 2 hduser hadoop 12288 Mar 9 18:52 ./__spark_libs__
  1547. 7089166 88 -r-x------ 1 hduser hadoop 87192 Mar 9 18:52 ./__spark_libs__/spire-macros_2.11-0.13.0.jar
  1548. 7089167 5612 -r-x------ 1 hduser hadoop 5744974 Mar 9 18:52 ./__spark_libs__/scala-library-2.11.8.jar
  1549. 7089168 20 -r-x------ 1 hduser hadoop 19827 Mar 9 18:52 ./__spark_libs__/opencsv-2.3.jar
  1550. 7089169 16 -r-x------ 1 hduser hadoop 13911 Mar 9 18:52 ./__spark_libs__/generex-1.0.1.jar
  1551. 7089170 364 -r-x------ 1 hduser hadoop 370119 Mar 9 18:52 ./__spark_libs__/lz4-java-1.4.0.jar
  1552. 7089171 256 -r-x------ 1 hduser hadoop 258919 Mar 9 18:52 ./__spark_libs__/jackson-core-2.6.7.jar
  1553. 7089172 684 -r-x------ 1 hduser hadoop 698375 Mar 9 18:52 ./__spark_libs__/jersey-common-2.22.2.jar
  1554. 7089173 472 -r-x------ 1 hduser hadoop 479881 Mar 9 18:52 ./__spark_libs__/commons-lang3-3.5.jar
  1555. 7089174 6852 -r-x------ 1 hduser hadoop 7015233 Mar 9 18:52 ./__spark_libs__/kubernetes-model-2.0.0.jar
  1556. 7089175 176 -r-x------ 1 hduser hadoop 178947 Mar 9 18:52 ./__spark_libs__/hk2-api-2.4.0-b34.jar
  1557. 7089176 7172 -r-x------ 1 hduser hadoop 7343426 Mar 9 18:52 ./__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
  1558. 7089177 504 -r-x------ 1 hduser hadoop 515645 Mar 9 18:52 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
  1559. 7089178 44 -r-x------ 1 hduser hadoop 42858 Mar 9 18:52 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar
  1560. 7089179 60 -r-x------ 1 hduser hadoop 60483 Mar 9 18:52 ./__spark_libs__/chill-java-0.8.4.jar
  1561. 7089180 132 -r-x------ 1 hduser hadoop 134595 Mar 9 18:52 ./__spark_libs__/commons-crypto-1.0.0.jar
  1562. 7089181 128 -r-x------ 1 hduser hadoop 130802 Mar 9 18:52 ./__spark_libs__/aircompressor-0.8.jar
  1563. 7089182 264 -r-x------ 1 hduser hadoop 269295 Mar 9 18:52 ./__spark_libs__/snakeyaml-1.15.jar
  1564. 7089183 1168 -r-x------ 1 hduser hadoop 1194003 Mar 9 18:52 ./__spark_libs__/arpack_combined_all-0.1.jar
  1565. 7089184 14760 -r-x------ 1 hduser hadoop 15113382 Mar 9 18:52 ./__spark_libs__/breeze_2.11-0.13.2.jar
  1566. 7089185 36 -r-x------ 1 hduser hadoop 35518 Mar 9 18:52 ./__spark_libs__/zjsonpatch-0.3.0.jar
  1567. 7089186 120 -r-x------ 1 hduser hadoop 120321 Mar 9 18:52 ./__spark_libs__/spark-repl_2.11-2.3.0.jar
  1568. 7089187 36 -r-x------ 1 hduser hadoop 34603 Mar 9 18:52 ./__spark_libs__/machinist_2.11-0.6.1.jar
  1569. 7089188 28 -r-x------ 1 hduser hadoop 26366 Mar 9 18:52 ./__spark_libs__/javax.annotation-api-1.2.jar
  1570. 7089189 180 -r-x------ 1 hduser hadoop 180736 Mar 9 18:52 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
  1571. 7089190 788 -r-x------ 1 hduser hadoop 802818 Mar 9 18:52 ./__spark_libs__/scalap-2.11.8.jar
  1572. 7089191 1504 -r-x------ 1 hduser hadoop 1537966 Mar 9 18:52 ./__spark_libs__/hadoop-mapreduce-client-core-2.6.5.jar
  1573. 7089192 1024 -r-x------ 1 hduser hadoop 1048116 Mar 9 18:52 ./__spark_libs__/parquet-jackson-1.8.2.jar
  1574. 7089193 172 -r-x------ 1 hduser hadoop 174351 Mar 9 18:52 ./__spark_libs__/stream-2.7.0.jar
  1575. 7089194 68 -r-x------ 1 hduser hadoop 66270 Mar 9 18:52 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar
  1576. 7089195 1632 -r-x------ 1 hduser hadoop 1671083 Mar 9 18:52 ./__spark_libs__/hppc-0.7.2.jar
  1577. 7089196 180 -r-x------ 1 hduser hadoop 183832 Mar 9 18:52 ./__spark_libs__/spark-mllib-local_2.11-2.3.0.jar
  1578. 7089197 656 -r-x------ 1 hduser hadoop 671138 Mar 9 18:52 ./__spark_libs__/scala-xml_2.11-1.0.5.jar
  1579. 7089198 200 -r-x------ 1 hduser hadoop 201928 Mar 9 18:52 ./__spark_libs__/RoaringBitmap-0.5.11.jar
  1580. 7089199 96 -r-x------ 1 hduser hadoop 94796 Mar 9 18:52 ./__spark_libs__/pyrolite-4.13.jar
  1581. 7089200 3692 -r-x------ 1 hduser hadoop 3780056 Mar 9 18:52 ./__spark_libs__/netty-all-4.1.17.Final.jar
  1582. 7089201 316 -r-x------ 1 hduser hadoop 320444 Mar 9 18:52 ./__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
  1583. 7089202 1024 -r-x------ 1 hduser hadoop 1045744 Mar 9 18:52 ./__spark_libs__/leveldbjni-all-1.8.jar
  1584. 7089203 376 -r-x------ 1 hduser hadoop 381098 Mar 9 18:52 ./__spark_libs__/spark-kubernetes_2.11-2.3.0.jar
  1585. 7089204 32 -r-x------ 1 hduser hadoop 32612 Mar 9 18:52 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
  1586. 7089205 636 -r-x------ 1 hduser hadoop 650236 Mar 9 18:52 ./__spark_libs__/spark-yarn_2.11-2.3.0.jar
  1587. 7089206 48 -r-x------ 1 hduser hadoop 46986 Mar 9 18:52 ./__spark_libs__/jackson-annotations-2.6.7.jar
  1588. 7089207 700 -r-x------ 1 hduser hadoop 714194 Mar 9 18:52 ./__spark_libs__/javassist-3.18.1-GA.jar
  1589. 7089208 248 -r-x------ 1 hduser hadoop 250377 Mar 9 18:52 ./__spark_libs__/parquet-hadoop-1.8.2.jar
  1590. 7089209 64 -r-x------ 1 hduser hadoop 64847 Mar 9 18:52 ./__spark_libs__/spark-network-shuffle_2.11-2.3.0.jar
  1591. 7089210 116 -r-x------ 1 hduser hadoop 116859 Mar 9 18:52 ./__spark_libs__/py4j-0.10.6.jar
  1592. 7089211 76 -r-x------ 1 hduser hadoop 75835 Mar 9 18:52 ./__spark_libs__/spark-launcher_2.11-2.3.0.jar
  1593. 7089212 20 -r-x------ 1 hduser hadoop 16430 Mar 9 18:52 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar
  1594. 7089213 52 -r-x------ 1 hduser hadoop 52076 Mar 9 18:52 ./__spark_libs__/arrow-format-0.8.0.jar
  1595. 7089214 84 -r-x------ 1 hduser hadoop 82421 Mar 9 18:52 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar
  1596. 7089215 740 -r-x------ 1 hduser hadoop 757423 Mar 9 18:52 ./__spark_libs__/orc-mapreduce-1.4.1-nohive.jar
  1597. 7089216 1300 -r-x------ 1 hduser hadoop 1330219 Mar 9 18:52 ./__spark_libs__/netty-3.9.9.Final.jar
  1598. 7089217 120 -r-x------ 1 hduser hadoop 118973 Mar 9 18:52 ./__spark_libs__/hk2-utils-2.4.0-b34.jar
  1599. 7089218 408 -r-x------ 1 hduser hadoop 416731 Mar 9 18:52 ./__spark_libs__/kubernetes-client-3.0.0.jar
  1600. 7089219 96 -r-x------ 1 hduser hadoop 95806 Mar 9 18:52 ./__spark_libs__/javax.servlet-api-3.1.0.jar
  1601. 7089220 936 -r-x------ 1 hduser hadoop 956984 Mar 9 18:52 ./__spark_libs__/parquet-column-1.8.2.jar
  1602. 7089221 1412 -r-x------ 1 hduser hadoop 1441994 Mar 9 18:52 ./__spark_libs__/orc-core-1.4.1-nohive.jar
  1603. 7089222 576 -r-x------ 1 hduser hadoop 589462 Mar 9 18:52 ./__spark_libs__/json4s-core_2.11-3.2.11.jar
  1604. 7089223 2328 -r-x------ 1 hduser hadoop 2382165 Mar 9 18:52 ./__spark_libs__/spark-network-common_2.11-2.3.0.jar
  1605. 7089224 12 -r-x------ 1 hduser hadoop 8268 Mar 9 18:52 ./__spark_libs__/logging-interceptor-3.8.1.jar
  1606. 7089225 40 -r-x------ 1 hduser hadoop 40341 Mar 9 18:52 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar
  1607. 7089226 52 -r-x------ 1 hduser hadoop 53007 Mar 9 18:52 ./__spark_libs__/spark-kvstore_2.11-2.3.0.jar
  1608. 7089227 8 -r-x------ 1 hduser hadoop 5950 Mar 9 18:52 ./__spark_libs__/javax.inject-2.4.0-b34.jar
  1609. 7089228 376 -r-x------ 1 hduser hadoop 384150 Mar 9 18:52 ./__spark_libs__/univocity-parsers-2.5.9.jar
  1610. 7089229 7560 -r-x------ 1 hduser hadoop 7738527 Mar 9 18:52 ./__spark_libs__/spark-mllib_2.11-2.3.0.jar
  1611. 7089230 116 -r-x------ 1 hduser hadoop 115534 Mar 9 18:52 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar
  1612. 7089231 72 -r-x------ 1 hduser hadoop 72733 Mar 9 18:52 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar
  1613. 7089232 748 -r-x------ 1 hduser hadoop 764569 Mar 9 18:52 ./__spark_libs__/jtransforms-2.4.0.jar
  1614. 7089233 12 -r-x------ 1 hduser hadoop 10166 Mar 9 18:52 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
  1615. 7089234 36 -r-x------ 1 hduser hadoop 34654 Mar 9 18:52 ./__spark_libs__/paranamer-2.8.jar
  1616. 7089235 8464 -r-x------ 1 hduser hadoop 8664987 Mar 9 18:52 ./__spark_libs__/spark-sql_2.11-2.3.0.jar
  1617. 7089236 164 -r-x------ 1 hduser hadoop 164422 Mar 9 18:52 ./__spark_libs__/core-1.1.2.jar
  1618. 7089237 44 -r-x------ 1 hduser hadoop 42032 Mar 9 18:52 ./__spark_libs__/parquet-common-1.8.2.jar
  1619. 7089238 36 -r-x------ 1 hduser hadoop 33015 Mar 9 18:52 ./__spark_libs__/jsr305-1.3.9.jar
  1620. 7089239 80 -r-x------ 1 hduser hadoop 81811 Mar 9 18:52 ./__spark_libs__/okio-1.13.0.jar
  1621. 7089240 328 -r-x------ 1 hduser hadoop 334662 Mar 9 18:52 ./__spark_libs__/antlr4-runtime-4.7.jar
  1622. 7089241 392 -r-x------ 1 hduser hadoop 398122 Mar 9 18:52 ./__spark_libs__/okhttp-3.8.1.jar
  1623. 7089242 352 -r-x------ 1 hduser hadoop 358390 Mar 9 18:52 ./__spark_libs__/kryo-shaded-3.0.3.jar
  1624. 7089243 80 -r-x------ 1 hduser hadoop 79150 Mar 9 18:52 ./__spark_libs__/arrow-memory-0.8.0.jar
  1625. 7089244 15128 -r-x------ 1 hduser hadoop 15487351 Mar 9 18:52 ./__spark_libs__/scala-compiler-2.11.8.jar
  1626. 7089245 164 -r-x------ 1 hduser hadoop 167421 Mar 9 18:52 ./__spark_libs__/jersey-client-2.22.2.jar
  1627. 7089246 384 -r-x------ 1 hduser hadoop 390733 Mar 9 18:52 ./__spark_libs__/parquet-format-2.3.1.jar
  1628. 7089247 64 -r-x------ 1 hduser hadoop 63777 Mar 9 18:52 ./__spark_libs__/validation-api-1.1.0.Final.jar
  1629. 7089248 4468 -r-x------ 1 hduser hadoop 4573750 Mar 9 18:52 ./__spark_libs__/scala-reflect-2.11.8.jar
  1630. 7089249 932 -r-x------ 1 hduser hadoop 951701 Mar 9 18:52 ./__spark_libs__/jersey-server-2.22.2.jar
  1631. 7089250 48 -r-x------ 1 hduser hadoop 48464 Mar 9 18:52 ./__spark_libs__/spark-unsafe_2.11-2.3.0.jar
  1632. 7089251 184 -r-x------ 1 hduser hadoop 187882 Mar 9 18:52 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar
  1633. 7089252 3444 -r-x------ 1 hduser hadoop 3522616 Mar 9 18:52 ./__spark_libs__/shapeless_2.11-2.3.2.jar
  1634. 7089253 8 -r-x------ 1 hduser hadoop 5711 Mar 9 18:52 ./__spark_libs__/minlog-1.3.0.jar
  1635. 7089254 20 -r-x------ 1 hduser hadoop 18098 Mar 9 18:52 ./__spark_libs__/jersey-container-servlet-2.22.2.jar
  1636. 7089255 40 -r-x------ 1 hduser hadoop 37887 Mar 9 18:52 ./__spark_libs__/commons-compiler-3.0.8.jar
  1637. 7089256 44 -r-x------ 1 hduser hadoop 41755 Mar 9 18:52 ./__spark_libs__/objenesis-2.1.jar
  1638. 7089257 280 -r-x------ 1 hduser hadoop 286071 Mar 9 18:52 ./__spark_libs__/parquet-encoding-1.8.2.jar
  1639. 7089258 220 -r-x------ 1 hduser hadoop 224167 Mar 9 18:52 ./__spark_libs__/chill_2.11-0.8.4.jar
  1640. 7089259 780 -r-x------ 1 hduser hadoop 796326 Mar 9 18:52 ./__spark_libs__/janino-3.0.8.jar
  1641. 7089260 1244 -r-x------ 1 hduser hadoop 1270377 Mar 9 18:52 ./__spark_libs__/arrow-vector-0.8.0.jar
  1642. 7089261 24 -r-x------ 1 hduser hadoop 21247 Mar 9 18:52 ./__spark_libs__/metrics-graphite-3.1.5.jar
  1643. 7089262 1988 -r-x------ 1 hduser hadoop 2035066 Mar 9 18:52 ./__spark_libs__/commons-math3-3.4.1.jar
  1644. 7089263 32 -r-x------ 1 hduser hadoop 30057 Mar 9 18:52 ./__spark_libs__/spark-sketch_2.11-2.3.0.jar
  1645. 7089264 2120 -r-x------ 1 hduser hadoop 2169825 Mar 9 18:52 ./__spark_libs__/spark-streaming_2.11-2.3.0.jar
  1646. 7089265 9888 -r-x------ 1 hduser hadoop 10121868 Mar 9 18:52 ./__spark_libs__/spire_2.11-0.13.0.jar
  1647. 7089266 8 -r-x------ 1 hduser hadoop 4596 Mar 9 18:52 ./__spark_libs__/jul-to-slf4j-1.7.16.jar
  1648. 7089267 20 -r-x------ 1 hduser hadoop 20235 Mar 9 18:52 ./__spark_libs__/osgi-resource-locator-1.0.1.jar
  1649. 7089268 4 -r-x------ 1 hduser hadoop 3142 Mar 9 18:52 ./__spark_libs__/macro-compat_2.11-1.1.1.jar
  1650. 7089269 280 -r-x------ 1 hduser hadoop 284220 Mar 9 18:52 ./__spark_libs__/commons-lang-2.6.jar
  1651. 7089270 1256 -r-x------ 1 hduser hadoop 1282424 Mar 9 18:52 ./__spark_libs__/ivy-2.4.0.jar
  1652. 7089271 80 -r-x------ 1 hduser hadoop 79845 Mar 9 18:52 ./__spark_libs__/compress-lzf-1.0.3.jar
  1653. 7089272 16 -r-x------ 1 hduser hadoop 15824 Mar 9 18:52 ./__spark_libs__/metrics-json-3.1.5.jar
  1654. 7089273 12680 -r-x------ 1 hduser hadoop 12982716 Mar 9 18:52 ./__spark_libs__/spark-core_2.11-2.3.0.jar
  1655. 7089274 180 -r-x------ 1 hduser hadoop 181271 Mar 9 18:52 ./__spark_libs__/hk2-locator-2.4.0-b34.jar
  1656. 7089275 416 -r-x------ 1 hduser hadoop 423753 Mar 9 18:52 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
  1657. 7089276 1140 -r-x------ 1 hduser hadoop 1165323 Mar 9 18:52 ./__spark_libs__/jackson-databind-2.6.7.1.jar
  1658. 7089277 208 -r-x------ 1 hduser hadoop 212453 Mar 9 18:52 ./__spark_libs__/commons-net-2.2.jar
  1659. 7089278 64 -r-x------ 1 hduser hadoop 65261 Mar 9 18:52 ./__spark_libs__/oro-2.0.8.jar
  1660. 7089279 144 -r-x------ 1 hduser hadoop 144660 Mar 9 18:52 ./__spark_libs__/xbean-asm5-shaded-4.4.jar
  1661. 7089280 8776 -r-x------ 1 hduser hadoop 8984422 Mar 9 18:52 ./__spark_libs__/spark-catalyst_2.11-2.3.0.jar
  1662. 7089281 40 -r-x------ 1 hduser hadoop 39283 Mar 9 18:52 ./__spark_libs__/metrics-jvm-3.1.5.jar
  1663. 7089282 280 -r-x------ 1 hduser hadoop 284184 Mar 9 18:52 ./__spark_libs__/commons-codec-1.10.jar
  1664. 7089283 176 -r-x------ 1 hduser hadoop 176285 Mar 9 18:52 ./__spark_libs__/automaton-1.11-8.jar
  1665. 7089284 2280 -r-x------ 1 hduser hadoop 2333186 Mar 9 18:52 ./__spark_libs__/zstd-jni-1.3.2-2.jar
  1666. 7089285 648 -r-x------ 1 hduser hadoop 663328 Mar 9 18:52 ./__spark_libs__/spark-mesos_2.11-2.3.0.jar
  1667. 7089286 696 -r-x------ 1 hduser hadoop 708667 Mar 9 18:52 ./__spark_libs__/spark-graphx_2.11-2.3.0.jar
  1668. 7089287 16 -r-x------ 1 hduser hadoop 15456 Mar 9 18:52 ./__spark_libs__/spark-tags_2.11-2.3.0.jar
  1669. 7089288 616 -r-x------ 1 hduser hadoop 627814 Mar 9 18:52 ./__spark_libs__/joda-time-2.9.3.jar
  1670. 7089289 120 -r-x------ 1 hduser hadoop 120465 Mar 9 18:52 ./__spark_libs__/metrics-core-3.1.5.jar
  1671. 7089290 16 -r-x------ 1 hduser hadoop 14766 Mar 9 18:52 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
  1672. 7089291 952 -r-x------ 1 hduser hadoop 971310 Mar 9 18:52 ./__spark_libs__/jersey-guava-2.22.2.jar
  1673. 7089871 4 -rw-r--r-- 1 hduser hadoop 56 Mar 9 18:54 ./.launch_container.sh.crc
  1674. 7089864 4 -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:54 ./container_tokens
  1675. 7089876 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:54 ./.default_container_executor_session.sh.crc
  1676. 7089873 4 -rwx------ 1 hduser hadoop 678 Mar 9 18:54 ./default_container_executor_session.sh
  1677. 1653456 4 drwx------ 3 hduser hadoop 4096 Mar 9 18:52 ./__spark_conf__
  1678. 7089340 4 -r-x------ 1 hduser hadoop 1192 Mar 9 18:52 ./__spark_conf__/__spark_conf__.properties
  1679. 1653459 4 drwx------ 2 hduser hadoop 4096 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__
  1680. 7089323 4 -r-x------ 1 hduser hadoop 2316 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example
  1681. 7089327 4 -r-x------ 1 hduser hadoop 891 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/slaves
  1682. 7089338 4 -r-x------ 1 hduser hadoop 682 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-site.xml
  1683. 7089332 4 -r-x------ 1 hduser hadoop 996 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/docker.properties.template
  1684. 7089319 4 -r-x------ 1 hduser hadoop 1764 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-env.sh
  1685. 7089312 4 -r-x------ 1 hduser hadoop 3518 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-acls.xml
  1686. 7089337 8 -r-x------ 1 hduser hadoop 4113 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template
  1687. 7089329 4 -r-x------ 1 hduser hadoop 21 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-signature.secret
  1688. 7089328 4 -r-x------ 1 hduser hadoop 1484 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-env.sh
  1689. 7089314 4 -r-x------ 1 hduser hadoop 482 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-env.sh
  1690. 7089303 4 -r-x------ 1 hduser hadoop 3670 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd
  1691. 7089299 4 -r-x------ 1 hduser hadoop 2016 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.spark.properties
  1692. 7089308 8 -r-x------ 1 hduser hadoop 4221 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-env.sh.template
  1693. 7089298 4 -r-x------ 1 hduser hadoop 137 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
  1694. 7089322 4 -r-x------ 1 hduser hadoop 3414 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-user-functions.sh.example
  1695. 7089310 4 -r-x------ 1 hduser hadoop 2250 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd
  1696. 7089317 4 -r-x------ 1 hduser hadoop 589 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/container-executor.cfg
  1697. 7089307 4 -r-x------ 1 hduser hadoop 890 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/workers
  1698. 7089336 4 -r-x------ 1 hduser hadoop 2697 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example
  1699. 7089300 4 -r-x------ 1 hduser hadoop 1657 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-log4j.properties
  1700. 7089320 8 -r-x------ 1 hduser hadoop 7801 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/metrics.properties.template
  1701. 7089318 4 -r-x------ 1 hduser hadoop 1781 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf
  1702. 7089306 4 -r-x------ 1 hduser hadoop 225 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/core-site.xml
  1703. 7089301 4 -r-x------ 1 hduser hadoop 3323 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties
  1704. 7089311 8 -r-x------ 1 hduser hadoop 7861 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml
  1705. 7089334 4 -r-x------ 1 hduser hadoop 1292 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf.template
  1706. 7089321 4 -r-x------ 1 hduser hadoop 2642 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/user_ec_policies.xml.template
  1707. 7089333 4 -r-x------ 1 hduser hadoop 951 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd
  1708. 7089330 4 -r-x------ 1 hduser hadoop 620 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/httpfs-site.xml
  1709. 7089335 4 -r-x------ 1 hduser hadoop 3589 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j2.xml
  1710. 7089325 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.properties.template
  1711. 7089305 4 -r-x------ 1 hduser hadoop 865 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/slaves.template
  1712. 7089316 4 -r-x------ 1 hduser hadoop 809 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
  1713. 7089297 16 -r-x------ 1 hduser hadoop 16270 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh
  1714. 7089313 4 -r-x------ 1 hduser hadoop 1335 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/configuration.xsl
  1715. 7089302 8 -r-x------ 1 hduser hadoop 8048 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
  1716. 7089296 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/log4j.properties
  1717. 7089324 4 -r-x------ 1 hduser hadoop 1351 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-env.sh
  1718. 7089309 4 -r-x------ 1 hduser hadoop 1747 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties
  1719. 7089326 12 -r-x------ 1 hduser hadoop 10206 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml
  1720. 7089331 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/metrics.properties
  1721. 7089315 8 -r-x------ 1 hduser hadoop 5502 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/yarn-env.sh
  1722. 7089304 4 -r-x------ 1 hduser hadoop 1105 Mar 9 18:52 ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.template
  1723. 7089293 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:52 ./__spark_conf__/log4j.properties
  1724. 7089339 172 -r-x------ 1 hduser hadoop 173257 Mar 9 18:52 ./__spark_conf__/__spark_hadoop_conf__.xml
  1725. 7089294 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:52 ./__spark_conf__/metrics.properties
  1726. 7089866 4 -rw-r--r-- 1 hduser hadoop 12 Mar 9 18:54 ./.container_tokens.crc
  1727. broken symlinks(find -L . -maxdepth 5 -type l -ls):
  1728.  
  1729. End of LogType:directory.info
  1730. *******************************************************************************
  1731.  
  1732. Container: container_1520619475936_0001_02_000003 on bcaf3c62a630_41471
  1733. LogAggregationType: AGGREGATED
  1734. =======================================================================
  1735. LogType:launch_container.sh
  1736. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1737. LogLength:6052
  1738. LogContents:
  1739. #!/bin/bash
  1740.  
  1741. set -o pipefail -e
  1742. export PRELAUNCH_OUT="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/prelaunch.out"
  1743. exec >"${PRELAUNCH_OUT}"
  1744. export PRELAUNCH_ERR="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/prelaunch.err"
  1745. exec 2>"${PRELAUNCH_ERR}"
  1746. echo "Setting up env variables"
  1747. export SPARK_YARN_STAGING_DIR="hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001"
  1748. export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
  1749. export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
  1750. export JAVA_HOME="/usr/java/default"
  1751. export SPARK_LOG_URL_STDOUT="http://bcaf3c62a630:18042/node/containerlogs/container_1520619475936_0001_02_000003/hduser/stdout?start=-4096"
  1752. export NM_HOST="bcaf3c62a630"
  1753. export HADOOP_HDFS_HOME="/usr/local/hadoop"
  1754. export LOGNAME="hduser"
  1755. export JVM_PID="$$"
  1756. export PWD="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_02_000003"
  1757. export HADOOP_COMMON_HOME="/usr/local/hadoop"
  1758. export LOCAL_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001"
  1759. export NM_HTTP_PORT="18042"
  1760. export SPARK_DIST_CLASSPATH="/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*"
  1761. export LOG_DIRS="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003"
  1762. export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  1763. "
  1764. export NM_PORT="41471"
  1765. export USER="hduser"
  1766. export HADOOP_YARN_HOME="/usr/local/hadoop"
  1767. export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$PWD/__spark_conf__/__hadoop_conf__"
  1768. export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_02_000003/container_tokens"
  1769. export SPARK_USER="hduser"
  1770. export LOCAL_USER_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/"
  1771. export HADOOP_HOME="/usr/local/hadoop"
  1772. export SPARK_LOG_URL_STDERR="http://bcaf3c62a630:18042/node/containerlogs/container_1520619475936_0001_02_000003/hduser/stderr?start=-4096"
  1773. export HOME="/home/"
  1774. export CONTAINER_ID="container_1520619475936_0001_02_000003"
  1775. export MALLOC_ARENA_MAX="4"
  1776. echo "Setting up job resources"
  1777. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip" "__spark_libs__"
  1778. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar" "__app__.jar"
  1779. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip" "__spark_conf__"
  1780. echo "Copying debugging information"
  1781. # Creating copy of launch script
  1782. cp "launch_container.sh" "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/launch_container.sh"
  1783. chmod 640 "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/launch_container.sh"
  1784. # Determining directory contents
  1785. echo "ls -l:" 1>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/directory.info"
  1786. ls -l 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/directory.info"
  1787. echo "find -L . -maxdepth 5 -ls:" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/directory.info"
  1788. find -L . -maxdepth 5 -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/directory.info"
  1789. echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/directory.info"
  1790. find -L . -maxdepth 5 -type l -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/directory.info"
  1791. echo "Launching container"
  1792. exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx1024m '-XX:MaxGCPauseMillis=950' -Djava.io.tmpdir=$PWD/tmp '-Dspark.driver.port=45035' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@608788f18ea1:45035 --executor-id 2 --hostname bcaf3c62a630 --cores 1 --app-id application_1520619475936_0001 --user-class-path file:$PWD/__app__.jar 1>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/stdout 2>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_02_000003/stderr"
  1793.  
  1794. End of LogType:launch_container.sh
  1795. ************************************************************************************
  1796.  
  1797.  
  1798. End of LogType:prelaunch.err
  1799. ******************************************************************************
  1800.  
  1801. Container: container_1520619475936_0001_02_000003 on bcaf3c62a630_41471
  1802. LogAggregationType: AGGREGATED
  1803. =======================================================================
  1804. LogType:prelaunch.out
  1805. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1806. LogLength:100
  1807. LogContents:
  1808. Setting up env variables
  1809. Setting up job resources
  1810. Copying debugging information
  1811. Launching container
  1812.  
  1813. End of LogType:prelaunch.out
  1814. ******************************************************************************
  1815.  
  1816. Container: container_1520619475936_0001_02_000003 on bcaf3c62a630_41471
  1817. LogAggregationType: AGGREGATED
  1818. =======================================================================
  1819. LogType:stderr
  1820. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1821. LogLength:3371
  1822. LogContents:
  1823. 18/03/09 18:54:22 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 13137@bcaf3c62a630
  1824. 18/03/09 18:54:22 INFO SignalUtils: Registered signal handler for TERM
  1825. 18/03/09 18:54:22 INFO SignalUtils: Registered signal handler for HUP
  1826. 18/03/09 18:54:22 INFO SignalUtils: Registered signal handler for INT
  1827. 18/03/09 18:54:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  1828. 18/03/09 18:54:23 INFO SecurityManager: Changing view acls to: hduser
  1829. 18/03/09 18:54:23 INFO SecurityManager: Changing modify acls to: hduser
  1830. 18/03/09 18:54:23 INFO SecurityManager: Changing view acls groups to:
  1831. 18/03/09 18:54:23 INFO SecurityManager: Changing modify acls groups to:
  1832. 18/03/09 18:54:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  1833. 18/03/09 18:54:23 INFO TransportClientFactory: Successfully created connection to 608788f18ea1/172.22.0.4:45035 after 151 ms (0 ms spent in bootstraps)
  1834. 18/03/09 18:54:24 INFO SecurityManager: Changing view acls to: hduser
  1835. 18/03/09 18:54:24 INFO SecurityManager: Changing modify acls to: hduser
  1836. 18/03/09 18:54:24 INFO SecurityManager: Changing view acls groups to:
  1837. 18/03/09 18:54:24 INFO SecurityManager: Changing modify acls groups to:
  1838. 18/03/09 18:54:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  1839. 18/03/09 18:54:24 INFO TransportClientFactory: Successfully created connection to 608788f18ea1/172.22.0.4:45035 after 24 ms (0 ms spent in bootstraps)
  1840. 18/03/09 18:54:24 INFO DiskBlockManager: Created local directory at /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/blockmgr-a104e028-90dc-4802-bcb3-89cc87ebee2b
  1841. 18/03/09 18:54:24 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
  1842. 18/03/09 18:54:26 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@608788f18ea1:45035
  1843. 18/03/09 18:54:26 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  1844. 18/03/09 18:54:26 INFO Executor: Starting executor ID 2 on host bcaf3c62a630
  1845. 18/03/09 18:54:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38431.
  1846. 18/03/09 18:54:26 INFO NettyBlockTransferService: Server created on bcaf3c62a630:38431
  1847. 18/03/09 18:54:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  1848. 18/03/09 18:54:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, bcaf3c62a630, 38431, None)
  1849. 18/03/09 18:54:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, bcaf3c62a630, 38431, None)
  1850. 18/03/09 18:54:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, bcaf3c62a630, 38431, None)
  1851. 18/03/09 18:54:37 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
  1852. 18/03/09 18:54:37 INFO MemoryStore: MemoryStore cleared
  1853. 18/03/09 18:54:37 INFO BlockManager: BlockManager stopped
  1854. 18/03/09 18:54:37 INFO ShutdownHookManager: Shutdown hook called
  1855.  
  1856. End of LogType:stderr
  1857. ***********************************************************************
  1858.  
  1859.  
  1860. End of LogType:stdout
  1861. ***********************************************************************
  1862.  
  1863. Container: container_1520619475936_0001_01_000003 on bf263d1bde4d_41271
  1864. LogAggregationType: AGGREGATED
  1865. =======================================================================
  1866. LogType:directory.info
  1867. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  1868. LogLength:21689
  1869. LogContents:
  1870. ls -l:
  1871. total 36
  1872. lrwxrwxrwx 1 hduser hadoop 89 Mar 9 18:53 __app__.jar -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar
  1873. lrwxrwxrwx 1 hduser hadoop 80 Mar 9 18:53 __spark_conf__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip
  1874. lrwxrwxrwx 1 hduser hadoop 99 Mar 9 18:53 __spark_libs__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip
  1875. -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:53 container_tokens
  1876. -rwx------ 1 hduser hadoop 733 Mar 9 18:53 default_container_executor.sh
  1877. -rwx------ 1 hduser hadoop 678 Mar 9 18:53 default_container_executor_session.sh
  1878. -rwx------ 1 hduser hadoop 6052 Mar 9 18:53 launch_container.sh
  1879. drwx--x--- 2 hduser hadoop 4096 Mar 9 18:53 tmp
  1880. find -L . -maxdepth 5 -ls:
  1881. 1658300 4 drwx--x--- 3 hduser hadoop 4096 Mar 9 18:53 .
  1882. 7089592 4 -rwx------ 1 hduser hadoop 733 Mar 9 18:53 ./default_container_executor.sh
  1883. 7089585 8 -rwx------ 1 hduser hadoop 6052 Mar 9 18:53 ./launch_container.sh
  1884. 1658302 4 drwx--x--- 2 hduser hadoop 4096 Mar 9 18:53 ./tmp
  1885. 7089389 517916 -r-x------ 1 hduser hadoop 530343861 Mar 9 18:53 ./__app__.jar
  1886. 7089594 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:53 ./.default_container_executor.sh.crc
  1887. 1654426 12 drwx------ 2 hduser hadoop 12288 Mar 9 18:53 ./__spark_libs__
  1888. 7089441 88 -r-x------ 1 hduser hadoop 87192 Mar 9 18:53 ./__spark_libs__/spire-macros_2.11-0.13.0.jar
  1889. 7089442 5612 -r-x------ 1 hduser hadoop 5744974 Mar 9 18:53 ./__spark_libs__/scala-library-2.11.8.jar
  1890. 7089443 20 -r-x------ 1 hduser hadoop 19827 Mar 9 18:53 ./__spark_libs__/opencsv-2.3.jar
  1891. 7089444 16 -r-x------ 1 hduser hadoop 13911 Mar 9 18:53 ./__spark_libs__/generex-1.0.1.jar
  1892. 7089445 364 -r-x------ 1 hduser hadoop 370119 Mar 9 18:53 ./__spark_libs__/lz4-java-1.4.0.jar
  1893. 7089446 256 -r-x------ 1 hduser hadoop 258919 Mar 9 18:53 ./__spark_libs__/jackson-core-2.6.7.jar
  1894. 7089447 684 -r-x------ 1 hduser hadoop 698375 Mar 9 18:53 ./__spark_libs__/jersey-common-2.22.2.jar
  1895. 7089448 472 -r-x------ 1 hduser hadoop 479881 Mar 9 18:53 ./__spark_libs__/commons-lang3-3.5.jar
  1896. 7089449 6852 -r-x------ 1 hduser hadoop 7015233 Mar 9 18:53 ./__spark_libs__/kubernetes-model-2.0.0.jar
  1897. 7089450 176 -r-x------ 1 hduser hadoop 178947 Mar 9 18:53 ./__spark_libs__/hk2-api-2.4.0-b34.jar
  1898. 7089451 7172 -r-x------ 1 hduser hadoop 7343426 Mar 9 18:53 ./__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
  1899. 7089452 504 -r-x------ 1 hduser hadoop 515645 Mar 9 18:53 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
  1900. 7089453 44 -r-x------ 1 hduser hadoop 42858 Mar 9 18:53 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar
  1901. 7089454 60 -r-x------ 1 hduser hadoop 60483 Mar 9 18:53 ./__spark_libs__/chill-java-0.8.4.jar
  1902. 7089455 132 -r-x------ 1 hduser hadoop 134595 Mar 9 18:53 ./__spark_libs__/commons-crypto-1.0.0.jar
  1903. 7089456 128 -r-x------ 1 hduser hadoop 130802 Mar 9 18:53 ./__spark_libs__/aircompressor-0.8.jar
  1904. 7089457 264 -r-x------ 1 hduser hadoop 269295 Mar 9 18:53 ./__spark_libs__/snakeyaml-1.15.jar
  1905. 7089458 1168 -r-x------ 1 hduser hadoop 1194003 Mar 9 18:53 ./__spark_libs__/arpack_combined_all-0.1.jar
  1906. 7089459 14760 -r-x------ 1 hduser hadoop 15113382 Mar 9 18:53 ./__spark_libs__/breeze_2.11-0.13.2.jar
  1907. 7089460 36 -r-x------ 1 hduser hadoop 35518 Mar 9 18:53 ./__spark_libs__/zjsonpatch-0.3.0.jar
  1908. 7089461 120 -r-x------ 1 hduser hadoop 120321 Mar 9 18:53 ./__spark_libs__/spark-repl_2.11-2.3.0.jar
  1909. 7089462 36 -r-x------ 1 hduser hadoop 34603 Mar 9 18:53 ./__spark_libs__/machinist_2.11-0.6.1.jar
  1910. 7089463 28 -r-x------ 1 hduser hadoop 26366 Mar 9 18:53 ./__spark_libs__/javax.annotation-api-1.2.jar
  1911. 7089464 180 -r-x------ 1 hduser hadoop 180736 Mar 9 18:53 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
  1912. 7089465 788 -r-x------ 1 hduser hadoop 802818 Mar 9 18:53 ./__spark_libs__/scalap-2.11.8.jar
  1913. 7089466 1504 -r-x------ 1 hduser hadoop 1537966 Mar 9 18:53 ./__spark_libs__/hadoop-mapreduce-client-core-2.6.5.jar
  1914. 7089467 1024 -r-x------ 1 hduser hadoop 1048116 Mar 9 18:53 ./__spark_libs__/parquet-jackson-1.8.2.jar
  1915. 7089468 172 -r-x------ 1 hduser hadoop 174351 Mar 9 18:53 ./__spark_libs__/stream-2.7.0.jar
  1916. 7089469 68 -r-x------ 1 hduser hadoop 66270 Mar 9 18:53 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar
  1917. 7089470 1632 -r-x------ 1 hduser hadoop 1671083 Mar 9 18:53 ./__spark_libs__/hppc-0.7.2.jar
  1918. 7089471 180 -r-x------ 1 hduser hadoop 183832 Mar 9 18:53 ./__spark_libs__/spark-mllib-local_2.11-2.3.0.jar
  1919. 7089472 656 -r-x------ 1 hduser hadoop 671138 Mar 9 18:53 ./__spark_libs__/scala-xml_2.11-1.0.5.jar
  1920. 7089473 200 -r-x------ 1 hduser hadoop 201928 Mar 9 18:53 ./__spark_libs__/RoaringBitmap-0.5.11.jar
  1921. 7089474 96 -r-x------ 1 hduser hadoop 94796 Mar 9 18:53 ./__spark_libs__/pyrolite-4.13.jar
  1922. 7089475 3692 -r-x------ 1 hduser hadoop 3780056 Mar 9 18:53 ./__spark_libs__/netty-all-4.1.17.Final.jar
  1923. 7089476 316 -r-x------ 1 hduser hadoop 320444 Mar 9 18:53 ./__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
  1924. 7089477 1024 -r-x------ 1 hduser hadoop 1045744 Mar 9 18:53 ./__spark_libs__/leveldbjni-all-1.8.jar
  1925. 7089478 376 -r-x------ 1 hduser hadoop 381098 Mar 9 18:53 ./__spark_libs__/spark-kubernetes_2.11-2.3.0.jar
  1926. 7089479 32 -r-x------ 1 hduser hadoop 32612 Mar 9 18:53 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
  1927. 7089480 636 -r-x------ 1 hduser hadoop 650236 Mar 9 18:53 ./__spark_libs__/spark-yarn_2.11-2.3.0.jar
  1928. 7089481 48 -r-x------ 1 hduser hadoop 46986 Mar 9 18:53 ./__spark_libs__/jackson-annotations-2.6.7.jar
  1929. 7089482 700 -r-x------ 1 hduser hadoop 714194 Mar 9 18:53 ./__spark_libs__/javassist-3.18.1-GA.jar
  1930. 7089483 248 -r-x------ 1 hduser hadoop 250377 Mar 9 18:53 ./__spark_libs__/parquet-hadoop-1.8.2.jar
  1931. 7089484 64 -r-x------ 1 hduser hadoop 64847 Mar 9 18:53 ./__spark_libs__/spark-network-shuffle_2.11-2.3.0.jar
  1932. 7089485 116 -r-x------ 1 hduser hadoop 116859 Mar 9 18:53 ./__spark_libs__/py4j-0.10.6.jar
  1933. 7089486 76 -r-x------ 1 hduser hadoop 75835 Mar 9 18:53 ./__spark_libs__/spark-launcher_2.11-2.3.0.jar
  1934. 7089487 20 -r-x------ 1 hduser hadoop 16430 Mar 9 18:53 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar
  1935. 7089488 52 -r-x------ 1 hduser hadoop 52076 Mar 9 18:53 ./__spark_libs__/arrow-format-0.8.0.jar
  1936. 7089489 84 -r-x------ 1 hduser hadoop 82421 Mar 9 18:53 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar
  1937. 7089490 740 -r-x------ 1 hduser hadoop 757423 Mar 9 18:53 ./__spark_libs__/orc-mapreduce-1.4.1-nohive.jar
  1938. 7089491 1300 -r-x------ 1 hduser hadoop 1330219 Mar 9 18:53 ./__spark_libs__/netty-3.9.9.Final.jar
  1939. 7089492 120 -r-x------ 1 hduser hadoop 118973 Mar 9 18:53 ./__spark_libs__/hk2-utils-2.4.0-b34.jar
  1940. 7089493 408 -r-x------ 1 hduser hadoop 416731 Mar 9 18:53 ./__spark_libs__/kubernetes-client-3.0.0.jar
  1941. 7089494 96 -r-x------ 1 hduser hadoop 95806 Mar 9 18:53 ./__spark_libs__/javax.servlet-api-3.1.0.jar
  1942. 7089495 936 -r-x------ 1 hduser hadoop 956984 Mar 9 18:53 ./__spark_libs__/parquet-column-1.8.2.jar
  1943. 7089496 1412 -r-x------ 1 hduser hadoop 1441994 Mar 9 18:53 ./__spark_libs__/orc-core-1.4.1-nohive.jar
  1944. 7089497 576 -r-x------ 1 hduser hadoop 589462 Mar 9 18:53 ./__spark_libs__/json4s-core_2.11-3.2.11.jar
  1945. 7089498 2328 -r-x------ 1 hduser hadoop 2382165 Mar 9 18:53 ./__spark_libs__/spark-network-common_2.11-2.3.0.jar
  1946. 7089499 12 -r-x------ 1 hduser hadoop 8268 Mar 9 18:53 ./__spark_libs__/logging-interceptor-3.8.1.jar
  1947. 7089500 40 -r-x------ 1 hduser hadoop 40341 Mar 9 18:53 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar
  1948. 7089501 52 -r-x------ 1 hduser hadoop 53007 Mar 9 18:53 ./__spark_libs__/spark-kvstore_2.11-2.3.0.jar
  1949. 7089502 8 -r-x------ 1 hduser hadoop 5950 Mar 9 18:53 ./__spark_libs__/javax.inject-2.4.0-b34.jar
  1950. 7089503 376 -r-x------ 1 hduser hadoop 384150 Mar 9 18:53 ./__spark_libs__/univocity-parsers-2.5.9.jar
  1951. 7089504 7560 -r-x------ 1 hduser hadoop 7738527 Mar 9 18:53 ./__spark_libs__/spark-mllib_2.11-2.3.0.jar
  1952. 7089505 116 -r-x------ 1 hduser hadoop 115534 Mar 9 18:53 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar
  1953. 7089506 72 -r-x------ 1 hduser hadoop 72733 Mar 9 18:53 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar
  1954. 7089507 748 -r-x------ 1 hduser hadoop 764569 Mar 9 18:53 ./__spark_libs__/jtransforms-2.4.0.jar
  1955. 7089508 12 -r-x------ 1 hduser hadoop 10166 Mar 9 18:53 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
  1956. 7089509 36 -r-x------ 1 hduser hadoop 34654 Mar 9 18:53 ./__spark_libs__/paranamer-2.8.jar
  1957. 7089510 8464 -r-x------ 1 hduser hadoop 8664987 Mar 9 18:53 ./__spark_libs__/spark-sql_2.11-2.3.0.jar
  1958. 7089511 164 -r-x------ 1 hduser hadoop 164422 Mar 9 18:53 ./__spark_libs__/core-1.1.2.jar
  1959. 7089512 44 -r-x------ 1 hduser hadoop 42032 Mar 9 18:53 ./__spark_libs__/parquet-common-1.8.2.jar
  1960. 7089513 36 -r-x------ 1 hduser hadoop 33015 Mar 9 18:53 ./__spark_libs__/jsr305-1.3.9.jar
  1961. 7089514 80 -r-x------ 1 hduser hadoop 81811 Mar 9 18:53 ./__spark_libs__/okio-1.13.0.jar
  1962. 7089515 328 -r-x------ 1 hduser hadoop 334662 Mar 9 18:53 ./__spark_libs__/antlr4-runtime-4.7.jar
  1963. 7089516 392 -r-x------ 1 hduser hadoop 398122 Mar 9 18:53 ./__spark_libs__/okhttp-3.8.1.jar
  1964. 7089517 352 -r-x------ 1 hduser hadoop 358390 Mar 9 18:53 ./__spark_libs__/kryo-shaded-3.0.3.jar
  1965. 7089518 80 -r-x------ 1 hduser hadoop 79150 Mar 9 18:53 ./__spark_libs__/arrow-memory-0.8.0.jar
  1966. 7089519 15128 -r-x------ 1 hduser hadoop 15487351 Mar 9 18:53 ./__spark_libs__/scala-compiler-2.11.8.jar
  1967. 7089520 164 -r-x------ 1 hduser hadoop 167421 Mar 9 18:53 ./__spark_libs__/jersey-client-2.22.2.jar
  1968. 7089521 384 -r-x------ 1 hduser hadoop 390733 Mar 9 18:53 ./__spark_libs__/parquet-format-2.3.1.jar
  1969. 7089522 64 -r-x------ 1 hduser hadoop 63777 Mar 9 18:53 ./__spark_libs__/validation-api-1.1.0.Final.jar
  1970. 7089523 4468 -r-x------ 1 hduser hadoop 4573750 Mar 9 18:53 ./__spark_libs__/scala-reflect-2.11.8.jar
  1971. 7089524 932 -r-x------ 1 hduser hadoop 951701 Mar 9 18:53 ./__spark_libs__/jersey-server-2.22.2.jar
  1972. 7089525 48 -r-x------ 1 hduser hadoop 48464 Mar 9 18:53 ./__spark_libs__/spark-unsafe_2.11-2.3.0.jar
  1973. 7089526 184 -r-x------ 1 hduser hadoop 187882 Mar 9 18:53 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar
  1974. 7089527 3444 -r-x------ 1 hduser hadoop 3522616 Mar 9 18:53 ./__spark_libs__/shapeless_2.11-2.3.2.jar
  1975. 7089528 8 -r-x------ 1 hduser hadoop 5711 Mar 9 18:53 ./__spark_libs__/minlog-1.3.0.jar
  1976. 7089529 20 -r-x------ 1 hduser hadoop 18098 Mar 9 18:53 ./__spark_libs__/jersey-container-servlet-2.22.2.jar
  1977. 7089530 40 -r-x------ 1 hduser hadoop 37887 Mar 9 18:53 ./__spark_libs__/commons-compiler-3.0.8.jar
  1978. 7089531 44 -r-x------ 1 hduser hadoop 41755 Mar 9 18:53 ./__spark_libs__/objenesis-2.1.jar
  1979. 7089532 280 -r-x------ 1 hduser hadoop 286071 Mar 9 18:53 ./__spark_libs__/parquet-encoding-1.8.2.jar
  1980. 7089533 220 -r-x------ 1 hduser hadoop 224167 Mar 9 18:53 ./__spark_libs__/chill_2.11-0.8.4.jar
  1981. 7089534 780 -r-x------ 1 hduser hadoop 796326 Mar 9 18:53 ./__spark_libs__/janino-3.0.8.jar
  1982. 7089535 1244 -r-x------ 1 hduser hadoop 1270377 Mar 9 18:53 ./__spark_libs__/arrow-vector-0.8.0.jar
  1983. 7089536 24 -r-x------ 1 hduser hadoop 21247 Mar 9 18:53 ./__spark_libs__/metrics-graphite-3.1.5.jar
  1984. 7089537 1988 -r-x------ 1 hduser hadoop 2035066 Mar 9 18:53 ./__spark_libs__/commons-math3-3.4.1.jar
  1985. 7089538 32 -r-x------ 1 hduser hadoop 30057 Mar 9 18:53 ./__spark_libs__/spark-sketch_2.11-2.3.0.jar
  1986. 7089539 2120 -r-x------ 1 hduser hadoop 2169825 Mar 9 18:53 ./__spark_libs__/spark-streaming_2.11-2.3.0.jar
  1987. 7089540 9888 -r-x------ 1 hduser hadoop 10121868 Mar 9 18:53 ./__spark_libs__/spire_2.11-0.13.0.jar
  1988. 7089541 8 -r-x------ 1 hduser hadoop 4596 Mar 9 18:53 ./__spark_libs__/jul-to-slf4j-1.7.16.jar
  1989. 7089542 20 -r-x------ 1 hduser hadoop 20235 Mar 9 18:53 ./__spark_libs__/osgi-resource-locator-1.0.1.jar
  1990. 7089543 4 -r-x------ 1 hduser hadoop 3142 Mar 9 18:53 ./__spark_libs__/macro-compat_2.11-1.1.1.jar
  1991. 7089544 280 -r-x------ 1 hduser hadoop 284220 Mar 9 18:53 ./__spark_libs__/commons-lang-2.6.jar
  1992. 7089545 1256 -r-x------ 1 hduser hadoop 1282424 Mar 9 18:53 ./__spark_libs__/ivy-2.4.0.jar
  1993. 7089546 80 -r-x------ 1 hduser hadoop 79845 Mar 9 18:53 ./__spark_libs__/compress-lzf-1.0.3.jar
  1994. 7089547 16 -r-x------ 1 hduser hadoop 15824 Mar 9 18:53 ./__spark_libs__/metrics-json-3.1.5.jar
  1995. 7089548 12680 -r-x------ 1 hduser hadoop 12982716 Mar 9 18:53 ./__spark_libs__/spark-core_2.11-2.3.0.jar
  1996. 7089549 180 -r-x------ 1 hduser hadoop 181271 Mar 9 18:53 ./__spark_libs__/hk2-locator-2.4.0-b34.jar
  1997. 7089550 416 -r-x------ 1 hduser hadoop 423753 Mar 9 18:53 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
  1998. 7089551 1140 -r-x------ 1 hduser hadoop 1165323 Mar 9 18:53 ./__spark_libs__/jackson-databind-2.6.7.1.jar
  1999. 7089552 208 -r-x------ 1 hduser hadoop 212453 Mar 9 18:53 ./__spark_libs__/commons-net-2.2.jar
  2000. 7089553 64 -r-x------ 1 hduser hadoop 65261 Mar 9 18:53 ./__spark_libs__/oro-2.0.8.jar
  2001. 7089554 144 -r-x------ 1 hduser hadoop 144660 Mar 9 18:53 ./__spark_libs__/xbean-asm5-shaded-4.4.jar
  2002. 7089555 8776 -r-x------ 1 hduser hadoop 8984422 Mar 9 18:53 ./__spark_libs__/spark-catalyst_2.11-2.3.0.jar
  2003. 7089556 40 -r-x------ 1 hduser hadoop 39283 Mar 9 18:53 ./__spark_libs__/metrics-jvm-3.1.5.jar
  2004. 7089557 280 -r-x------ 1 hduser hadoop 284184 Mar 9 18:53 ./__spark_libs__/commons-codec-1.10.jar
  2005. 7089558 176 -r-x------ 1 hduser hadoop 176285 Mar 9 18:53 ./__spark_libs__/automaton-1.11-8.jar
  2006. 7089559 2280 -r-x------ 1 hduser hadoop 2333186 Mar 9 18:53 ./__spark_libs__/zstd-jni-1.3.2-2.jar
  2007. 7089560 648 -r-x------ 1 hduser hadoop 663328 Mar 9 18:53 ./__spark_libs__/spark-mesos_2.11-2.3.0.jar
  2008. 7089561 696 -r-x------ 1 hduser hadoop 708667 Mar 9 18:53 ./__spark_libs__/spark-graphx_2.11-2.3.0.jar
  2009. 7089562 16 -r-x------ 1 hduser hadoop 15456 Mar 9 18:53 ./__spark_libs__/spark-tags_2.11-2.3.0.jar
  2010. 7089563 616 -r-x------ 1 hduser hadoop 627814 Mar 9 18:53 ./__spark_libs__/joda-time-2.9.3.jar
  2011. 7089564 120 -r-x------ 1 hduser hadoop 120465 Mar 9 18:53 ./__spark_libs__/metrics-core-3.1.5.jar
  2012. 7089565 16 -r-x------ 1 hduser hadoop 14766 Mar 9 18:53 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
  2013. 7089566 952 -r-x------ 1 hduser hadoop 971310 Mar 9 18:53 ./__spark_libs__/jersey-guava-2.22.2.jar
  2014. 7089587 4 -rw-r--r-- 1 hduser hadoop 56 Mar 9 18:53 ./.launch_container.sh.crc
  2015. 7089581 4 -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:53 ./container_tokens
  2016. 7089590 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:53 ./.default_container_executor_session.sh.crc
  2017. 7089589 4 -rwx------ 1 hduser hadoop 678 Mar 9 18:53 ./default_container_executor_session.sh
  2018. 1658227 4 drwx------ 3 hduser hadoop 4096 Mar 9 18:53 ./__spark_conf__
  2019. 7089438 4 -r-x------ 1 hduser hadoop 1192 Mar 9 18:53 ./__spark_conf__/__spark_conf__.properties
  2020. 1658230 4 drwx------ 2 hduser hadoop 4096 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__
  2021. 7089421 4 -r-x------ 1 hduser hadoop 2316 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example
  2022. 7089425 4 -r-x------ 1 hduser hadoop 891 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/slaves
  2023. 7089436 4 -r-x------ 1 hduser hadoop 682 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-site.xml
  2024. 7089430 4 -r-x------ 1 hduser hadoop 996 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/docker.properties.template
  2025. 7089417 4 -r-x------ 1 hduser hadoop 1764 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-env.sh
  2026. 7089410 4 -r-x------ 1 hduser hadoop 3518 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-acls.xml
  2027. 7089435 8 -r-x------ 1 hduser hadoop 4113 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template
  2028. 7089427 4 -r-x------ 1 hduser hadoop 21 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-signature.secret
  2029. 7089426 4 -r-x------ 1 hduser hadoop 1484 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-env.sh
  2030. 7089412 4 -r-x------ 1 hduser hadoop 482 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-env.sh
  2031. 7089401 4 -r-x------ 1 hduser hadoop 3670 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd
  2032. 7089397 4 -r-x------ 1 hduser hadoop 2016 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.spark.properties
  2033. 7089406 8 -r-x------ 1 hduser hadoop 4221 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-env.sh.template
  2034. 7089396 4 -r-x------ 1 hduser hadoop 137 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
  2035. 7089420 4 -r-x------ 1 hduser hadoop 3414 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-user-functions.sh.example
  2036. 7089408 4 -r-x------ 1 hduser hadoop 2250 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd
  2037. 7089415 4 -r-x------ 1 hduser hadoop 589 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/container-executor.cfg
  2038. 7089405 4 -r-x------ 1 hduser hadoop 890 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/workers
  2039. 7089434 4 -r-x------ 1 hduser hadoop 2697 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example
  2040. 7089398 4 -r-x------ 1 hduser hadoop 1657 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-log4j.properties
  2041. 7089418 8 -r-x------ 1 hduser hadoop 7801 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/metrics.properties.template
  2042. 7089416 4 -r-x------ 1 hduser hadoop 1781 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf
  2043. 7089404 4 -r-x------ 1 hduser hadoop 225 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/core-site.xml
  2044. 7089399 4 -r-x------ 1 hduser hadoop 3323 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties
  2045. 7089409 8 -r-x------ 1 hduser hadoop 7861 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml
  2046. 7089432 4 -r-x------ 1 hduser hadoop 1292 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf.template
  2047. 7089419 4 -r-x------ 1 hduser hadoop 2642 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/user_ec_policies.xml.template
  2048. 7089431 4 -r-x------ 1 hduser hadoop 951 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd
  2049. 7089428 4 -r-x------ 1 hduser hadoop 620 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-site.xml
  2050. 7089433 4 -r-x------ 1 hduser hadoop 3589 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j2.xml
  2051. 7089423 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.properties.template
  2052. 7089403 4 -r-x------ 1 hduser hadoop 865 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/slaves.template
  2053. 7089414 4 -r-x------ 1 hduser hadoop 809 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
  2054. 7089395 16 -r-x------ 1 hduser hadoop 16270 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh
  2055. 7089411 4 -r-x------ 1 hduser hadoop 1335 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/configuration.xsl
  2056. 7089400 8 -r-x------ 1 hduser hadoop 8048 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
  2057. 7089394 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.properties
  2058. 7089422 4 -r-x------ 1 hduser hadoop 1351 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-env.sh
  2059. 7089407 4 -r-x------ 1 hduser hadoop 1747 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties
  2060. 7089424 12 -r-x------ 1 hduser hadoop 10206 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml
  2061. 7089429 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/metrics.properties
  2062. 7089413 8 -r-x------ 1 hduser hadoop 5502 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-env.sh
  2063. 7089402 4 -r-x------ 1 hduser hadoop 1105 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.template
  2064. 7089391 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/log4j.properties
  2065. 7089437 172 -r-x------ 1 hduser hadoop 173257 Mar 9 18:53 ./__spark_conf__/__spark_hadoop_conf__.xml
  2066. 7089392 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:53 ./__spark_conf__/metrics.properties
  2067. 7089583 4 -rw-r--r-- 1 hduser hadoop 12 Mar 9 18:53 ./.container_tokens.crc
  2068. broken symlinks(find -L . -maxdepth 5 -type l -ls):
  2069.  
  2070. End of LogType:directory.info
  2071. *******************************************************************************
  2072.  
  2073. Container: container_1520619475936_0001_01_000003 on bf263d1bde4d_41271
  2074. LogAggregationType: AGGREGATED
  2075. =======================================================================
  2076. LogType:launch_container.sh
  2077. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2078. LogLength:6052
  2079. LogContents:
  2080. #!/bin/bash
  2081.  
  2082. set -o pipefail -e
  2083. export PRELAUNCH_OUT="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/prelaunch.out"
  2084. exec >"${PRELAUNCH_OUT}"
  2085. export PRELAUNCH_ERR="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/prelaunch.err"
  2086. exec 2>"${PRELAUNCH_ERR}"
  2087. echo "Setting up env variables"
  2088. export SPARK_YARN_STAGING_DIR="hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001"
  2089. export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
  2090. export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
  2091. export JAVA_HOME="/usr/java/default"
  2092. export SPARK_LOG_URL_STDOUT="http://bf263d1bde4d:18042/node/containerlogs/container_1520619475936_0001_01_000003/hduser/stdout?start=-4096"
  2093. export NM_HOST="bf263d1bde4d"
  2094. export HADOOP_HDFS_HOME="/usr/local/hadoop"
  2095. export LOGNAME="hduser"
  2096. export JVM_PID="$$"
  2097. export PWD="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_01_000003"
  2098. export HADOOP_COMMON_HOME="/usr/local/hadoop"
  2099. export LOCAL_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001"
  2100. export NM_HTTP_PORT="18042"
  2101. export SPARK_DIST_CLASSPATH="/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*"
  2102. export LOG_DIRS="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003"
  2103. export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  2104. "
  2105. export NM_PORT="41271"
  2106. export USER="hduser"
  2107. export HADOOP_YARN_HOME="/usr/local/hadoop"
  2108. export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$PWD/__spark_conf__/__hadoop_conf__"
  2109. export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_01_000003/container_tokens"
  2110. export SPARK_USER="hduser"
  2111. export LOCAL_USER_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/"
  2112. export HADOOP_HOME="/usr/local/hadoop"
  2113. export SPARK_LOG_URL_STDERR="http://bf263d1bde4d:18042/node/containerlogs/container_1520619475936_0001_01_000003/hduser/stderr?start=-4096"
  2114. export HOME="/home/"
  2115. export CONTAINER_ID="container_1520619475936_0001_01_000003"
  2116. export MALLOC_ARENA_MAX="4"
  2117. echo "Setting up job resources"
  2118. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip" "__spark_libs__"
  2119. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar" "__app__.jar"
  2120. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip" "__spark_conf__"
  2121. echo "Copying debugging information"
  2122. # Creating copy of launch script
  2123. cp "launch_container.sh" "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/launch_container.sh"
  2124. chmod 640 "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/launch_container.sh"
  2125. # Determining directory contents
  2126. echo "ls -l:" 1>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/directory.info"
  2127. ls -l 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/directory.info"
  2128. echo "find -L . -maxdepth 5 -ls:" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/directory.info"
  2129. find -L . -maxdepth 5 -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/directory.info"
  2130. echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/directory.info"
  2131. find -L . -maxdepth 5 -type l -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/directory.info"
  2132. echo "Launching container"
  2133. exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx1024m '-XX:MaxGCPauseMillis=950' -Djava.io.tmpdir=$PWD/tmp '-Dspark.driver.port=36031' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@bcaf3c62a630:36031 --executor-id 2 --hostname bf263d1bde4d --cores 1 --app-id application_1520619475936_0001 --user-class-path file:$PWD/__app__.jar 1>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/stdout 2>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000003/stderr"
  2134.  
  2135. End of LogType:launch_container.sh
  2136. ************************************************************************************
  2137.  
  2138.  
  2139. End of LogType:prelaunch.err
  2140. ******************************************************************************
  2141.  
  2142. Container: container_1520619475936_0001_01_000003 on bf263d1bde4d_41271
  2143. LogAggregationType: AGGREGATED
  2144. =======================================================================
  2145. LogType:prelaunch.out
  2146. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2147. LogLength:100
  2148. LogContents:
  2149. Setting up env variables
  2150. Setting up job resources
  2151. Copying debugging information
  2152. Launching container
  2153.  
  2154. End of LogType:prelaunch.out
  2155. ******************************************************************************
  2156.  
  2157. Container: container_1520619475936_0001_01_000003 on bf263d1bde4d_41271
  2158. LogAggregationType: AGGREGATED
  2159. =======================================================================
  2160. LogType:stderr
  2161. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2162. LogLength:3595
  2163. LogContents:
  2164. 18/03/09 18:53:37 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 13314@bf263d1bde4d
  2165. 18/03/09 18:53:37 INFO SignalUtils: Registered signal handler for TERM
  2166. 18/03/09 18:53:37 INFO SignalUtils: Registered signal handler for HUP
  2167. 18/03/09 18:53:37 INFO SignalUtils: Registered signal handler for INT
  2168. 18/03/09 18:53:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2169. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls to: hduser
  2170. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls to: hduser
  2171. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls groups to:
  2172. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls groups to:
  2173. 18/03/09 18:53:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  2174. 18/03/09 18:53:38 INFO TransportClientFactory: Successfully created connection to bcaf3c62a630/172.22.0.3:36031 after 80 ms (0 ms spent in bootstraps)
  2175. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls to: hduser
  2176. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls to: hduser
  2177. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls groups to:
  2178. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls groups to:
  2179. 18/03/09 18:53:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  2180. 18/03/09 18:53:38 INFO TransportClientFactory: Successfully created connection to bcaf3c62a630/172.22.0.3:36031 after 1 ms (0 ms spent in bootstraps)
  2181. 18/03/09 18:53:38 INFO DiskBlockManager: Created local directory at /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/blockmgr-daacda36-62e7-4a02-8ef8-7044aaecc806
  2182. 18/03/09 18:53:38 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
  2183. 18/03/09 18:53:39 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@bcaf3c62a630:36031
  2184. 18/03/09 18:53:39 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  2185. 18/03/09 18:53:39 INFO Executor: Starting executor ID 2 on host bf263d1bde4d
  2186. 18/03/09 18:53:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39531.
  2187. 18/03/09 18:53:39 INFO NettyBlockTransferService: Server created on bf263d1bde4d:39531
  2188. 18/03/09 18:53:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  2189. 18/03/09 18:53:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, bf263d1bde4d, 39531, None)
  2190. 18/03/09 18:53:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, bf263d1bde4d, 39531, None)
  2191. 18/03/09 18:53:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, bf263d1bde4d, 39531, None)
  2192. 18/03/09 18:53:48 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
  2193. 18/03/09 18:53:48 INFO MemoryStore: MemoryStore cleared
  2194. 18/03/09 18:53:48 INFO BlockManager: BlockManager stopped
  2195. 18/03/09 18:53:48 INFO CoarseGrainedExecutorBackend: Driver from bcaf3c62a630:36031 disconnected during shutdown
  2196. 18/03/09 18:53:48 INFO CoarseGrainedExecutorBackend: Driver from bcaf3c62a630:36031 disconnected during shutdown
  2197. 18/03/09 18:53:48 INFO ShutdownHookManager: Shutdown hook called
  2198.  
  2199. End of LogType:stderr
  2200. ***********************************************************************
  2201.  
  2202.  
  2203. End of LogType:stdout
  2204. ***********************************************************************
  2205.  
  2206. Container: container_1520619475936_0001_01_000002 on bf263d1bde4d_41271
  2207. LogAggregationType: AGGREGATED
  2208. =======================================================================
  2209. LogType:directory.info
  2210. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2211. LogLength:21689
  2212. LogContents:
  2213. ls -l:
  2214. total 36
  2215. lrwxrwxrwx 1 hduser hadoop 89 Mar 9 18:53 __app__.jar -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar
  2216. lrwxrwxrwx 1 hduser hadoop 80 Mar 9 18:53 __spark_conf__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip
  2217. lrwxrwxrwx 1 hduser hadoop 99 Mar 9 18:53 __spark_libs__ -> /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip
  2218. -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:53 container_tokens
  2219. -rwx------ 1 hduser hadoop 733 Mar 9 18:53 default_container_executor.sh
  2220. -rwx------ 1 hduser hadoop 678 Mar 9 18:53 default_container_executor_session.sh
  2221. -rwx------ 1 hduser hadoop 6052 Mar 9 18:53 launch_container.sh
  2222. drwx--x--- 2 hduser hadoop 4096 Mar 9 18:53 tmp
  2223. find -L . -maxdepth 5 -ls:
  2224. 1668591 4 drwx--x--- 3 hduser hadoop 4096 Mar 9 18:53 .
  2225. 7089593 4 -rwx------ 1 hduser hadoop 733 Mar 9 18:53 ./default_container_executor.sh
  2226. 7089584 8 -rwx------ 1 hduser hadoop 6052 Mar 9 18:53 ./launch_container.sh
  2227. 1668593 4 drwx--x--- 2 hduser hadoop 4096 Mar 9 18:53 ./tmp
  2228. 7089389 517916 -r-x------ 1 hduser hadoop 530343861 Mar 9 18:53 ./__app__.jar
  2229. 7089595 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:53 ./.default_container_executor.sh.crc
  2230. 1654426 12 drwx------ 2 hduser hadoop 12288 Mar 9 18:53 ./__spark_libs__
  2231. 7089441 88 -r-x------ 1 hduser hadoop 87192 Mar 9 18:53 ./__spark_libs__/spire-macros_2.11-0.13.0.jar
  2232. 7089442 5612 -r-x------ 1 hduser hadoop 5744974 Mar 9 18:53 ./__spark_libs__/scala-library-2.11.8.jar
  2233. 7089443 20 -r-x------ 1 hduser hadoop 19827 Mar 9 18:53 ./__spark_libs__/opencsv-2.3.jar
  2234. 7089444 16 -r-x------ 1 hduser hadoop 13911 Mar 9 18:53 ./__spark_libs__/generex-1.0.1.jar
  2235. 7089445 364 -r-x------ 1 hduser hadoop 370119 Mar 9 18:53 ./__spark_libs__/lz4-java-1.4.0.jar
  2236. 7089446 256 -r-x------ 1 hduser hadoop 258919 Mar 9 18:53 ./__spark_libs__/jackson-core-2.6.7.jar
  2237. 7089447 684 -r-x------ 1 hduser hadoop 698375 Mar 9 18:53 ./__spark_libs__/jersey-common-2.22.2.jar
  2238. 7089448 472 -r-x------ 1 hduser hadoop 479881 Mar 9 18:53 ./__spark_libs__/commons-lang3-3.5.jar
  2239. 7089449 6852 -r-x------ 1 hduser hadoop 7015233 Mar 9 18:53 ./__spark_libs__/kubernetes-model-2.0.0.jar
  2240. 7089450 176 -r-x------ 1 hduser hadoop 178947 Mar 9 18:53 ./__spark_libs__/hk2-api-2.4.0-b34.jar
  2241. 7089451 7172 -r-x------ 1 hduser hadoop 7343426 Mar 9 18:53 ./__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
  2242. 7089452 504 -r-x------ 1 hduser hadoop 515645 Mar 9 18:53 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
  2243. 7089453 44 -r-x------ 1 hduser hadoop 42858 Mar 9 18:53 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar
  2244. 7089454 60 -r-x------ 1 hduser hadoop 60483 Mar 9 18:53 ./__spark_libs__/chill-java-0.8.4.jar
  2245. 7089455 132 -r-x------ 1 hduser hadoop 134595 Mar 9 18:53 ./__spark_libs__/commons-crypto-1.0.0.jar
  2246. 7089456 128 -r-x------ 1 hduser hadoop 130802 Mar 9 18:53 ./__spark_libs__/aircompressor-0.8.jar
  2247. 7089457 264 -r-x------ 1 hduser hadoop 269295 Mar 9 18:53 ./__spark_libs__/snakeyaml-1.15.jar
  2248. 7089458 1168 -r-x------ 1 hduser hadoop 1194003 Mar 9 18:53 ./__spark_libs__/arpack_combined_all-0.1.jar
  2249. 7089459 14760 -r-x------ 1 hduser hadoop 15113382 Mar 9 18:53 ./__spark_libs__/breeze_2.11-0.13.2.jar
  2250. 7089460 36 -r-x------ 1 hduser hadoop 35518 Mar 9 18:53 ./__spark_libs__/zjsonpatch-0.3.0.jar
  2251. 7089461 120 -r-x------ 1 hduser hadoop 120321 Mar 9 18:53 ./__spark_libs__/spark-repl_2.11-2.3.0.jar
  2252. 7089462 36 -r-x------ 1 hduser hadoop 34603 Mar 9 18:53 ./__spark_libs__/machinist_2.11-0.6.1.jar
  2253. 7089463 28 -r-x------ 1 hduser hadoop 26366 Mar 9 18:53 ./__spark_libs__/javax.annotation-api-1.2.jar
  2254. 7089464 180 -r-x------ 1 hduser hadoop 180736 Mar 9 18:53 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
  2255. 7089465 788 -r-x------ 1 hduser hadoop 802818 Mar 9 18:53 ./__spark_libs__/scalap-2.11.8.jar
  2256. 7089466 1504 -r-x------ 1 hduser hadoop 1537966 Mar 9 18:53 ./__spark_libs__/hadoop-mapreduce-client-core-2.6.5.jar
  2257. 7089467 1024 -r-x------ 1 hduser hadoop 1048116 Mar 9 18:53 ./__spark_libs__/parquet-jackson-1.8.2.jar
  2258. 7089468 172 -r-x------ 1 hduser hadoop 174351 Mar 9 18:53 ./__spark_libs__/stream-2.7.0.jar
  2259. 7089469 68 -r-x------ 1 hduser hadoop 66270 Mar 9 18:53 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar
  2260. 7089470 1632 -r-x------ 1 hduser hadoop 1671083 Mar 9 18:53 ./__spark_libs__/hppc-0.7.2.jar
  2261. 7089471 180 -r-x------ 1 hduser hadoop 183832 Mar 9 18:53 ./__spark_libs__/spark-mllib-local_2.11-2.3.0.jar
  2262. 7089472 656 -r-x------ 1 hduser hadoop 671138 Mar 9 18:53 ./__spark_libs__/scala-xml_2.11-1.0.5.jar
  2263. 7089473 200 -r-x------ 1 hduser hadoop 201928 Mar 9 18:53 ./__spark_libs__/RoaringBitmap-0.5.11.jar
  2264. 7089474 96 -r-x------ 1 hduser hadoop 94796 Mar 9 18:53 ./__spark_libs__/pyrolite-4.13.jar
  2265. 7089475 3692 -r-x------ 1 hduser hadoop 3780056 Mar 9 18:53 ./__spark_libs__/netty-all-4.1.17.Final.jar
  2266. 7089476 316 -r-x------ 1 hduser hadoop 320444 Mar 9 18:53 ./__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
  2267. 7089477 1024 -r-x------ 1 hduser hadoop 1045744 Mar 9 18:53 ./__spark_libs__/leveldbjni-all-1.8.jar
  2268. 7089478 376 -r-x------ 1 hduser hadoop 381098 Mar 9 18:53 ./__spark_libs__/spark-kubernetes_2.11-2.3.0.jar
  2269. 7089479 32 -r-x------ 1 hduser hadoop 32612 Mar 9 18:53 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
  2270. 7089480 636 -r-x------ 1 hduser hadoop 650236 Mar 9 18:53 ./__spark_libs__/spark-yarn_2.11-2.3.0.jar
  2271. 7089481 48 -r-x------ 1 hduser hadoop 46986 Mar 9 18:53 ./__spark_libs__/jackson-annotations-2.6.7.jar
  2272. 7089482 700 -r-x------ 1 hduser hadoop 714194 Mar 9 18:53 ./__spark_libs__/javassist-3.18.1-GA.jar
  2273. 7089483 248 -r-x------ 1 hduser hadoop 250377 Mar 9 18:53 ./__spark_libs__/parquet-hadoop-1.8.2.jar
  2274. 7089484 64 -r-x------ 1 hduser hadoop 64847 Mar 9 18:53 ./__spark_libs__/spark-network-shuffle_2.11-2.3.0.jar
  2275. 7089485 116 -r-x------ 1 hduser hadoop 116859 Mar 9 18:53 ./__spark_libs__/py4j-0.10.6.jar
  2276. 7089486 76 -r-x------ 1 hduser hadoop 75835 Mar 9 18:53 ./__spark_libs__/spark-launcher_2.11-2.3.0.jar
  2277. 7089487 20 -r-x------ 1 hduser hadoop 16430 Mar 9 18:53 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar
  2278. 7089488 52 -r-x------ 1 hduser hadoop 52076 Mar 9 18:53 ./__spark_libs__/arrow-format-0.8.0.jar
  2279. 7089489 84 -r-x------ 1 hduser hadoop 82421 Mar 9 18:53 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar
  2280. 7089490 740 -r-x------ 1 hduser hadoop 757423 Mar 9 18:53 ./__spark_libs__/orc-mapreduce-1.4.1-nohive.jar
  2281. 7089491 1300 -r-x------ 1 hduser hadoop 1330219 Mar 9 18:53 ./__spark_libs__/netty-3.9.9.Final.jar
  2282. 7089492 120 -r-x------ 1 hduser hadoop 118973 Mar 9 18:53 ./__spark_libs__/hk2-utils-2.4.0-b34.jar
  2283. 7089493 408 -r-x------ 1 hduser hadoop 416731 Mar 9 18:53 ./__spark_libs__/kubernetes-client-3.0.0.jar
  2284. 7089494 96 -r-x------ 1 hduser hadoop 95806 Mar 9 18:53 ./__spark_libs__/javax.servlet-api-3.1.0.jar
  2285. 7089495 936 -r-x------ 1 hduser hadoop 956984 Mar 9 18:53 ./__spark_libs__/parquet-column-1.8.2.jar
  2286. 7089496 1412 -r-x------ 1 hduser hadoop 1441994 Mar 9 18:53 ./__spark_libs__/orc-core-1.4.1-nohive.jar
  2287. 7089497 576 -r-x------ 1 hduser hadoop 589462 Mar 9 18:53 ./__spark_libs__/json4s-core_2.11-3.2.11.jar
  2288. 7089498 2328 -r-x------ 1 hduser hadoop 2382165 Mar 9 18:53 ./__spark_libs__/spark-network-common_2.11-2.3.0.jar
  2289. 7089499 12 -r-x------ 1 hduser hadoop 8268 Mar 9 18:53 ./__spark_libs__/logging-interceptor-3.8.1.jar
  2290. 7089500 40 -r-x------ 1 hduser hadoop 40341 Mar 9 18:53 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar
  2291. 7089501 52 -r-x------ 1 hduser hadoop 53007 Mar 9 18:53 ./__spark_libs__/spark-kvstore_2.11-2.3.0.jar
  2292. 7089502 8 -r-x------ 1 hduser hadoop 5950 Mar 9 18:53 ./__spark_libs__/javax.inject-2.4.0-b34.jar
  2293. 7089503 376 -r-x------ 1 hduser hadoop 384150 Mar 9 18:53 ./__spark_libs__/univocity-parsers-2.5.9.jar
  2294. 7089504 7560 -r-x------ 1 hduser hadoop 7738527 Mar 9 18:53 ./__spark_libs__/spark-mllib_2.11-2.3.0.jar
  2295. 7089505 116 -r-x------ 1 hduser hadoop 115534 Mar 9 18:53 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar
  2296. 7089506 72 -r-x------ 1 hduser hadoop 72733 Mar 9 18:53 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar
  2297. 7089507 748 -r-x------ 1 hduser hadoop 764569 Mar 9 18:53 ./__spark_libs__/jtransforms-2.4.0.jar
  2298. 7089508 12 -r-x------ 1 hduser hadoop 10166 Mar 9 18:53 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
  2299. 7089509 36 -r-x------ 1 hduser hadoop 34654 Mar 9 18:53 ./__spark_libs__/paranamer-2.8.jar
  2300. 7089510 8464 -r-x------ 1 hduser hadoop 8664987 Mar 9 18:53 ./__spark_libs__/spark-sql_2.11-2.3.0.jar
  2301. 7089511 164 -r-x------ 1 hduser hadoop 164422 Mar 9 18:53 ./__spark_libs__/core-1.1.2.jar
  2302. 7089512 44 -r-x------ 1 hduser hadoop 42032 Mar 9 18:53 ./__spark_libs__/parquet-common-1.8.2.jar
  2303. 7089513 36 -r-x------ 1 hduser hadoop 33015 Mar 9 18:53 ./__spark_libs__/jsr305-1.3.9.jar
  2304. 7089514 80 -r-x------ 1 hduser hadoop 81811 Mar 9 18:53 ./__spark_libs__/okio-1.13.0.jar
  2305. 7089515 328 -r-x------ 1 hduser hadoop 334662 Mar 9 18:53 ./__spark_libs__/antlr4-runtime-4.7.jar
  2306. 7089516 392 -r-x------ 1 hduser hadoop 398122 Mar 9 18:53 ./__spark_libs__/okhttp-3.8.1.jar
  2307. 7089517 352 -r-x------ 1 hduser hadoop 358390 Mar 9 18:53 ./__spark_libs__/kryo-shaded-3.0.3.jar
  2308. 7089518 80 -r-x------ 1 hduser hadoop 79150 Mar 9 18:53 ./__spark_libs__/arrow-memory-0.8.0.jar
  2309. 7089519 15128 -r-x------ 1 hduser hadoop 15487351 Mar 9 18:53 ./__spark_libs__/scala-compiler-2.11.8.jar
  2310. 7089520 164 -r-x------ 1 hduser hadoop 167421 Mar 9 18:53 ./__spark_libs__/jersey-client-2.22.2.jar
  2311. 7089521 384 -r-x------ 1 hduser hadoop 390733 Mar 9 18:53 ./__spark_libs__/parquet-format-2.3.1.jar
  2312. 7089522 64 -r-x------ 1 hduser hadoop 63777 Mar 9 18:53 ./__spark_libs__/validation-api-1.1.0.Final.jar
  2313. 7089523 4468 -r-x------ 1 hduser hadoop 4573750 Mar 9 18:53 ./__spark_libs__/scala-reflect-2.11.8.jar
  2314. 7089524 932 -r-x------ 1 hduser hadoop 951701 Mar 9 18:53 ./__spark_libs__/jersey-server-2.22.2.jar
  2315. 7089525 48 -r-x------ 1 hduser hadoop 48464 Mar 9 18:53 ./__spark_libs__/spark-unsafe_2.11-2.3.0.jar
  2316. 7089526 184 -r-x------ 1 hduser hadoop 187882 Mar 9 18:53 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar
  2317. 7089527 3444 -r-x------ 1 hduser hadoop 3522616 Mar 9 18:53 ./__spark_libs__/shapeless_2.11-2.3.2.jar
  2318. 7089528 8 -r-x------ 1 hduser hadoop 5711 Mar 9 18:53 ./__spark_libs__/minlog-1.3.0.jar
  2319. 7089529 20 -r-x------ 1 hduser hadoop 18098 Mar 9 18:53 ./__spark_libs__/jersey-container-servlet-2.22.2.jar
  2320. 7089530 40 -r-x------ 1 hduser hadoop 37887 Mar 9 18:53 ./__spark_libs__/commons-compiler-3.0.8.jar
  2321. 7089531 44 -r-x------ 1 hduser hadoop 41755 Mar 9 18:53 ./__spark_libs__/objenesis-2.1.jar
  2322. 7089532 280 -r-x------ 1 hduser hadoop 286071 Mar 9 18:53 ./__spark_libs__/parquet-encoding-1.8.2.jar
  2323. 7089533 220 -r-x------ 1 hduser hadoop 224167 Mar 9 18:53 ./__spark_libs__/chill_2.11-0.8.4.jar
  2324. 7089534 780 -r-x------ 1 hduser hadoop 796326 Mar 9 18:53 ./__spark_libs__/janino-3.0.8.jar
  2325. 7089535 1244 -r-x------ 1 hduser hadoop 1270377 Mar 9 18:53 ./__spark_libs__/arrow-vector-0.8.0.jar
  2326. 7089536 24 -r-x------ 1 hduser hadoop 21247 Mar 9 18:53 ./__spark_libs__/metrics-graphite-3.1.5.jar
  2327. 7089537 1988 -r-x------ 1 hduser hadoop 2035066 Mar 9 18:53 ./__spark_libs__/commons-math3-3.4.1.jar
  2328. 7089538 32 -r-x------ 1 hduser hadoop 30057 Mar 9 18:53 ./__spark_libs__/spark-sketch_2.11-2.3.0.jar
  2329. 7089539 2120 -r-x------ 1 hduser hadoop 2169825 Mar 9 18:53 ./__spark_libs__/spark-streaming_2.11-2.3.0.jar
  2330. 7089540 9888 -r-x------ 1 hduser hadoop 10121868 Mar 9 18:53 ./__spark_libs__/spire_2.11-0.13.0.jar
  2331. 7089541 8 -r-x------ 1 hduser hadoop 4596 Mar 9 18:53 ./__spark_libs__/jul-to-slf4j-1.7.16.jar
  2332. 7089542 20 -r-x------ 1 hduser hadoop 20235 Mar 9 18:53 ./__spark_libs__/osgi-resource-locator-1.0.1.jar
  2333. 7089543 4 -r-x------ 1 hduser hadoop 3142 Mar 9 18:53 ./__spark_libs__/macro-compat_2.11-1.1.1.jar
  2334. 7089544 280 -r-x------ 1 hduser hadoop 284220 Mar 9 18:53 ./__spark_libs__/commons-lang-2.6.jar
  2335. 7089545 1256 -r-x------ 1 hduser hadoop 1282424 Mar 9 18:53 ./__spark_libs__/ivy-2.4.0.jar
  2336. 7089546 80 -r-x------ 1 hduser hadoop 79845 Mar 9 18:53 ./__spark_libs__/compress-lzf-1.0.3.jar
  2337. 7089547 16 -r-x------ 1 hduser hadoop 15824 Mar 9 18:53 ./__spark_libs__/metrics-json-3.1.5.jar
  2338. 7089548 12680 -r-x------ 1 hduser hadoop 12982716 Mar 9 18:53 ./__spark_libs__/spark-core_2.11-2.3.0.jar
  2339. 7089549 180 -r-x------ 1 hduser hadoop 181271 Mar 9 18:53 ./__spark_libs__/hk2-locator-2.4.0-b34.jar
  2340. 7089550 416 -r-x------ 1 hduser hadoop 423753 Mar 9 18:53 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
  2341. 7089551 1140 -r-x------ 1 hduser hadoop 1165323 Mar 9 18:53 ./__spark_libs__/jackson-databind-2.6.7.1.jar
  2342. 7089552 208 -r-x------ 1 hduser hadoop 212453 Mar 9 18:53 ./__spark_libs__/commons-net-2.2.jar
  2343. 7089553 64 -r-x------ 1 hduser hadoop 65261 Mar 9 18:53 ./__spark_libs__/oro-2.0.8.jar
  2344. 7089554 144 -r-x------ 1 hduser hadoop 144660 Mar 9 18:53 ./__spark_libs__/xbean-asm5-shaded-4.4.jar
  2345. 7089555 8776 -r-x------ 1 hduser hadoop 8984422 Mar 9 18:53 ./__spark_libs__/spark-catalyst_2.11-2.3.0.jar
  2346. 7089556 40 -r-x------ 1 hduser hadoop 39283 Mar 9 18:53 ./__spark_libs__/metrics-jvm-3.1.5.jar
  2347. 7089557 280 -r-x------ 1 hduser hadoop 284184 Mar 9 18:53 ./__spark_libs__/commons-codec-1.10.jar
  2348. 7089558 176 -r-x------ 1 hduser hadoop 176285 Mar 9 18:53 ./__spark_libs__/automaton-1.11-8.jar
  2349. 7089559 2280 -r-x------ 1 hduser hadoop 2333186 Mar 9 18:53 ./__spark_libs__/zstd-jni-1.3.2-2.jar
  2350. 7089560 648 -r-x------ 1 hduser hadoop 663328 Mar 9 18:53 ./__spark_libs__/spark-mesos_2.11-2.3.0.jar
  2351. 7089561 696 -r-x------ 1 hduser hadoop 708667 Mar 9 18:53 ./__spark_libs__/spark-graphx_2.11-2.3.0.jar
  2352. 7089562 16 -r-x------ 1 hduser hadoop 15456 Mar 9 18:53 ./__spark_libs__/spark-tags_2.11-2.3.0.jar
  2353. 7089563 616 -r-x------ 1 hduser hadoop 627814 Mar 9 18:53 ./__spark_libs__/joda-time-2.9.3.jar
  2354. 7089564 120 -r-x------ 1 hduser hadoop 120465 Mar 9 18:53 ./__spark_libs__/metrics-core-3.1.5.jar
  2355. 7089565 16 -r-x------ 1 hduser hadoop 14766 Mar 9 18:53 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
  2356. 7089566 952 -r-x------ 1 hduser hadoop 971310 Mar 9 18:53 ./__spark_libs__/jersey-guava-2.22.2.jar
  2357. 7089586 4 -rw-r--r-- 1 hduser hadoop 56 Mar 9 18:53 ./.launch_container.sh.crc
  2358. 7089580 4 -rw-r--r-- 1 hduser hadoop 81 Mar 9 18:53 ./container_tokens
  2359. 7089591 4 -rw-r--r-- 1 hduser hadoop 16 Mar 9 18:53 ./.default_container_executor_session.sh.crc
  2360. 7089588 4 -rwx------ 1 hduser hadoop 678 Mar 9 18:53 ./default_container_executor_session.sh
  2361. 1658227 4 drwx------ 3 hduser hadoop 4096 Mar 9 18:53 ./__spark_conf__
  2362. 7089438 4 -r-x------ 1 hduser hadoop 1192 Mar 9 18:53 ./__spark_conf__/__spark_conf__.properties
  2363. 1658230 4 drwx------ 2 hduser hadoop 4096 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__
  2364. 7089421 4 -r-x------ 1 hduser hadoop 2316 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example
  2365. 7089425 4 -r-x------ 1 hduser hadoop 891 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/slaves
  2366. 7089436 4 -r-x------ 1 hduser hadoop 682 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-site.xml
  2367. 7089430 4 -r-x------ 1 hduser hadoop 996 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/docker.properties.template
  2368. 7089417 4 -r-x------ 1 hduser hadoop 1764 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-env.sh
  2369. 7089410 4 -r-x------ 1 hduser hadoop 3518 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-acls.xml
  2370. 7089435 8 -r-x------ 1 hduser hadoop 4113 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template
  2371. 7089427 4 -r-x------ 1 hduser hadoop 21 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-signature.secret
  2372. 7089426 4 -r-x------ 1 hduser hadoop 1484 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-env.sh
  2373. 7089412 4 -r-x------ 1 hduser hadoop 482 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-env.sh
  2374. 7089401 4 -r-x------ 1 hduser hadoop 3670 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd
  2375. 7089397 4 -r-x------ 1 hduser hadoop 2016 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.spark.properties
  2376. 7089406 8 -r-x------ 1 hduser hadoop 4221 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-env.sh.template
  2377. 7089396 4 -r-x------ 1 hduser hadoop 137 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
  2378. 7089420 4 -r-x------ 1 hduser hadoop 3414 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-user-functions.sh.example
  2379. 7089408 4 -r-x------ 1 hduser hadoop 2250 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd
  2380. 7089415 4 -r-x------ 1 hduser hadoop 589 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/container-executor.cfg
  2381. 7089405 4 -r-x------ 1 hduser hadoop 890 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/workers
  2382. 7089434 4 -r-x------ 1 hduser hadoop 2697 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example
  2383. 7089398 4 -r-x------ 1 hduser hadoop 1657 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-log4j.properties
  2384. 7089418 8 -r-x------ 1 hduser hadoop 7801 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/metrics.properties.template
  2385. 7089416 4 -r-x------ 1 hduser hadoop 1781 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf
  2386. 7089404 4 -r-x------ 1 hduser hadoop 225 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/core-site.xml
  2387. 7089399 4 -r-x------ 1 hduser hadoop 3323 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties
  2388. 7089409 8 -r-x------ 1 hduser hadoop 7861 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml
  2389. 7089432 4 -r-x------ 1 hduser hadoop 1292 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/spark-defaults.conf.template
  2390. 7089419 4 -r-x------ 1 hduser hadoop 2642 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/user_ec_policies.xml.template
  2391. 7089431 4 -r-x------ 1 hduser hadoop 951 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd
  2392. 7089428 4 -r-x------ 1 hduser hadoop 620 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/httpfs-site.xml
  2393. 7089433 4 -r-x------ 1 hduser hadoop 3589 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j2.xml
  2394. 7089423 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.properties.template
  2395. 7089403 4 -r-x------ 1 hduser hadoop 865 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/slaves.template
  2396. 7089414 4 -r-x------ 1 hduser hadoop 809 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
  2397. 7089395 16 -r-x------ 1 hduser hadoop 16270 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh
  2398. 7089411 4 -r-x------ 1 hduser hadoop 1335 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/configuration.xsl
  2399. 7089400 8 -r-x------ 1 hduser hadoop 8048 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
  2400. 7089394 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/log4j.properties
  2401. 7089422 4 -r-x------ 1 hduser hadoop 1351 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-env.sh
  2402. 7089407 4 -r-x------ 1 hduser hadoop 1747 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties
  2403. 7089424 12 -r-x------ 1 hduser hadoop 10206 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml
  2404. 7089429 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/metrics.properties
  2405. 7089413 8 -r-x------ 1 hduser hadoop 5502 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/yarn-env.sh
  2406. 7089402 4 -r-x------ 1 hduser hadoop 1105 Mar 9 18:53 ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.template
  2407. 7089391 4 -r-x------ 1 hduser hadoop 2025 Mar 9 18:53 ./__spark_conf__/log4j.properties
  2408. 7089437 172 -r-x------ 1 hduser hadoop 173257 Mar 9 18:53 ./__spark_conf__/__spark_hadoop_conf__.xml
  2409. 7089392 4 -r-x------ 1 hduser hadoop 43 Mar 9 18:53 ./__spark_conf__/metrics.properties
  2410. 7089582 4 -rw-r--r-- 1 hduser hadoop 12 Mar 9 18:53 ./.container_tokens.crc
  2411. broken symlinks(find -L . -maxdepth 5 -type l -ls):
  2412.  
  2413. End of LogType:directory.info
  2414. *******************************************************************************
  2415.  
  2416. Container: container_1520619475936_0001_01_000002 on bf263d1bde4d_41271
  2417. LogAggregationType: AGGREGATED
  2418. =======================================================================
  2419. LogType:launch_container.sh
  2420. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2421. LogLength:6052
  2422. LogContents:
  2423. #!/bin/bash
  2424.  
  2425. set -o pipefail -e
  2426. export PRELAUNCH_OUT="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/prelaunch.out"
  2427. exec >"${PRELAUNCH_OUT}"
  2428. export PRELAUNCH_ERR="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/prelaunch.err"
  2429. exec 2>"${PRELAUNCH_ERR}"
  2430. echo "Setting up env variables"
  2431. export SPARK_YARN_STAGING_DIR="hdfs://master:9000/user/hduser/.sparkStaging/application_1520619475936_0001"
  2432. export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
  2433. export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
  2434. export JAVA_HOME="/usr/java/default"
  2435. export SPARK_LOG_URL_STDOUT="http://bf263d1bde4d:18042/node/containerlogs/container_1520619475936_0001_01_000002/hduser/stdout?start=-4096"
  2436. export NM_HOST="bf263d1bde4d"
  2437. export HADOOP_HDFS_HOME="/usr/local/hadoop"
  2438. export LOGNAME="hduser"
  2439. export JVM_PID="$$"
  2440. export PWD="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_01_000002"
  2441. export HADOOP_COMMON_HOME="/usr/local/hadoop"
  2442. export LOCAL_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001"
  2443. export NM_HTTP_PORT="18042"
  2444. export SPARK_DIST_CLASSPATH="/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*"
  2445. export LOG_DIRS="/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002"
  2446. export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  2447. "
  2448. export NM_PORT="41271"
  2449. export USER="hduser"
  2450. export HADOOP_YARN_HOME="/usr/local/hadoop"
  2451. export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:$PWD/__spark_conf__/__hadoop_conf__"
  2452. export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/container_1520619475936_0001_01_000002/container_tokens"
  2453. export SPARK_USER="hduser"
  2454. export LOCAL_USER_DIRS="/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/"
  2455. export HADOOP_HOME="/usr/local/hadoop"
  2456. export SPARK_LOG_URL_STDERR="http://bf263d1bde4d:18042/node/containerlogs/container_1520619475936_0001_01_000002/hduser/stderr?start=-4096"
  2457. export HOME="/home/"
  2458. export CONTAINER_ID="container_1520619475936_0001_01_000002"
  2459. export MALLOC_ARENA_MAX="4"
  2460. echo "Setting up job resources"
  2461. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/10/__spark_libs__1007958283373928587.zip" "__spark_libs__"
  2462. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/12/icu-mortality-1.0.0-all.jar" "__app__.jar"
  2463. ln -sf "/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/filecache/11/__spark_conf__.zip" "__spark_conf__"
  2464. echo "Copying debugging information"
  2465. # Creating copy of launch script
  2466. cp "launch_container.sh" "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/launch_container.sh"
  2467. chmod 640 "/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/launch_container.sh"
  2468. # Determining directory contents
  2469. echo "ls -l:" 1>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/directory.info"
  2470. ls -l 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/directory.info"
  2471. echo "find -L . -maxdepth 5 -ls:" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/directory.info"
  2472. find -L . -maxdepth 5 -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/directory.info"
  2473. echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/directory.info"
  2474. find -L . -maxdepth 5 -type l -ls 1>>"/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/directory.info"
  2475. echo "Launching container"
  2476. exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx1024m '-XX:MaxGCPauseMillis=950' -Djava.io.tmpdir=$PWD/tmp '-Dspark.driver.port=36031' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@bcaf3c62a630:36031 --executor-id 1 --hostname bf263d1bde4d --cores 1 --app-id application_1520619475936_0001 --user-class-path file:$PWD/__app__.jar 1>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/stdout 2>/usr/local/hadoop/logs/userlogs/application_1520619475936_0001/container_1520619475936_0001_01_000002/stderr"
  2477.  
  2478. End of LogType:launch_container.sh
  2479. ************************************************************************************
  2480.  
  2481.  
  2482. End of LogType:prelaunch.err
  2483. ******************************************************************************
  2484.  
  2485. Container: container_1520619475936_0001_01_000002 on bf263d1bde4d_41271
  2486. LogAggregationType: AGGREGATED
  2487. =======================================================================
  2488. LogType:prelaunch.out
  2489. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2490. LogLength:100
  2491. LogContents:
  2492. Setting up env variables
  2493. Setting up job resources
  2494. Copying debugging information
  2495. Launching container
  2496.  
  2497. End of LogType:prelaunch.out
  2498. ******************************************************************************
  2499.  
  2500. Container: container_1520619475936_0001_01_000002 on bf263d1bde4d_41271
  2501. LogAggregationType: AGGREGATED
  2502. =======================================================================
  2503. LogType:stderr
  2504. LogLastModifiedTime:Fri Mar 09 18:55:11 +0000 2018
  2505. LogLength:3595
  2506. LogContents:
  2507. 18/03/09 18:53:37 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 13313@bf263d1bde4d
  2508. 18/03/09 18:53:37 INFO SignalUtils: Registered signal handler for TERM
  2509. 18/03/09 18:53:37 INFO SignalUtils: Registered signal handler for HUP
  2510. 18/03/09 18:53:37 INFO SignalUtils: Registered signal handler for INT
  2511. 18/03/09 18:53:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2512. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls to: hduser
  2513. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls to: hduser
  2514. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls groups to:
  2515. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls groups to:
  2516. 18/03/09 18:53:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  2517. 18/03/09 18:53:38 INFO TransportClientFactory: Successfully created connection to bcaf3c62a630/172.22.0.3:36031 after 67 ms (0 ms spent in bootstraps)
  2518. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls to: hduser
  2519. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls to: hduser
  2520. 18/03/09 18:53:38 INFO SecurityManager: Changing view acls groups to:
  2521. 18/03/09 18:53:38 INFO SecurityManager: Changing modify acls groups to:
  2522. 18/03/09 18:53:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser); groups with modify permissions: Set()
  2523. 18/03/09 18:53:38 INFO TransportClientFactory: Successfully created connection to bcaf3c62a630/172.22.0.3:36031 after 1 ms (0 ms spent in bootstraps)
  2524. 18/03/09 18:53:38 INFO DiskBlockManager: Created local directory at /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1520619475936_0001/blockmgr-0bc31b66-e462-4972-9989-728ed680e432
  2525. 18/03/09 18:53:38 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
  2526. 18/03/09 18:53:39 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@bcaf3c62a630:36031
  2527. 18/03/09 18:53:39 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  2528. 18/03/09 18:53:39 INFO Executor: Starting executor ID 1 on host bf263d1bde4d
  2529. 18/03/09 18:53:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38407.
  2530. 18/03/09 18:53:39 INFO NettyBlockTransferService: Server created on bf263d1bde4d:38407
  2531. 18/03/09 18:53:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  2532. 18/03/09 18:53:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, bf263d1bde4d, 38407, None)
  2533. 18/03/09 18:53:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, bf263d1bde4d, 38407, None)
  2534. 18/03/09 18:53:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, bf263d1bde4d, 38407, None)
  2535. 18/03/09 18:53:48 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
  2536. 18/03/09 18:53:48 INFO CoarseGrainedExecutorBackend: Driver from bcaf3c62a630:36031 disconnected during shutdown
  2537. 18/03/09 18:53:48 INFO CoarseGrainedExecutorBackend: Driver from bcaf3c62a630:36031 disconnected during shutdown
  2538. 18/03/09 18:53:48 INFO MemoryStore: MemoryStore cleared
  2539. 18/03/09 18:53:48 INFO BlockManager: BlockManager stopped
  2540. 18/03/09 18:53:48 INFO ShutdownHookManager: Shutdown hook called
  2541.  
  2542. End of LogType:stderr
  2543. ***********************************************************************
  2544.  
  2545.  
  2546. End of LogType:stdout
  2547. ***********************************************************************
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!