SHARE
TWEET

Untitled

a guest Oct 10th, 2019 77 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. 19/10/10 09:15:39 WARN DependencyUtils: Skip remote jar s3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar.
  2. 19/10/10 09:15:40 INFO RMProxy: Connecting to ResourceManager at ip-172-31-18-212.ec2.internal/172.31.18.212:8032
  3. 19/10/10 09:15:40 INFO Client: Requesting a new application from cluster with 4 NodeManagers
  4. 19/10/10 09:15:40 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (53248 MB per container)
  5. 19/10/10 09:15:40 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
  6. 19/10/10 09:15:40 INFO Client: Setting up container launch context for our AM
  7. 19/10/10 09:15:40 INFO Client: Setting up the launch environment for our AM container
  8. 19/10/10 09:15:40 INFO Client: Preparing resources for our AM container
  9. 19/10/10 09:15:40 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  10. 19/10/10 09:15:42 INFO Client: Uploading resource file:/mnt/tmp/spark-7d80d4d2-5091-4503-bf90-91d2c360f4e3/__spark_libs__1144593390807618974.zip -> hdfs://ip-172-31-18-212.ec2.internal:8020/user/hadoop/.sparkStaging/application_1570698154192_0002/__spark_libs__1144593390807618974.zip
  11. 19/10/10 09:15:44 INFO Client: Uploading resource s3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar -> hdfs://ip-172-31-18-212.ec2.internal:8020/user/hadoop/.sparkStaging/application_1570698154192_0002/lab1_2.11-1.0.jar
  12. 19/10/10 09:15:45 INFO S3NativeFileSystem: Opening 's3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar' for reading
  13. 19/10/10 09:15:45 INFO Client: Uploading resource file:/mnt/tmp/spark-7d80d4d2-5091-4503-bf90-91d2c360f4e3/__spark_conf__5373847283083138489.zip -> hdfs://ip-172-31-18-212.ec2.internal:8020/user/hadoop/.sparkStaging/application_1570698154192_0002/__spark_conf__.zip
  14. 19/10/10 09:15:45 INFO SecurityManager: Changing view acls to: hadoop
  15. 19/10/10 09:15:45 INFO SecurityManager: Changing modify acls to: hadoop
  16. 19/10/10 09:15:45 INFO SecurityManager: Changing view acls groups to:
  17. 19/10/10 09:15:45 INFO SecurityManager: Changing modify acls groups to:
  18. 19/10/10 09:15:45 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
  19. 19/10/10 09:15:46 INFO Client: Submitting application application_1570698154192_0002 to ResourceManager
  20. 19/10/10 09:15:46 INFO YarnClientImpl: Submitted application application_1570698154192_0002
  21. 19/10/10 09:15:47 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  22. 19/10/10 09:15:47 INFO Client:
  23.      client token: N/A
  24.      diagnostics: AM container is launched, waiting for AM container to Register with RM
  25.      ApplicationMaster host: N/A
  26.      ApplicationMaster RPC port: -1
  27.      queue: default
  28.      start time: 1570698946500
  29.      final status: UNDEFINED
  30.      tracking URL: http://ip-172-31-18-212.ec2.internal:20888/proxy/application_1570698154192_0002/
  31.      user: hadoop
  32. 19/10/10 09:15:48 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  33. 19/10/10 09:15:49 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  34. 19/10/10 09:15:50 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  35. 19/10/10 09:15:51 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  36. 19/10/10 09:15:52 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  37. 19/10/10 09:15:52 INFO Client:
  38.      client token: N/A
  39.      diagnostics: N/A
  40.      ApplicationMaster host: ip-172-31-17-220.ec2.internal
  41.      ApplicationMaster RPC port: 45841
  42.      queue: default
  43.      start time: 1570698946500
  44.      final status: UNDEFINED
  45.      tracking URL: http://ip-172-31-18-212.ec2.internal:20888/proxy/application_1570698154192_0002/
  46.      user: hadoop
  47. 19/10/10 09:15:53 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  48. 19/10/10 09:15:54 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  49. 19/10/10 09:15:55 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  50. 19/10/10 09:15:56 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  51. 19/10/10 09:15:57 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  52. 19/10/10 09:15:58 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  53. 19/10/10 09:15:59 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  54. 19/10/10 09:16:00 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  55. 19/10/10 09:16:01 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  56. 19/10/10 09:16:02 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  57. 19/10/10 09:16:03 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  58. 19/10/10 09:16:04 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  59. 19/10/10 09:16:05 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  60. 19/10/10 09:16:06 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  61. 19/10/10 09:16:07 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  62. 19/10/10 09:16:08 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  63. 19/10/10 09:16:09 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  64. 19/10/10 09:16:10 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  65. 19/10/10 09:16:11 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  66. 19/10/10 09:16:12 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  67. 19/10/10 09:16:13 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  68. 19/10/10 09:16:14 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  69. 19/10/10 09:16:15 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  70. 19/10/10 09:16:16 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  71. 19/10/10 09:16:17 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  72. 19/10/10 09:16:18 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  73. 19/10/10 09:16:19 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  74. 19/10/10 09:16:20 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  75. 19/10/10 09:16:21 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  76. 19/10/10 09:16:22 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  77. 19/10/10 09:16:23 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  78. 19/10/10 09:16:24 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  79. 19/10/10 09:16:25 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  80. 19/10/10 09:16:26 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  81. 19/10/10 09:16:27 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  82. 19/10/10 09:16:28 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  83. 19/10/10 09:16:29 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  84. 19/10/10 09:16:30 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  85. 19/10/10 09:16:31 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  86. 19/10/10 09:16:32 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  87. 19/10/10 09:16:33 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  88. 19/10/10 09:16:34 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  89. 19/10/10 09:16:35 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  90. 19/10/10 09:16:36 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  91. 19/10/10 09:16:37 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  92. 19/10/10 09:16:38 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  93. 19/10/10 09:16:39 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  94. 19/10/10 09:16:40 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  95. 19/10/10 09:16:41 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  96. 19/10/10 09:16:42 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  97. 19/10/10 09:16:43 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  98. 19/10/10 09:16:44 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  99. 19/10/10 09:16:44 INFO Client:
  100.      client token: N/A
  101.      diagnostics: AM container is launched, waiting for AM container to Register with RM
  102.      ApplicationMaster host: N/A
  103.      ApplicationMaster RPC port: -1
  104.      queue: default
  105.      start time: 1570698946500
  106.      final status: UNDEFINED
  107.      tracking URL: http://ip-172-31-18-212.ec2.internal:20888/proxy/application_1570698154192_0002/
  108.      user: hadoop
  109. 19/10/10 09:16:45 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  110. 19/10/10 09:16:46 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  111. 19/10/10 09:16:47 INFO Client: Application report for application_1570698154192_0002 (state: ACCEPTED)
  112. 19/10/10 09:16:48 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  113. 19/10/10 09:16:48 INFO Client:
  114.      client token: N/A
  115.      diagnostics: N/A
  116.      ApplicationMaster host: ip-172-31-25-170.ec2.internal
  117.      ApplicationMaster RPC port: 41631
  118.      queue: default
  119.      start time: 1570698946500
  120.      final status: UNDEFINED
  121.      tracking URL: http://ip-172-31-18-212.ec2.internal:20888/proxy/application_1570698154192_0002/
  122.      user: hadoop
  123. 19/10/10 09:16:49 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  124. 19/10/10 09:16:50 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  125. 19/10/10 09:16:51 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  126. 19/10/10 09:16:52 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  127. 19/10/10 09:16:53 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  128. 19/10/10 09:16:54 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  129. 19/10/10 09:16:55 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  130. 19/10/10 09:16:56 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  131. 19/10/10 09:16:57 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  132. 19/10/10 09:16:58 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  133. 19/10/10 09:16:59 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  134. 19/10/10 09:17:00 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  135. 19/10/10 09:17:01 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  136. 19/10/10 09:17:02 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  137. 19/10/10 09:17:03 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  138. 19/10/10 09:17:04 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  139. 19/10/10 09:17:05 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  140. 19/10/10 09:17:06 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  141. 19/10/10 09:17:07 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  142. 19/10/10 09:17:08 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  143. 19/10/10 09:17:09 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  144. 19/10/10 09:17:10 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  145. 19/10/10 09:17:11 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  146. 19/10/10 09:17:12 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  147. 19/10/10 09:17:13 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  148. 19/10/10 09:17:14 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  149. 19/10/10 09:17:15 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  150. 19/10/10 09:17:16 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  151. 19/10/10 09:17:17 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  152. 19/10/10 09:17:18 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  153. 19/10/10 09:17:19 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  154. 19/10/10 09:17:20 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  155. 19/10/10 09:17:21 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  156. 19/10/10 09:17:22 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  157. 19/10/10 09:17:23 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  158. 19/10/10 09:17:24 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  159. 19/10/10 09:17:25 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  160. 19/10/10 09:17:26 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  161. 19/10/10 09:17:27 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  162. 19/10/10 09:17:28 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  163. 19/10/10 09:17:29 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  164. 19/10/10 09:17:30 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  165. 19/10/10 09:17:31 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  166. 19/10/10 09:17:32 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  167. 19/10/10 09:17:33 INFO Client: Application report for application_1570698154192_0002 (state: RUNNING)
  168. 19/10/10 09:17:34 INFO Client: Application report for application_1570698154192_0002 (state: FAILED)
  169. 19/10/10 09:17:34 INFO Client:
  170.      client token: N/A
  171.      diagnostics: Application application_1570698154192_0002 failed 2 times due to AM Container for appattempt_1570698154192_0002_000002 exited with  exitCode: -104
  172. Failing this attempt.Diagnostics: Container [pid=18260,containerID=container_1570698154192_0002_02_000001] is running beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory used; 3.6 GB of 6.9 GB virtual memory used. Killing container.
  173. Dump of the process-tree for container_1570698154192_0002_02_000001 :
  174.     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
  175.     |- 18266 18260 18260 18260 (java) 10738 360 3763003392 360750 /usr/lib/jvm/java-openjdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/tmp -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError=kill -9 %p -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class DatasetApplication --jar s3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar --arg s3://gdelt-open-data/v2/gkg/201*.gkg.csv --arg s3://et4310-lab2-ruben-us-east-1 --properties-file /mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/__spark_conf__/__spark_conf__.properties
  176.     |- 18260 18258 18260 18260 (bash) 0 0 115871744 708 /bin/bash -c LD_LIBRARY_PATH="/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:::/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native::/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native" /usr/lib/jvm/java-openjdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/tmp '-XX:+UseConcMarkSweepGC' '-XX:CMSInitiatingOccupancyFraction=70' '-XX:MaxHeapFreeRatio=70' '-XX:+CMSClassUnloadingEnabled' '-XX:OnOutOfMemoryError=kill -9 %p' -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'DatasetApplication' --jar s3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar --arg 's3://gdelt-open-data/v2/gkg/201*.gkg.csv' --arg 's3://et4310-lab2-ruben-us-east-1' --properties-file /mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001/stdout 2> /var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001/stderr
  177.  
  178. Container killed on request. Exit code is 143
  179. Container exited with a non-zero exit code 143
  180. For more detailed output, check the application tracking page: http://ip-172-31-18-212.ec2.internal:8088/cluster/app/application_1570698154192_0002 Then click on links to logs of each attempt.
  181. . Failing the application.
  182.      ApplicationMaster host: N/A
  183.      ApplicationMaster RPC port: -1
  184.      queue: default
  185.      start time: 1570698946500
  186.      final status: FAILED
  187.      tracking URL: http://ip-172-31-18-212.ec2.internal:8088/cluster/app/application_1570698154192_0002
  188.      user: hadoop
  189. 19/10/10 09:17:34 INFO Client: Deleted staging directory hdfs://ip-172-31-18-212.ec2.internal:8020/user/hadoop/.sparkStaging/application_1570698154192_0002
  190. 19/10/10 09:17:34 ERROR Client: Application diagnostics message: Application application_1570698154192_0002 failed 2 times due to AM Container for appattempt_1570698154192_0002_000002 exited with  exitCode: -104
  191. Failing this attempt.Diagnostics: Container [pid=18260,containerID=container_1570698154192_0002_02_000001] is running beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory used; 3.6 GB of 6.9 GB virtual memory used. Killing container.
  192. Dump of the process-tree for container_1570698154192_0002_02_000001 :
  193.     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
  194.     |- 18266 18260 18260 18260 (java) 10738 360 3763003392 360750 /usr/lib/jvm/java-openjdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/tmp -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError=kill -9 %p -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class DatasetApplication --jar s3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar --arg s3://gdelt-open-data/v2/gkg/201*.gkg.csv --arg s3://et4310-lab2-ruben-us-east-1 --properties-file /mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/__spark_conf__/__spark_conf__.properties
  195.     |- 18260 18258 18260 18260 (bash) 0 0 115871744 708 /bin/bash -c LD_LIBRARY_PATH="/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:::/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native::/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native" /usr/lib/jvm/java-openjdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/tmp '-XX:+UseConcMarkSweepGC' '-XX:CMSInitiatingOccupancyFraction=70' '-XX:MaxHeapFreeRatio=70' '-XX:+CMSClassUnloadingEnabled' '-XX:OnOutOfMemoryError=kill -9 %p' -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'DatasetApplication' --jar s3://et4310-lab2-ruben-us-east-1/lab1_2.11-1.0.jar --arg 's3://gdelt-open-data/v2/gkg/201*.gkg.csv' --arg 's3://et4310-lab2-ruben-us-east-1' --properties-file /mnt3/yarn/usercache/hadoop/appcache/application_1570698154192_0002/container_1570698154192_0002_02_000001/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001/stdout 2> /var/log/hadoop-yarn/containers/application_1570698154192_0002/container_1570698154192_0002_02_000001/stderr
  196.  
  197. Container killed on request. Exit code is 143
  198. Container exited with a non-zero exit code 143
  199. For more detailed output, check the application tracking page: http://ip-172-31-18-212.ec2.internal:8088/cluster/app/application_1570698154192_0002 Then click on links to logs of each attempt.
  200. . Failing the application.
  201. Exception in thread "main" org.apache.spark.SparkException: Application application_1570698154192_0002 finished with failed status
  202.     at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
  203.     at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
  204.     at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
  205.     at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
  206.     at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
  207.     at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
  208.     at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
  209.     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
  210.     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  211. 19/10/10 09:17:34 INFO ShutdownHookManager: Shutdown hook called
  212. 19/10/10 09:17:34 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-cac55d4e-a491-4b44-9ecb-69f3dea8107c
  213. 19/10/10 09:17:34 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-7d80d4d2-5091-4503-bf90-91d2c360f4e3
  214. Command exiting with ret '1'
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!
 
Top