API tools faq
paste
Advertisement
Guest User

pysparktrace

a guest
Mar 11th, 2018
371
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 8.89 KB | None | 0 0
raw download clone embed print report
  1. 18/03/11 18:45:23 INFO org.spark_project.jetty.util.log: Logging initialized @2766ms
  2. 18/03/11 18:45:23 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT
  3. 18/03/11 18:45:23 INFO org.spark_project.jetty.server.Server: Started @2857ms
  4. 18/03/11 18:45:23 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@46b7c1a0{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
  5. 18/03/11 18:45:24 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.3-hadoop2
  6. 18/03/11 18:45:25 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at cluster-main-m/<ip>:8032
  7. 18/03/11 18:45:27 WARN org.apache.hadoop.hdfs.DataStreamer: Caught exception
  8. java.lang.InterruptedException
  9. at java.lang.Object.wait(Native Method)
  10. at java.lang.Thread.join(Thread.java:1252)
  11. at java.lang.Thread.join(Thread.java:1326)
  12. at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:973)
  13. at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:624)
  14. at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:801)
  15. 18/03/11 18:45:27 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1519879216511_0014
  16. Traceback (most recent call last):
  17. File "/tmp/job-eb92287a/streaming.py", line 17, in <module>
  18. .option("startingOffsets", "earliest") \
  19. File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/streaming.py", line 397, in load
  20.  
  21. File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  22. File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  23. File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
  24. py4j.protocol.Py4JJavaError: An error occurred while calling o58.load.
  25. : java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at http://spark.apache.org/third-party-projects.html
  26. at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
  27. at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
  28. at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
  29. at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:195)
  30. at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:87)
  31. at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:87)
  32. at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
  33. at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:150)
  34. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  35. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  36. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  37. at java.lang.reflect.Method.invoke(Method.java:498)
  38. at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  39. at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
  40. at py4j.Gateway.invoke(Gateway.java:280)
  41. at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  42. at py4j.commands.CallCommand.execute(CallCommand.java:79)
  43. at py4j.GatewayConnection.run(GatewayConnection.java:214)
  44. at java.lang.Thread.run(Thread.java:748)
  45. Caused by: java.lang.ClassNotFoundException: kafka.DefaultSource
  46. at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  47. at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  48. at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  49. at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21$$anonfun$apply$12.apply(DataSource.scala:533)
  50. at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21$$anonfun$apply$12.apply(DataSource.scala:533)
  51. at scala.util.Try$.apply(Try.scala:192)
  52. at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21.apply(DataSource.scala:533)
  53. at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21.apply(DataSource.scala:533)
  54. at scala.util.Try.orElse(Try.scala:84)
  55. at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:533)
  56. ... 18 more
  57.  
  58. 18/03/11 18:45:36 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@46b7c1a0{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
  59. 18/03/11 18:45:36 WARN org.apache.hadoop.ipc.Client: interrupted waiting to send rpc request to server
  60. java.lang.InterruptedException
  61. at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
  62. at java.util.concurrent.FutureTask.get(FutureTask.java:191)
  63. at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1135)
  64. at org.apache.hadoop.ipc.Client.call(Client.java:1384)
  65. at org.apache.hadoop.ipc.Client.call(Client.java:1342)
  66. at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
  67. at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
  68. at com.sun.proxy.$Proxy15.getApplicationReport(Unknown Source)
  69. at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:228)
  70. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  71. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  72. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  73. at java.lang.reflect.Method.invoke(Method.java:498)
  74. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
  75. at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
  76. at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
  77. at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
  78. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
  79. at com.sun.proxy.$Proxy16.getApplicationReport(Unknown Source)
  80. at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:480)
  81. at org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:284)
  82. at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:998)
  83. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:105)
  84. 18/03/11 18:45:36 ERROR org.apache.spark.deploy.yarn.Client: Failed to contact YARN for application application_1519879216511_0014.
  85. java.io.IOException: java.lang.InterruptedException
  86. at org.apache.hadoop.ipc.Client.call(Client.java:1390)
  87. at org.apache.hadoop.ipc.Client.call(Client.java:1342)
  88. at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
  89. at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
  90. at com.sun.proxy.$Proxy15.getApplicationReport(Unknown Source)
  91. at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:228)
  92. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  93. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  94. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  95. at java.lang.reflect.Method.invoke(Method.java:498)
  96. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
  97. at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
  98. at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
  99. at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
  100. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
  101. at com.sun.proxy.$Proxy16.getApplicationReport(Unknown Source)
  102. at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:480)
  103. at org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:284)
  104. at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:998)
  105. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:105)
  106. Caused by: java.lang.InterruptedException
  107. at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
  108. at java.util.concurrent.FutureTask.get(FutureTask.java:191)
  109. at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1135)
  110. at org.apache.hadoop.ipc.Client.call(Client.java:1384)
  111. ... 19 more
  112. 18/03/11 18:45:36 ERROR org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FAILED!
  113. Job output is complete
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!