Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- id1, id2
- id2, id3
- ...
- idnnnn, idnnnn+1
- joinData = graphTuples.join(listIds)
- Job aborted due to stage failure: Task 351 in stage 2.0 failed 4 times, most recent failure: Lost task 351.3 in stage 2.0 (TID 476, c536.ant-net): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
- File "/data/3/tmp/hadoop-hadoop/nm-local-dir/usercache/user/appcache/application_1468851295159_0020/container_1468851295159_0020_01_000016/pyspark.zip/pyspark/worker.py", line 111, in main
- process()
- File "/data/3/tmp/hadoop-hadoop/nm-local-dir/usercache/user/appcache/application_1468851295159_0020/container_1468851295159_0020_01_000016/pyspark.zip/pyspark/worker.py", line 106, in process
- serializer.dump_stream(func(split_index, iterator), outfile)
- File "/data/3/tmp/hadoop-hadoop/nm-local-dir/usercache/user/appcache/application_1468851295159_0020/container_1468851295159_0020_01_000016/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
- vs = list(itertools.islice(iterator, batch))
- File "/usr/local/spark/python/pyspark/rdd.py", line 1898, in <lambda>
- IndexError: list index out of range
- at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
- at org.apache.spark.api.python.PythonRunner$$anon$1.next(PythonRDD.scala:129)
- at org.apache.spark.api.python.PythonRunner$$anon$1.next(PythonRDD.scala:125)
- at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43)
- at scala.collection.Iterator$class.foreach(Iterator.scala:727)
- at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
- at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:452)
- at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:280)
- at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765)
- at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:239)
- Driver stacktrace:
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement