Advertisement
Guest User

Untitled

a guest
Nov 21st, 2014
174
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 3.51 KB | None | 0 0
  1. 14/11/21 12:39:23 ERROR TaskSetManager: Task 3967.0:0 failed 4 times; aborting job
  2. org.apache.spark.SparkException: Job aborted due to stage failure: Task 3967.0:0 failed 4 times, most recent failure: Exception failure in TID 43518 on host ********: java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  3. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
  4. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
  5. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
  6. Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3967.0:0 failed 4 times, most recent failure: Exception failure in TID 43518 on host ********: java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  7. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
  8. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
  9. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
  10.  
  11. 14/11/21 12:34:18 INFO MemoryStore: Block input-0-1416573258200 stored as bytes to memory (size 85.8 KB, free 2.3 GB)
  12. 14/11/21 12:34:18 INFO BlockManagerMaster: Updated info of block input-0-1416573258200
  13. 14/11/21 12:34:18 INFO BlockGenerator: Pushed block input-0-1416573258200
  14. org.apache.spark.SparkException: Error sending message to BlockManagerMaster [message = GetLocations(input-0-1416573258200)]
  15. java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  16. 14/11/21 12:37:35 INFO BlockManagerInfo: Added input-0-1416573258200 in memory on ********:43117 (size: 85.8 KB, free: 2.3 GB)
  17. org.apache.spark.SparkException: Error sending message to BlockManagerMaster [message = GetLocations(input-0-1416573258200)]
  18. java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  19. org.apache.spark.SparkException: Job aborted due to stage failure: Task 3967.0:0 failed 4 times, most recent failure: Exception failure in TID 43518 on host ********: java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  20. java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  21. Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3967.0:0 failed 4 times, most recent failure: Exception failure in TID 43518 on host ********: java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  22. java.lang.Exception: Could not compute split, block input-0-1416573258200 not found
  23.  
  24. SparkConf conf = new SparkConf();
  25. JavaSparkContext sc = new JavaSparkContext(conf);
  26. JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000));
  27. jssc.checkpoint(checkpointDir);
  28.  
  29. HashMap<String, Integer> topics = new HashMap<String, Integer>();
  30. topics.put(KAFKA_TOPIC, 1);
  31.  
  32. HashMap<String, String> kafkaParams = new HashMap<String, String>();
  33. kafkaParams.put("group.id", "spark-streaming-test");
  34. kafkaParams.put("zookeeper.connect", ZOOKEEPER_QUORUM);
  35. kafkaParams.put("zookeeper.connection.timeout.ms", "1000");
  36. kafkaParams.put("auto.offset.reset", "smallest");
  37.  
  38. JavaPairReceiverInputDStream<String, String> kafkaStream =
  39. KafkaUtils.createStream(jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topics, StorageLevels.MEMORY_AND_DISK_SER);
  40.  
  41. JavaPairDStream<String, String> streamPair = kafkaStream.flatMapToPair(...).reduceByKey(...);
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement