API tools faq
paste
Advertisement
greenz

Maprederror

greenz
Jun 19th, 2013
214
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 4.88 KB | None | 0 0
raw download clone embed print report
  1. hduser@intel-HP-Pavilion-g6-Notebook-PC:~/hduser/hadoop$ sudo ./bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.4.jar -file /home/hduser/map.py -mapper /home/hduser/map.py -file /home/hduser/red.py -reducer /home/hduser/red.py -input /home/hduser/tmp/cddb.txt -output /home/hduser/op1
  2. packageJobJar: [/home/hduser/map.py, /home/hduser/red.py] [] /tmp/streamjob7455767556382290755.jar tmpDir=null
  3. 13/06/20 12:43:55 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  4. 13/06/20 12:43:55 WARN snappy.LoadSnappy: Snappy native library not loaded
  5. 13/06/20 12:43:55 INFO mapred.FileInputFormat: Total input paths to process : 1
  6. 13/06/20 12:43:55 WARN mapred.LocalJobRunner: LocalJobRunner does not support symlinking into current working dir.
  7. 13/06/20 12:43:56 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-root/mapred/local]
  8. 13/06/20 12:43:56 INFO streaming.StreamJob: Running job: job_local_0001
  9. 13/06/20 12:43:56 INFO streaming.StreamJob: Job running in-process (local Hadoop)
  10. 13/06/20 12:43:56 INFO util.ProcessTree: setsid exited with exit code 0
  11. 13/06/20 12:43:56 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@e2081
  12. 13/06/20 12:43:56 INFO mapred.MapTask: numReduceTasks: 1
  13. 13/06/20 12:43:56 INFO mapred.MapTask: io.sort.mb = 100
  14. 13/06/20 12:43:56 INFO mapred.MapTask: data buffer = 79691776/99614720
  15. 13/06/20 12:43:56 INFO mapred.MapTask: record buffer = 262144/327680
  16. 13/06/20 12:43:56 INFO streaming.PipeMapRed: PipeMapRed exec [/home/hduser/hduser/hadoop/./map.py]
  17. 13/06/20 12:43:56 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
  18. 13/06/20 12:43:57 INFO streaming.StreamJob: map 0% reduce 0%
  19. 13/06/20 12:44:02 INFO mapred.LocalJobRunner: file:/home/hduser/tmp/cddb.txt:0+1205
  20. 13/06/20 12:44:03 INFO streaming.StreamJob: map 100% reduce 0%
  21. 13/06/20 12:48:11 INFO streaming.PipeMapRed: Records R/W=9/1
  22. 13/06/20 12:48:11 INFO streaming.PipeMapRed: MRErrorThread done
  23. 13/06/20 12:48:11 INFO streaming.PipeMapRed: mapRedFinished
  24. 13/06/20 12:48:11 INFO mapred.MapTask: Starting flush of map output
  25. 13/06/20 12:48:11 INFO mapred.MapTask: Finished spill 0
  26. 13/06/20 12:48:11 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  27. 13/06/20 12:48:11 INFO mapred.LocalJobRunner: Records R/W=9/1
  28. 13/06/20 12:48:11 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
  29. 13/06/20 12:48:11 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1c84be9
  30. 13/06/20 12:48:11 INFO mapred.LocalJobRunner:
  31. 13/06/20 12:48:11 INFO mapred.Merger: Merging 1 sorted segments
  32. 13/06/20 12:48:11 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1356 bytes
  33. 13/06/20 12:48:11 INFO mapred.LocalJobRunner:
  34. 13/06/20 12:48:11 INFO streaming.PipeMapRed: PipeMapRed exec [/home/hduser/hduser/hadoop/./red.py]
  35. 13/06/20 12:48:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
  36. 13/06/20 12:48:11 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]
  37. Traceback (most recent call last):
  38. File "/home/hduser/hduser/hadoop/./red.py", line 30, in <module>
  39. main()
  40. File "/home/hduser/hduser/hadoop/./red.py", line 19, in main
  41. for similarity, group in groupby(data, itemgetter(0), reverse=True):
  42. TypeError: groupby() takes at most 2 arguments (3 given)
  43. 13/06/20 12:48:11 INFO streaming.PipeMapRed: MRErrorThread done
  44. 13/06/20 12:48:11 INFO streaming.PipeMapRed: PipeMapRed failed!
  45. java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
  46. at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
  47. at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
  48. at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137)
  49. at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:529)
  50. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
  51. at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
  52. 13/06/20 12:48:11 WARN mapred.LocalJobRunner: job_local_0001
  53. java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
  54. at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
  55. at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
  56. at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137)
  57. at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:529)
  58. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
  59. at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
  60. 13/06/20 12:48:12 INFO streaming.StreamJob: Job running in-process (local Hadoop)
  61. 13/06/20 12:48:12 ERROR streaming.StreamJob: Job not successful. Error: NA
  62. 13/06/20 12:48:12 INFO streaming.StreamJob: killJob...
  63. Streaming Command Failed!
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!