API tools faq
paste
Advertisement
Guest User

sortie_hadoop

a guest
Mar 14th, 2015
263
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 11.38 KB | None | 0 0
raw download clone embed print report
  1. 15/03/14 16:40:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2. 15/03/14 16:40:13 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
  3. 15/03/14 16:40:13 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
  4. 15/03/14 16:40:13 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
  5. 15/03/14 16:40:13 INFO mapred.FileInputFormat: Total input paths to process : 1
  6. 15/03/14 16:40:13 INFO mapreduce.JobSubmitter: number of splits:1
  7. 15/03/14 16:40:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local515442_0001
  8. 15/03/14 16:40:14 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
  9. 15/03/14 16:40:14 INFO mapred.LocalJobRunner: OutputCommitter set in config null
  10. 15/03/14 16:40:14 INFO mapreduce.Job: Running job: job_local515442_0001
  11. 15/03/14 16:40:14 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
  12. 15/03/14 16:40:14 INFO mapred.LocalJobRunner: Waiting for map tasks
  13. 15/03/14 16:40:14 INFO mapred.LocalJobRunner: Starting task: attempt_local515442_0001_m_000000_0
  14. 15/03/14 16:40:15 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
  15. 15/03/14 16:40:15 INFO mapred.MapTask: Processing split: file:$HOME/Téléchargements/StatRDF/src/max_propriete/output_15-39/result1/part-00000:0+2934790
  16. 15/03/14 16:40:15 INFO mapred.MapTask: numReduceTasks: 1
  17. 15/03/14 16:40:15 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
  18. 15/03/14 16:40:15 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
  19. 15/03/14 16:40:15 INFO mapred.MapTask: soft limit at 83886080
  20. 15/03/14 16:40:15 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
  21. 15/03/14 16:40:15 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
  22. 15/03/14 16:40:15 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  23. 15/03/14 16:40:15 INFO streaming.PipeMapRed: PipeMapRed exec [$HOME/Téléchargements/StatRDF/src/max_propriete/./mapper2.py]
  24. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
  25. 15/03/14 16:40:15 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
  26. 15/03/14 16:40:15 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
  27. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
  28. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
  29. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
  30. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
  31. 15/03/14 16:40:15 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
  32. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
  33. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir
  34. 15/03/14 16:40:15 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
  35. 15/03/14 16:40:15 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
  36. 15/03/14 16:40:15 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
  37. 15/03/14 16:40:15 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]
  38. 15/03/14 16:40:15 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]
  39. 15/03/14 16:40:15 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]
  40. 15/03/14 16:40:15 INFO streaming.PipeMapRed: Records R/W=1352/1
  41. 15/03/14 16:40:15 INFO mapreduce.Job: Job job_local515442_0001 running in uber mode : false
  42. 15/03/14 16:40:15 INFO mapreduce.Job: map 0% reduce 0%
  43. 15/03/14 16:40:16 INFO streaming.PipeMapRed: R/W/S=10000/8734/0 in:NA [rec/s] out:NA [rec/s]
  44. 15/03/14 16:40:16 INFO streaming.PipeMapRed: MRErrorThread done
  45. 15/03/14 16:40:16 INFO streaming.PipeMapRed: mapRedFinished
  46. 15/03/14 16:40:16 INFO mapred.LocalJobRunner:
  47. 15/03/14 16:40:16 INFO mapred.MapTask: Starting flush of map output
  48. 15/03/14 16:40:16 INFO mapred.MapTask: Spilling map output
  49. 15/03/14 16:40:16 INFO mapred.MapTask: bufstart = 0; bufend = 2934790; bufvoid = 104857600
  50. 15/03/14 16:40:16 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26083480(104333920); length = 130917/6553600
  51. 15/03/14 16:40:16 INFO mapred.MapTask: Finished spill 0
  52. 15/03/14 16:40:16 INFO mapred.Task: Task:attempt_local515442_0001_m_000000_0 is done. And is in the process of committing
  53. 15/03/14 16:40:16 INFO mapred.LocalJobRunner: Records R/W=1352/1
  54. 15/03/14 16:40:16 INFO mapred.Task: Task 'attempt_local515442_0001_m_000000_0' done.
  55. 15/03/14 16:40:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local515442_0001_m_000000_0
  56. 15/03/14 16:40:16 INFO mapred.LocalJobRunner: map task executor complete.
  57. 15/03/14 16:40:16 INFO mapred.LocalJobRunner: Waiting for reduce tasks
  58. 15/03/14 16:40:16 INFO mapred.LocalJobRunner: Starting task: attempt_local515442_0001_r_000000_0
  59. 15/03/14 16:40:16 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
  60. 15/03/14 16:40:16 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@40cb3d8b
  61. 15/03/14 16:40:16 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
  62. 15/03/14 16:40:17 INFO reduce.EventFetcher: attempt_local515442_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
  63. 15/03/14 16:40:17 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local515442_0001_m_000000_0 decomp: 3000252 len: 3000256 to MEMORY
  64. 15/03/14 16:40:17 INFO reduce.InMemoryMapOutput: Read 3000252 bytes from map-output for attempt_local515442_0001_m_000000_0
  65. 15/03/14 16:40:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 3000252, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->3000252
  66. 15/03/14 16:40:17 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
  67. 15/03/14 16:40:17 INFO mapred.LocalJobRunner: 1 / 1 copied.
  68. 15/03/14 16:40:17 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
  69. 15/03/14 16:40:17 INFO mapred.Merger: Merging 1 sorted segments
  70. 15/03/14 16:40:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 3000247 bytes
  71. 15/03/14 16:40:17 INFO reduce.MergeManagerImpl: Merged 1 segments, 3000252 bytes to disk to satisfy reduce memory limit
  72. 15/03/14 16:40:17 INFO reduce.MergeManagerImpl: Merging 1 files, 3000256 bytes from disk
  73. 15/03/14 16:40:17 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
  74. 15/03/14 16:40:17 INFO mapred.Merger: Merging 1 sorted segments
  75. 15/03/14 16:40:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 3000247 bytes
  76. 15/03/14 16:40:17 INFO mapred.LocalJobRunner: 1 / 1 copied.
  77. 15/03/14 16:40:17 INFO streaming.PipeMapRed: PipeMapRed exec [$HOME/Téléchargements/StatRDF/src/max_propriete/./reducer2.py]
  78. 15/03/14 16:40:17 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
  79. 15/03/14 16:40:17 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  80. 15/03/14 16:40:17 INFO mapreduce.Job: map 100% reduce 0%
  81. 15/03/14 16:40:18 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
  82. 15/03/14 16:40:18 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]
  83. 15/03/14 16:40:18 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]
  84. 15/03/14 16:40:18 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]
  85. 15/03/14 16:40:18 INFO streaming.PipeMapRed: Records R/W=1352/1
  86. 15/03/14 16:40:18 INFO streaming.PipeMapRed: MRErrorThread done
  87. 15/03/14 16:40:18 WARN streaming.PipeMapRed: java.io.IOException: Broken pipe
  88. 15/03/14 16:40:18 INFO streaming.PipeMapRed: mapRedFinished
  89. 15/03/14 16:40:18 WARN streaming.PipeMapRed: java.io.IOException: Stream Closed
  90. 15/03/14 16:40:18 INFO streaming.PipeMapRed: mapRedFinished
  91. 15/03/14 16:40:18 INFO mapred.LocalJobRunner: reduce task executor complete.
  92. 15/03/14 16:40:18 WARN mapred.LocalJobRunner: job_local515442_0001
  93. java.lang.Exception: java.io.IOException: subprocess exited successfully
  94. R/W/S=1352/10/0 in:NA [rec/s] out:NA [rec/s]
  95. minRecWrittenToEnableSkip_=9223372036854775807 HOST=null
  96. USER=XXX
  97. HADOOP_USER=null
  98. last tool output: |http://data.europeana.eu/aggregation/europeana/08703/01190334615FF214D843AB9295189F6564D701ED 10|
  99. Broken pipe
  100. at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
  101. at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
  102. Caused by: java.io.IOException: subprocess exited successfully
  103. R/W/S=1352/10/0 in:NA [rec/s] out:NA [rec/s]
  104. minRecWrittenToEnableSkip_=9223372036854775807 HOST=null
  105. USER=XXX
  106. HADOOP_USER=null
  107. last tool output: |http://data.europeana.eu/aggregation/europeana/08703/01190334615FF214D843AB9295189F6564D701ED 10|
  108. Broken pipe
  109. at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:129)
  110. at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
  111. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
  112. at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
  113. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  114. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  115. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  116. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  117. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  118. at java.lang.Thread.run(Thread.java:701)
  119. 15/03/14 16:40:18 INFO mapreduce.Job: Job job_local515442_0001 failed with state FAILED due to: NA
  120. 15/03/14 16:40:19 INFO mapreduce.Job: Counters: 33
  121. File System Counters
  122. FILE: Number of bytes read=3062917
  123. FILE: Number of bytes written=3358691
  124. FILE: Number of read operations=0
  125. FILE: Number of large read operations=0
  126. FILE: Number of write operations=0
  127. Map-Reduce Framework
  128. Map input records=32730
  129. Map output records=32730
  130. Map output bytes=2934790
  131. Map output materialized bytes=3000256
  132. Input split bytes=145
  133. Combine input records=0
  134. Combine output records=0
  135. Reduce input groups=0
  136. Reduce shuffle bytes=3000256
  137. Reduce input records=0
  138. Reduce output records=0
  139. Spilled Records=32730
  140. Shuffled Maps =1
  141. Failed Shuffles=0
  142. Merged Map outputs=1
  143. GC time elapsed (ms)=186
  144. CPU time spent (ms)=0
  145. Physical memory (bytes) snapshot=0
  146. Virtual memory (bytes) snapshot=0
  147. Total committed heap usage (bytes)=113520640
  148. Shuffle Errors
  149. BAD_ID=0
  150. CONNECTION=0
  151. IO_ERROR=0
  152. WRONG_LENGTH=0
  153. WRONG_MAP=0
  154. WRONG_REDUCE=0
  155. File Input Format Counters
  156. Bytes Read=2957738
  157. File Output Format Counters
  158. Bytes Written=0
  159. 15/03/14 16:40:19 ERROR streaming.StreamJob: Job not successful!
  160. Streaming Command Failed!
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!