API tools faq
paste
Guest User

Mahout 0.9 Streaming K-Means MR log

a guest
Jan 19th, 2014
96
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 64.92 KB | None | 0 0
raw download clone embed print report
  1. Please select a number to choose the corresponding clustering algorithm
  2. 1. kmeans clustering
  3. 2. fuzzykmeans clustering
  4. 3. lda clustering
  5. 4. streamingkmeans clustering
  6. Enter your choice : ok. You chose 4 and we'll use streamingkmeans Clustering
  7. creating work directory at /tmp/mahout-work-frank
  8. MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
  9. Warning: $HADOOP_HOME is deprecated.
  10.  
  11. Running on hadoop, using /opt/hadoop/bin/hadoop and HADOOP_CONF_DIR=$HADOOP_HOME/conf
  12. MAHOUT-JOB: /home/frank/Desktop/mahout-distribution-0.9/examples/target/mahout-examples-0.9-job.jar
  13. Warning: $HADOOP_HOME is deprecated.
  14.  
  15. 14/01/19 17:47:03 INFO vectorizer.SparseVectorsFromSequenceFiles: Maximum n-gram size is: 1
  16. 14/01/19 17:47:03 INFO common.HadoopUtil: Deleting /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans
  17. 14/01/19 17:47:03 INFO vectorizer.SparseVectorsFromSequenceFiles: Minimum LLR value: 1.0
  18. 14/01/19 17:47:03 INFO vectorizer.SparseVectorsFromSequenceFiles: Number of reduce tasks: 1
  19. 14/01/19 17:47:03 INFO vectorizer.SparseVectorsFromSequenceFiles: Tokenizing documents in /tmp/mahout-work-frank/reuters-out-seqdir
  20. 14/01/19 17:47:03 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  21. 14/01/19 17:47:03 INFO input.FileInputFormat: Total input paths to process : 1
  22. 14/01/19 17:47:03 INFO mapred.JobClient: Running job: job_local1192442957_0001
  23. 14/01/19 17:47:03 INFO mapred.LocalJobRunner: Waiting for map tasks
  24. 14/01/19 17:47:03 INFO mapred.LocalJobRunner: Starting task: attempt_local1192442957_0001_m_000000_0
  25. 14/01/19 17:47:03 INFO util.ProcessTree: setsid exited with exit code 0
  26. 14/01/19 17:47:03 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5e4a4c05
  27. 14/01/19 17:47:03 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir/chunk-0:0+18403906
  28. 14/01/19 17:47:04 INFO mapred.JobClient: map 0% reduce 0%
  29. 14/01/19 17:47:05 INFO mapred.Task: Task:attempt_local1192442957_0001_m_000000_0 is done. And is in the process of commiting
  30. 14/01/19 17:47:05 INFO mapred.LocalJobRunner:
  31. 14/01/19 17:47:05 INFO mapred.Task: Task attempt_local1192442957_0001_m_000000_0 is allowed to commit now
  32. 14/01/19 17:47:05 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1192442957_0001_m_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tokenized-documents
  33. 14/01/19 17:47:05 INFO mapred.LocalJobRunner:
  34. 14/01/19 17:47:05 INFO mapred.Task: Task 'attempt_local1192442957_0001_m_000000_0' done.
  35. 14/01/19 17:47:05 INFO mapred.LocalJobRunner: Finishing task: attempt_local1192442957_0001_m_000000_0
  36. 14/01/19 17:47:05 INFO mapred.LocalJobRunner: Map task executor complete.
  37. 14/01/19 17:47:05 INFO mapred.JobClient: map 100% reduce 0%
  38. 14/01/19 17:47:05 INFO mapred.JobClient: Job complete: job_local1192442957_0001
  39. 14/01/19 17:47:05 INFO mapred.JobClient: Counters: 12
  40. 14/01/19 17:47:05 INFO mapred.JobClient: File Output Format Counters
  41. 14/01/19 17:47:05 INFO mapred.JobClient: Bytes Written=15194092
  42. 14/01/19 17:47:05 INFO mapred.JobClient: File Input Format Counters
  43. 14/01/19 17:47:05 INFO mapred.JobClient: Bytes Read=18547698
  44. 14/01/19 17:47:05 INFO mapred.JobClient: FileSystemCounters
  45. 14/01/19 17:47:05 INFO mapred.JobClient: FILE_BYTES_READ=42605926
  46. 14/01/19 17:47:05 INFO mapred.JobClient: FILE_BYTES_WRITTEN=39492261
  47. 14/01/19 17:47:05 INFO mapred.JobClient: Map-Reduce Framework
  48. 14/01/19 17:47:05 INFO mapred.JobClient: Map input records=21578
  49. 14/01/19 17:47:05 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  50. 14/01/19 17:47:05 INFO mapred.JobClient: Spilled Records=0
  51. 14/01/19 17:47:05 INFO mapred.JobClient: Total committed heap usage (bytes)=360251392
  52. 14/01/19 17:47:05 INFO mapred.JobClient: CPU time spent (ms)=0
  53. 14/01/19 17:47:05 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  54. 14/01/19 17:47:05 INFO mapred.JobClient: SPLIT_RAW_BYTES=119
  55. 14/01/19 17:47:05 INFO mapred.JobClient: Map output records=21578
  56. 14/01/19 17:47:05 INFO vectorizer.SparseVectorsFromSequenceFiles: Creating Term Frequency Vectors
  57. 14/01/19 17:47:05 INFO vectorizer.DictionaryVectorizer: Creating dictionary from /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tokenized-documents and saving at /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/wordcount
  58. 14/01/19 17:47:05 INFO input.FileInputFormat: Total input paths to process : 1
  59. 14/01/19 17:47:05 INFO mapred.JobClient: Running job: job_local1030747702_0002
  60. 14/01/19 17:47:05 INFO mapred.LocalJobRunner: Waiting for map tasks
  61. 14/01/19 17:47:05 INFO mapred.LocalJobRunner: Starting task: attempt_local1030747702_0002_m_000000_0
  62. 14/01/19 17:47:05 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2d7bcd8b
  63. 14/01/19 17:47:05 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tokenized-documents/part-m-00000:0+15076300
  64. 14/01/19 17:47:05 INFO mapred.MapTask: io.sort.mb = 100
  65. 14/01/19 17:47:05 INFO mapred.MapTask: data buffer = 79691776/99614720
  66. 14/01/19 17:47:05 INFO mapred.MapTask: record buffer = 262144/327680
  67. 14/01/19 17:47:06 INFO mapred.MapTask: Spilling map output: record full = true
  68. 14/01/19 17:47:06 INFO mapred.MapTask: bufstart = 0; bufend = 3828344; bufvoid = 99614720
  69. 14/01/19 17:47:06 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
  70. 14/01/19 17:47:06 INFO mapred.MapTask: Finished spill 0
  71. 14/01/19 17:47:06 INFO mapred.MapTask: Spilling map output: record full = true
  72. 14/01/19 17:47:06 INFO mapred.MapTask: bufstart = 3828344; bufend = 7658606; bufvoid = 99614720
  73. 14/01/19 17:47:06 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
  74. 14/01/19 17:47:06 INFO mapred.JobClient: map 0% reduce 0%
  75. 14/01/19 17:47:06 INFO mapred.MapTask: Finished spill 1
  76. 14/01/19 17:47:06 INFO mapred.MapTask: Spilling map output: record full = true
  77. 14/01/19 17:47:06 INFO mapred.MapTask: bufstart = 7658606; bufend = 11485868; bufvoid = 99614720
  78. 14/01/19 17:47:06 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
  79. 14/01/19 17:47:07 INFO mapred.MapTask: Finished spill 2
  80. 14/01/19 17:47:07 INFO mapred.MapTask: Spilling map output: record full = true
  81. 14/01/19 17:47:07 INFO mapred.MapTask: bufstart = 11485868; bufend = 15312160; bufvoid = 99614720
  82. 14/01/19 17:47:07 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
  83. 14/01/19 17:47:07 INFO mapred.MapTask: Finished spill 3
  84. 14/01/19 17:47:07 INFO mapred.MapTask: Spilling map output: record full = true
  85. 14/01/19 17:47:07 INFO mapred.MapTask: bufstart = 15312160; bufend = 19138533; bufvoid = 99614720
  86. 14/01/19 17:47:07 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
  87. 14/01/19 17:47:07 INFO mapred.MapTask: Finished spill 4
  88. 14/01/19 17:47:07 INFO mapred.MapTask: Starting flush of map output
  89. 14/01/19 17:47:08 INFO mapred.MapTask: Finished spill 5
  90. 14/01/19 17:47:08 INFO mapred.Merger: Merging 6 sorted segments
  91. 14/01/19 17:47:08 INFO mapred.Merger: Down to the last merge-pass, with 6 segments left of total size: 3044575 bytes
  92. 14/01/19 17:47:08 INFO mapred.Task: Task:attempt_local1030747702_0002_m_000000_0 is done. And is in the process of commiting
  93. 14/01/19 17:47:08 INFO mapred.LocalJobRunner:
  94. 14/01/19 17:47:08 INFO mapred.Task: Task 'attempt_local1030747702_0002_m_000000_0' done.
  95. 14/01/19 17:47:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1030747702_0002_m_000000_0
  96. 14/01/19 17:47:08 INFO mapred.LocalJobRunner: Map task executor complete.
  97. 14/01/19 17:47:08 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@463b1057
  98. 14/01/19 17:47:08 INFO mapred.LocalJobRunner:
  99. 14/01/19 17:47:08 INFO mapred.Merger: Merging 1 sorted segments
  100. 14/01/19 17:47:08 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1305776 bytes
  101. 14/01/19 17:47:08 INFO mapred.LocalJobRunner:
  102. 14/01/19 17:47:08 INFO mapred.Task: Task:attempt_local1030747702_0002_r_000000_0 is done. And is in the process of commiting
  103. 14/01/19 17:47:08 INFO mapred.LocalJobRunner:
  104. 14/01/19 17:47:08 INFO mapred.Task: Task attempt_local1030747702_0002_r_000000_0 is allowed to commit now
  105. 14/01/19 17:47:08 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1030747702_0002_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/wordcount
  106. 14/01/19 17:47:08 INFO mapred.LocalJobRunner: reduce > reduce
  107. 14/01/19 17:47:08 INFO mapred.Task: Task 'attempt_local1030747702_0002_r_000000_0' done.
  108. 14/01/19 17:47:08 INFO mapred.JobClient: map 100% reduce 100%
  109. 14/01/19 17:47:08 INFO mapred.JobClient: Job complete: job_local1030747702_0002
  110. 14/01/19 17:47:08 INFO mapred.JobClient: Counters: 20
  111. 14/01/19 17:47:08 INFO mapred.JobClient: File Output Format Counters
  112. 14/01/19 17:47:08 INFO mapred.JobClient: Bytes Written=1000743
  113. 14/01/19 17:47:08 INFO mapred.JobClient: File Input Format Counters
  114. 14/01/19 17:47:08 INFO mapred.JobClient: Bytes Read=15194092
  115. 14/01/19 17:47:08 INFO mapred.JobClient: FileSystemCounters
  116. 14/01/19 17:47:08 INFO mapred.JobClient: FILE_BYTES_READ=171111564
  117. 14/01/19 17:47:08 INFO mapred.JobClient: FILE_BYTES_WRITTEN=137284179
  118. 14/01/19 17:47:08 INFO mapred.JobClient: Map-Reduce Framework
  119. 14/01/19 17:47:08 INFO mapred.JobClient: Reduce input groups=73413
  120. 14/01/19 17:47:08 INFO mapred.JobClient: Map output materialized bytes=1305780
  121. 14/01/19 17:47:08 INFO mapred.JobClient: Combine output records=246193
  122. 14/01/19 17:47:08 INFO mapred.JobClient: Map input records=21578
  123. 14/01/19 17:47:08 INFO mapred.JobClient: Reduce shuffle bytes=0
  124. 14/01/19 17:47:08 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  125. 14/01/19 17:47:08 INFO mapred.JobClient: Reduce output records=41807
  126. 14/01/19 17:47:08 INFO mapred.JobClient: Spilled Records=319606
  127. 14/01/19 17:47:08 INFO mapred.JobClient: Map output bytes=22501851
  128. 14/01/19 17:47:08 INFO mapred.JobClient: Total committed heap usage (bytes)=1050542080
  129. 14/01/19 17:47:08 INFO mapred.JobClient: CPU time spent (ms)=0
  130. 14/01/19 17:47:08 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  131. 14/01/19 17:47:08 INFO mapred.JobClient: SPLIT_RAW_BYTES=167
  132. 14/01/19 17:47:08 INFO mapred.JobClient: Map output records=1540960
  133. 14/01/19 17:47:08 INFO mapred.JobClient: Combine input records=1713740
  134. 14/01/19 17:47:08 INFO mapred.JobClient: Reduce input records=73413
  135. 14/01/19 17:47:09 INFO input.FileInputFormat: Total input paths to process : 1
  136. 14/01/19 17:47:09 INFO filecache.TrackerDistributedCacheManager: Creating dictionary.file-0 in /tmp/hadoop-frank/mapred/local/archive/-3614556510696807819_-998685638_728108391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans-work-6115709348464182811 with rwxr-xr-x
  137. 14/01/19 17:47:09 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/dictionary.file-0 as /tmp/hadoop-frank/mapred/local/archive/-3614556510696807819_-998685638_728108391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/dictionary.file-0
  138. 14/01/19 17:47:09 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/dictionary.file-0 as /tmp/hadoop-frank/mapred/local/archive/-3614556510696807819_-998685638_728108391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/dictionary.file-0
  139. 14/01/19 17:47:09 INFO mapred.JobClient: Running job: job_local1651471746_0003
  140. 14/01/19 17:47:09 INFO mapred.LocalJobRunner: Waiting for map tasks
  141. 14/01/19 17:47:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1651471746_0003_m_000000_0
  142. 14/01/19 17:47:09 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4709fa8b
  143. 14/01/19 17:47:09 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tokenized-documents/part-m-00000:0+15076300
  144. 14/01/19 17:47:09 INFO mapred.MapTask: io.sort.mb = 100
  145. 14/01/19 17:47:09 INFO mapred.MapTask: data buffer = 79691776/99614720
  146. 14/01/19 17:47:09 INFO mapred.MapTask: record buffer = 262144/327680
  147. 14/01/19 17:47:09 INFO mapred.MapTask: Starting flush of map output
  148. 14/01/19 17:47:09 INFO mapred.MapTask: Finished spill 0
  149. 14/01/19 17:47:09 INFO mapred.Task: Task:attempt_local1651471746_0003_m_000000_0 is done. And is in the process of commiting
  150. 14/01/19 17:47:09 INFO mapred.LocalJobRunner:
  151. 14/01/19 17:47:09 INFO mapred.Task: Task 'attempt_local1651471746_0003_m_000000_0' done.
  152. 14/01/19 17:47:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1651471746_0003_m_000000_0
  153. 14/01/19 17:47:09 INFO mapred.LocalJobRunner: Map task executor complete.
  154. 14/01/19 17:47:09 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@15902929
  155. 14/01/19 17:47:09 INFO mapred.LocalJobRunner:
  156. 14/01/19 17:47:09 INFO mapred.Merger: Merging 1 sorted segments
  157. 14/01/19 17:47:09 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 14870362 bytes
  158. 14/01/19 17:47:09 INFO mapred.LocalJobRunner:
  159. 14/01/19 17:47:10 INFO mapred.JobClient: map 100% reduce 0%
  160. 14/01/19 17:47:10 INFO mapred.Task: Task:attempt_local1651471746_0003_r_000000_0 is done. And is in the process of commiting
  161. 14/01/19 17:47:10 INFO mapred.LocalJobRunner:
  162. 14/01/19 17:47:10 INFO mapred.Task: Task attempt_local1651471746_0003_r_000000_0 is allowed to commit now
  163. 14/01/19 17:47:10 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1651471746_0003_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/partial-vectors-0
  164. 14/01/19 17:47:10 INFO mapred.LocalJobRunner: reduce > reduce
  165. 14/01/19 17:47:10 INFO mapred.Task: Task 'attempt_local1651471746_0003_r_000000_0' done.
  166. 14/01/19 17:47:11 INFO mapred.JobClient: map 100% reduce 100%
  167. 14/01/19 17:47:11 INFO mapred.JobClient: Job complete: job_local1651471746_0003
  168. 14/01/19 17:47:11 INFO mapred.JobClient: Counters: 20
  169. 14/01/19 17:47:11 INFO mapred.JobClient: File Output Format Counters
  170. 14/01/19 17:47:11 INFO mapred.JobClient: Bytes Written=17780621
  171. 14/01/19 17:47:11 INFO mapred.JobClient: File Input Format Counters
  172. 14/01/19 17:47:11 INFO mapred.JobClient: Bytes Read=15194092
  173. 14/01/19 17:47:11 INFO mapred.JobClient: FileSystemCounters
  174. 14/01/19 17:47:11 INFO mapred.JobClient: FILE_BYTES_READ=270285532
  175. 14/01/19 17:47:11 INFO mapred.JobClient: FILE_BYTES_WRITTEN=237731271
  176. 14/01/19 17:47:11 INFO mapred.JobClient: Map-Reduce Framework
  177. 14/01/19 17:47:11 INFO mapred.JobClient: Reduce input groups=21578
  178. 14/01/19 17:47:11 INFO mapred.JobClient: Map output materialized bytes=14870366
  179. 14/01/19 17:47:11 INFO mapred.JobClient: Combine output records=0
  180. 14/01/19 17:47:11 INFO mapred.JobClient: Map input records=21578
  181. 14/01/19 17:47:11 INFO mapred.JobClient: Reduce shuffle bytes=0
  182. 14/01/19 17:47:11 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  183. 14/01/19 17:47:11 INFO mapred.JobClient: Reduce output records=21578
  184. 14/01/19 17:47:11 INFO mapred.JobClient: Spilled Records=43156
  185. 14/01/19 17:47:11 INFO mapred.JobClient: Map output bytes=14791487
  186. 14/01/19 17:47:11 INFO mapred.JobClient: Total committed heap usage (bytes)=1255931904
  187. 14/01/19 17:47:11 INFO mapred.JobClient: CPU time spent (ms)=0
  188. 14/01/19 17:47:11 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  189. 14/01/19 17:47:11 INFO mapred.JobClient: SPLIT_RAW_BYTES=167
  190. 14/01/19 17:47:11 INFO mapred.JobClient: Map output records=21578
  191. 14/01/19 17:47:11 INFO mapred.JobClient: Combine input records=0
  192. 14/01/19 17:47:11 INFO mapred.JobClient: Reduce input records=21578
  193. 14/01/19 17:47:11 INFO input.FileInputFormat: Total input paths to process : 1
  194. 14/01/19 17:47:11 INFO mapred.JobClient: Running job: job_local602088658_0004
  195. 14/01/19 17:47:11 INFO mapred.LocalJobRunner: Waiting for map tasks
  196. 14/01/19 17:47:11 INFO mapred.LocalJobRunner: Starting task: attempt_local602088658_0004_m_000000_0
  197. 14/01/19 17:47:11 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@15115676
  198. 14/01/19 17:47:11 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/partial-vectors-0/part-r-00000:0+17642777
  199. 14/01/19 17:47:11 INFO mapred.MapTask: io.sort.mb = 100
  200. 14/01/19 17:47:11 INFO mapred.MapTask: data buffer = 79691776/99614720
  201. 14/01/19 17:47:11 INFO mapred.MapTask: record buffer = 262144/327680
  202. 14/01/19 17:47:11 INFO mapred.MapTask: Starting flush of map output
  203. 14/01/19 17:47:11 INFO mapred.MapTask: Finished spill 0
  204. 14/01/19 17:47:11 INFO mapred.Task: Task:attempt_local602088658_0004_m_000000_0 is done. And is in the process of commiting
  205. 14/01/19 17:47:11 INFO mapred.LocalJobRunner:
  206. 14/01/19 17:47:11 INFO mapred.Task: Task 'attempt_local602088658_0004_m_000000_0' done.
  207. 14/01/19 17:47:11 INFO mapred.LocalJobRunner: Finishing task: attempt_local602088658_0004_m_000000_0
  208. 14/01/19 17:47:11 INFO mapred.LocalJobRunner: Map task executor complete.
  209. 14/01/19 17:47:11 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@696c38ad
  210. 14/01/19 17:47:11 INFO mapred.LocalJobRunner:
  211. 14/01/19 17:47:11 INFO mapred.Merger: Merging 1 sorted segments
  212. 14/01/19 17:47:11 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 17420434 bytes
  213. 14/01/19 17:47:11 INFO mapred.LocalJobRunner:
  214. 14/01/19 17:47:12 INFO mapred.Task: Task:attempt_local602088658_0004_r_000000_0 is done. And is in the process of commiting
  215. 14/01/19 17:47:12 INFO mapred.LocalJobRunner:
  216. 14/01/19 17:47:12 INFO mapred.Task: Task attempt_local602088658_0004_r_000000_0 is allowed to commit now
  217. 14/01/19 17:47:12 INFO output.FileOutputCommitter: Saved output of task 'attempt_local602088658_0004_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-toprune
  218. 14/01/19 17:47:12 INFO mapred.LocalJobRunner: reduce > reduce
  219. 14/01/19 17:47:12 INFO mapred.Task: Task 'attempt_local602088658_0004_r_000000_0' done.
  220. 14/01/19 17:47:12 INFO mapred.JobClient: map 100% reduce 100%
  221. 14/01/19 17:47:12 INFO mapred.JobClient: Job complete: job_local602088658_0004
  222. 14/01/19 17:47:12 INFO mapred.JobClient: Counters: 20
  223. 14/01/19 17:47:12 INFO mapred.JobClient: File Output Format Counters
  224. 14/01/19 17:47:12 INFO mapred.JobClient: Bytes Written=17780621
  225. 14/01/19 17:47:12 INFO mapred.JobClient: File Input Format Counters
  226. 14/01/19 17:47:12 INFO mapred.JobClient: Bytes Read=17780621
  227. 14/01/19 17:47:12 INFO mapred.JobClient: FileSystemCounters
  228. 14/01/19 17:47:12 INFO mapred.JobClient: FILE_BYTES_READ=387084660
  229. 14/01/19 17:47:12 INFO mapred.JobClient: FILE_BYTES_WRITTEN=356732171
  230. 14/01/19 17:47:12 INFO mapred.JobClient: Map-Reduce Framework
  231. 14/01/19 17:47:12 INFO mapred.JobClient: Reduce input groups=21578
  232. 14/01/19 17:47:12 INFO mapred.JobClient: Map output materialized bytes=17420438
  233. 14/01/19 17:47:12 INFO mapred.JobClient: Combine output records=0
  234. 14/01/19 17:47:12 INFO mapred.JobClient: Map input records=21578
  235. 14/01/19 17:47:12 INFO mapred.JobClient: Reduce shuffle bytes=0
  236. 14/01/19 17:47:12 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  237. 14/01/19 17:47:12 INFO mapred.JobClient: Reduce output records=21578
  238. 14/01/19 17:47:12 INFO mapred.JobClient: Spilled Records=43156
  239. 14/01/19 17:47:12 INFO mapred.JobClient: Map output bytes=17337483
  240. 14/01/19 17:47:12 INFO mapred.JobClient: Total committed heap usage (bytes)=1257766912
  241. 14/01/19 17:47:12 INFO mapred.JobClient: CPU time spent (ms)=0
  242. 14/01/19 17:47:12 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  243. 14/01/19 17:47:12 INFO mapred.JobClient: SPLIT_RAW_BYTES=165
  244. 14/01/19 17:47:12 INFO mapred.JobClient: Map output records=21578
  245. 14/01/19 17:47:12 INFO mapred.JobClient: Combine input records=0
  246. 14/01/19 17:47:12 INFO mapred.JobClient: Reduce input records=21578
  247. 14/01/19 17:47:12 INFO common.HadoopUtil: Deleting /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/partial-vectors-0
  248. 14/01/19 17:47:12 INFO vectorizer.SparseVectorsFromSequenceFiles: Calculating IDF
  249. 14/01/19 17:47:12 INFO input.FileInputFormat: Total input paths to process : 1
  250. 14/01/19 17:47:12 INFO mapred.JobClient: Running job: job_local333051041_0005
  251. 14/01/19 17:47:12 INFO mapred.LocalJobRunner: Waiting for map tasks
  252. 14/01/19 17:47:12 INFO mapred.LocalJobRunner: Starting task: attempt_local333051041_0005_m_000000_0
  253. 14/01/19 17:47:12 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30c6115f
  254. 14/01/19 17:47:12 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-toprune/part-r-00000:0+17642777
  255. 14/01/19 17:47:12 INFO mapred.MapTask: io.sort.mb = 100
  256. 14/01/19 17:47:12 INFO mapred.MapTask: data buffer = 79691776/99614720
  257. 14/01/19 17:47:12 INFO mapred.MapTask: record buffer = 262144/327680
  258. 14/01/19 17:47:12 INFO mapred.MapTask: Spilling map output: record full = true
  259. 14/01/19 17:47:12 INFO mapred.MapTask: bufstart = 0; bufend = 3145728; bufvoid = 99614720
  260. 14/01/19 17:47:12 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
  261. 14/01/19 17:47:12 INFO mapred.MapTask: Finished spill 0
  262. 14/01/19 17:47:12 INFO mapred.MapTask: Spilling map output: record full = true
  263. 14/01/19 17:47:12 INFO mapred.MapTask: bufstart = 3145728; bufend = 6291444; bufvoid = 99614720
  264. 14/01/19 17:47:12 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
  265. 14/01/19 17:47:12 INFO mapred.MapTask: Finished spill 1
  266. 14/01/19 17:47:12 INFO mapred.MapTask: Spilling map output: record full = true
  267. 14/01/19 17:47:12 INFO mapred.MapTask: bufstart = 6291444; bufend = 9437160; bufvoid = 99614720
  268. 14/01/19 17:47:12 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
  269. 14/01/19 17:47:12 INFO mapred.MapTask: Finished spill 2
  270. 14/01/19 17:47:13 INFO mapred.MapTask: Spilling map output: record full = true
  271. 14/01/19 17:47:13 INFO mapred.MapTask: bufstart = 9437160; bufend = 12582876; bufvoid = 99614720
  272. 14/01/19 17:47:13 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
  273. 14/01/19 17:47:13 INFO mapred.MapTask: Finished spill 3
  274. 14/01/19 17:47:13 INFO mapred.JobClient: map 0% reduce 0%
  275. 14/01/19 17:47:13 INFO mapred.MapTask: Spilling map output: record full = true
  276. 14/01/19 17:47:13 INFO mapred.MapTask: bufstart = 12582876; bufend = 15728604; bufvoid = 99614720
  277. 14/01/19 17:47:13 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
  278. 14/01/19 17:47:13 INFO mapred.MapTask: Finished spill 4
  279. 14/01/19 17:47:13 INFO mapred.MapTask: Starting flush of map output
  280. 14/01/19 17:47:13 INFO mapred.MapTask: Finished spill 5
  281. 14/01/19 17:47:13 INFO mapred.Merger: Merging 6 sorted segments
  282. 14/01/19 17:47:13 INFO mapred.Merger: Down to the last merge-pass, with 6 segments left of total size: 1871798 bytes
  283. 14/01/19 17:47:13 INFO mapred.Task: Task:attempt_local333051041_0005_m_000000_0 is done. And is in the process of commiting
  284. 14/01/19 17:47:13 INFO mapred.LocalJobRunner:
  285. 14/01/19 17:47:13 INFO mapred.Task: Task 'attempt_local333051041_0005_m_000000_0' done.
  286. 14/01/19 17:47:13 INFO mapred.LocalJobRunner: Finishing task: attempt_local333051041_0005_m_000000_0
  287. 14/01/19 17:47:13 INFO mapred.LocalJobRunner: Map task executor complete.
  288. 14/01/19 17:47:13 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4dbe38f5
  289. 14/01/19 17:47:13 INFO mapred.LocalJobRunner:
  290. 14/01/19 17:47:13 INFO mapred.Merger: Merging 1 sorted segments
  291. 14/01/19 17:47:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 585314 bytes
  292. 14/01/19 17:47:13 INFO mapred.LocalJobRunner:
  293. 14/01/19 17:47:13 INFO mapred.Task: Task:attempt_local333051041_0005_r_000000_0 is done. And is in the process of commiting
  294. 14/01/19 17:47:13 INFO mapred.LocalJobRunner:
  295. 14/01/19 17:47:13 INFO mapred.Task: Task attempt_local333051041_0005_r_000000_0 is allowed to commit now
  296. 14/01/19 17:47:13 INFO output.FileOutputCommitter: Saved output of task 'attempt_local333051041_0005_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/df-count
  297. 14/01/19 17:47:13 INFO mapred.LocalJobRunner: reduce > reduce
  298. 14/01/19 17:47:13 INFO mapred.Task: Task 'attempt_local333051041_0005_r_000000_0' done.
  299. 14/01/19 17:47:14 INFO mapred.JobClient: map 100% reduce 100%
  300. 14/01/19 17:47:14 INFO mapred.JobClient: Job complete: job_local333051041_0005
  301. 14/01/19 17:47:14 INFO mapred.JobClient: Counters: 20
  302. 14/01/19 17:47:14 INFO mapred.JobClient: File Output Format Counters
  303. 14/01/19 17:47:14 INFO mapred.JobClient: Bytes Written=851221
  304. 14/01/19 17:47:14 INFO mapred.JobClient: File Input Format Counters
  305. 14/01/19 17:47:14 INFO mapred.JobClient: Bytes Read=17780621
  306. 14/01/19 17:47:14 INFO mapred.JobClient: FileSystemCounters
  307. 14/01/19 17:47:14 INFO mapred.JobClient: FILE_BYTES_READ=492511852
  308. 14/01/19 17:47:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=428876229
  309. 14/01/19 17:47:14 INFO mapred.JobClient: Map-Reduce Framework
  310. 14/01/19 17:47:14 INFO mapred.JobClient: Reduce input groups=41808
  311. 14/01/19 17:47:14 INFO mapred.JobClient: Map output materialized bytes=585318
  312. 14/01/19 17:47:14 INFO mapred.JobClient: Combine output records=175507
  313. 14/01/19 17:47:14 INFO mapred.JobClient: Map input records=21578
  314. 14/01/19 17:47:14 INFO mapred.JobClient: Reduce shuffle bytes=0
  315. 14/01/19 17:47:14 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  316. 14/01/19 17:47:14 INFO mapred.JobClient: Reduce output records=41808
  317. 14/01/19 17:47:14 INFO mapred.JobClient: Spilled Records=217315
  318. 14/01/19 17:47:14 INFO mapred.JobClient: Map output bytes=18371184
  319. 14/01/19 17:47:14 INFO mapred.JobClient: Total committed heap usage (bytes)=1162870784
  320. 14/01/19 17:47:14 INFO mapred.JobClient: CPU time spent (ms)=0
  321. 14/01/19 17:47:14 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  322. 14/01/19 17:47:14 INFO mapred.JobClient: SPLIT_RAW_BYTES=166
  323. 14/01/19 17:47:14 INFO mapred.JobClient: Map output records=1530932
  324. 14/01/19 17:47:14 INFO mapred.JobClient: Combine input records=1664631
  325. 14/01/19 17:47:14 INFO mapred.JobClient: Reduce input records=41808
  326. 14/01/19 17:47:14 INFO vectorizer.SparseVectorsFromSequenceFiles: Pruning
  327. 14/01/19 17:47:14 INFO input.FileInputFormat: Total input paths to process : 1
  328. 14/01/19 17:47:14 INFO filecache.TrackerDistributedCacheManager: Creating frequency.file-0 in /tmp/hadoop-frank/mapred/local/archive/7819576480302196673_-1609338674_728114391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans-work--7044987863602312150 with rwxr-xr-x
  329. 14/01/19 17:47:14 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0 as /tmp/hadoop-frank/mapred/local/archive/7819576480302196673_-1609338674_728114391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0
  330. 14/01/19 17:47:14 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0 as /tmp/hadoop-frank/mapred/local/archive/7819576480302196673_-1609338674_728114391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0
  331. 14/01/19 17:47:14 INFO mapred.JobClient: Running job: job_local1502901614_0006
  332. 14/01/19 17:47:14 INFO mapred.LocalJobRunner: Waiting for map tasks
  333. 14/01/19 17:47:14 INFO mapred.LocalJobRunner: Starting task: attempt_local1502901614_0006_m_000000_0
  334. 14/01/19 17:47:14 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@51be8406
  335. 14/01/19 17:47:14 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-toprune/part-r-00000:0+17642777
  336. 14/01/19 17:47:14 INFO mapred.MapTask: io.sort.mb = 100
  337. 14/01/19 17:47:14 INFO mapred.MapTask: data buffer = 79691776/99614720
  338. 14/01/19 17:47:14 INFO mapred.MapTask: record buffer = 262144/327680
  339. 14/01/19 17:47:14 INFO mapred.MapTask: Starting flush of map output
  340. 14/01/19 17:47:14 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  341. 14/01/19 17:47:14 INFO compress.CodecPool: Got brand-new compressor
  342. 14/01/19 17:47:15 INFO mapred.MapTask: Finished spill 0
  343. 14/01/19 17:47:15 INFO mapred.Task: Task:attempt_local1502901614_0006_m_000000_0 is done. And is in the process of commiting
  344. 14/01/19 17:47:15 INFO mapred.LocalJobRunner:
  345. 14/01/19 17:47:15 INFO mapred.Task: Task 'attempt_local1502901614_0006_m_000000_0' done.
  346. 14/01/19 17:47:15 INFO mapred.LocalJobRunner: Finishing task: attempt_local1502901614_0006_m_000000_0
  347. 14/01/19 17:47:15 INFO mapred.LocalJobRunner: Map task executor complete.
  348. 14/01/19 17:47:15 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30c4f4d8
  349. 14/01/19 17:47:15 INFO mapred.LocalJobRunner:
  350. 14/01/19 17:47:15 INFO mapred.Merger: Merging 1 sorted segments
  351. 14/01/19 17:47:15 INFO compress.CodecPool: Got brand-new decompressor
  352. 14/01/19 17:47:15 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4441445 bytes
  353. 14/01/19 17:47:15 INFO mapred.LocalJobRunner:
  354. 14/01/19 17:47:15 INFO mapred.JobClient: map 100% reduce 0%
  355. 14/01/19 17:47:15 INFO mapred.Task: Task:attempt_local1502901614_0006_r_000000_0 is done. And is in the process of commiting
  356. 14/01/19 17:47:15 INFO mapred.LocalJobRunner:
  357. 14/01/19 17:47:15 INFO mapred.Task: Task attempt_local1502901614_0006_r_000000_0 is allowed to commit now
  358. 14/01/19 17:47:15 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1502901614_0006_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-partial/partial-0
  359. 14/01/19 17:47:15 INFO mapred.LocalJobRunner: reduce > reduce
  360. 14/01/19 17:47:15 INFO mapred.Task: Task 'attempt_local1502901614_0006_r_000000_0' done.
  361. 14/01/19 17:47:16 INFO mapred.JobClient: map 100% reduce 100%
  362. 14/01/19 17:47:16 INFO mapred.JobClient: Job complete: job_local1502901614_0006
  363. 14/01/19 17:47:16 INFO mapred.JobClient: Counters: 20
  364. 14/01/19 17:47:16 INFO mapred.JobClient: File Output Format Counters
  365. 14/01/19 17:47:16 INFO mapred.JobClient: Bytes Written=17156391
  366. 14/01/19 17:47:16 INFO mapred.JobClient: File Input Format Counters
  367. 14/01/19 17:47:16 INFO mapred.JobClient: Bytes Read=17780621
  368. 14/01/19 17:47:16 INFO mapred.JobClient: FileSystemCounters
  369. 14/01/19 17:47:16 INFO mapred.JobClient: FILE_BYTES_READ=585472456
  370. 14/01/19 17:47:16 INFO mapred.JobClient: FILE_BYTES_WRITTEN=507772373
  371. 14/01/19 17:47:16 INFO mapred.JobClient: Map-Reduce Framework
  372. 14/01/19 17:47:16 INFO mapred.JobClient: Reduce input groups=21578
  373. 14/01/19 17:47:16 INFO mapred.JobClient: Map output materialized bytes=4441449
  374. 14/01/19 17:47:16 INFO mapred.JobClient: Combine output records=0
  375. 14/01/19 17:47:16 INFO mapred.JobClient: Map input records=21578
  376. 14/01/19 17:47:16 INFO mapred.JobClient: Reduce shuffle bytes=0
  377. 14/01/19 17:47:16 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  378. 14/01/19 17:47:16 INFO mapred.JobClient: Reduce output records=21578
  379. 14/01/19 17:47:16 INFO mapred.JobClient: Spilled Records=43156
  380. 14/01/19 17:47:16 INFO mapred.JobClient: Map output bytes=17337483
  381. 14/01/19 17:47:16 INFO mapred.JobClient: Total committed heap usage (bytes)=1163526144
  382. 14/01/19 17:47:16 INFO mapred.JobClient: CPU time spent (ms)=0
  383. 14/01/19 17:47:16 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  384. 14/01/19 17:47:16 INFO mapred.JobClient: SPLIT_RAW_BYTES=166
  385. 14/01/19 17:47:16 INFO mapred.JobClient: Map output records=21578
  386. 14/01/19 17:47:16 INFO mapred.JobClient: Combine input records=0
  387. 14/01/19 17:47:16 INFO mapred.JobClient: Reduce input records=21578
  388. 14/01/19 17:47:16 INFO input.FileInputFormat: Total input paths to process : 1
  389. 14/01/19 17:47:16 INFO mapred.JobClient: Running job: job_local1946861464_0007
  390. 14/01/19 17:47:16 INFO mapred.LocalJobRunner: Waiting for map tasks
  391. 14/01/19 17:47:16 INFO mapred.LocalJobRunner: Starting task: attempt_local1946861464_0007_m_000000_0
  392. 14/01/19 17:47:16 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@39c9c99a
  393. 14/01/19 17:47:16 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-partial/partial-0/part-r-00000:0+17023387
  394. 14/01/19 17:47:16 INFO mapred.MapTask: io.sort.mb = 100
  395. 14/01/19 17:47:16 INFO mapred.MapTask: data buffer = 79691776/99614720
  396. 14/01/19 17:47:16 INFO mapred.MapTask: record buffer = 262144/327680
  397. 14/01/19 17:47:16 INFO mapred.MapTask: Starting flush of map output
  398. 14/01/19 17:47:16 INFO mapred.MapTask: Finished spill 0
  399. 14/01/19 17:47:16 INFO mapred.Task: Task:attempt_local1946861464_0007_m_000000_0 is done. And is in the process of commiting
  400. 14/01/19 17:47:16 INFO mapred.LocalJobRunner:
  401. 14/01/19 17:47:16 INFO mapred.Task: Task 'attempt_local1946861464_0007_m_000000_0' done.
  402. 14/01/19 17:47:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local1946861464_0007_m_000000_0
  403. 14/01/19 17:47:16 INFO mapred.LocalJobRunner: Map task executor complete.
  404. 14/01/19 17:47:16 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7b7e3427
  405. 14/01/19 17:47:16 INFO mapred.LocalJobRunner:
  406. 14/01/19 17:47:16 INFO mapred.Merger: Merging 1 sorted segments
  407. 14/01/19 17:47:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 16804860 bytes
  408. 14/01/19 17:47:16 INFO mapred.LocalJobRunner:
  409. 14/01/19 17:47:17 INFO mapred.Task: Task:attempt_local1946861464_0007_r_000000_0 is done. And is in the process of commiting
  410. 14/01/19 17:47:17 INFO mapred.LocalJobRunner:
  411. 14/01/19 17:47:17 INFO mapred.Task: Task attempt_local1946861464_0007_r_000000_0 is allowed to commit now
  412. 14/01/19 17:47:17 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1946861464_0007_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors
  413. 14/01/19 17:47:17 INFO mapred.LocalJobRunner: reduce > reduce
  414. 14/01/19 17:47:17 INFO mapred.Task: Task 'attempt_local1946861464_0007_r_000000_0' done.
  415. 14/01/19 17:47:17 INFO mapred.JobClient: map 100% reduce 100%
  416. 14/01/19 17:47:17 INFO mapred.JobClient: Job complete: job_local1946861464_0007
  417. 14/01/19 17:47:17 INFO mapred.JobClient: Counters: 20
  418. 14/01/19 17:47:17 INFO mapred.JobClient: File Output Format Counters
  419. 14/01/19 17:47:17 INFO mapred.JobClient: Bytes Written=17156391
  420. 14/01/19 17:47:17 INFO mapred.JobClient: File Input Format Counters
  421. 14/01/19 17:47:17 INFO mapred.JobClient: Bytes Read=17156391
  422. 14/01/19 17:47:17 INFO mapred.JobClient: FileSystemCounters
  423. 14/01/19 17:47:17 INFO mapred.JobClient: FILE_BYTES_READ=689999322
  424. 14/01/19 17:47:17 INFO mapred.JobClient: FILE_BYTES_WRITTEN=624292583
  425. 14/01/19 17:47:17 INFO mapred.JobClient: Map-Reduce Framework
  426. 14/01/19 17:47:17 INFO mapred.JobClient: Reduce input groups=21578
  427. 14/01/19 17:47:17 INFO mapred.JobClient: Map output materialized bytes=16804864
  428. 14/01/19 17:47:17 INFO mapred.JobClient: Combine output records=0
  429. 14/01/19 17:47:17 INFO mapred.JobClient: Map input records=21578
  430. 14/01/19 17:47:17 INFO mapred.JobClient: Reduce shuffle bytes=0
  431. 14/01/19 17:47:17 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  432. 14/01/19 17:47:17 INFO mapred.JobClient: Reduce output records=21578
  433. 14/01/19 17:47:17 INFO mapred.JobClient: Spilled Records=43156
  434. 14/01/19 17:47:17 INFO mapred.JobClient: Map output bytes=16722273
  435. 14/01/19 17:47:17 INFO mapred.JobClient: Total committed heap usage (bytes)=1166344192
  436. 14/01/19 17:47:17 INFO mapred.JobClient: CPU time spent (ms)=0
  437. 14/01/19 17:47:17 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  438. 14/01/19 17:47:17 INFO mapred.JobClient: SPLIT_RAW_BYTES=176
  439. 14/01/19 17:47:17 INFO mapred.JobClient: Map output records=21578
  440. 14/01/19 17:47:17 INFO mapred.JobClient: Combine input records=0
  441. 14/01/19 17:47:17 INFO mapred.JobClient: Reduce input records=21578
  442. 14/01/19 17:47:17 INFO common.HadoopUtil: Deleting /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-partial
  443. 14/01/19 17:47:17 INFO common.HadoopUtil: Deleting /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors-toprune
  444. 14/01/19 17:47:17 INFO input.FileInputFormat: Total input paths to process : 1
  445. 14/01/19 17:47:17 INFO filecache.TrackerDistributedCacheManager: Creating frequency.file-0 in /tmp/hadoop-frank/mapred/local/archive/-4994632083952568315_-1609338674_728114391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans-work--2462720769925472264 with rwxr-xr-x
  446. 14/01/19 17:47:17 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0 as /tmp/hadoop-frank/mapred/local/archive/-4994632083952568315_-1609338674_728114391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0
  447. 14/01/19 17:47:17 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0 as /tmp/hadoop-frank/mapred/local/archive/-4994632083952568315_-1609338674_728114391/file/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/frequency.file-0
  448. 14/01/19 17:47:17 INFO mapred.JobClient: Running job: job_local1646191673_0008
  449. 14/01/19 17:47:17 INFO mapred.LocalJobRunner: Waiting for map tasks
  450. 14/01/19 17:47:17 INFO mapred.LocalJobRunner: Starting task: attempt_local1646191673_0008_m_000000_0
  451. 14/01/19 17:47:17 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@28b0dc0b
  452. 14/01/19 17:47:17 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tf-vectors/part-r-00000:0+17023387
  453. 14/01/19 17:47:17 INFO mapred.MapTask: io.sort.mb = 100
  454. 14/01/19 17:47:17 INFO mapred.MapTask: data buffer = 79691776/99614720
  455. 14/01/19 17:47:17 INFO mapred.MapTask: record buffer = 262144/327680
  456. 14/01/19 17:47:17 INFO mapred.MapTask: Starting flush of map output
  457. 14/01/19 17:47:17 INFO mapred.MapTask: Finished spill 0
  458. 14/01/19 17:47:17 INFO mapred.Task: Task:attempt_local1646191673_0008_m_000000_0 is done. And is in the process of commiting
  459. 14/01/19 17:47:17 INFO mapred.LocalJobRunner:
  460. 14/01/19 17:47:17 INFO mapred.Task: Task 'attempt_local1646191673_0008_m_000000_0' done.
  461. 14/01/19 17:47:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local1646191673_0008_m_000000_0
  462. 14/01/19 17:47:17 INFO mapred.LocalJobRunner: Map task executor complete.
  463. 14/01/19 17:47:17 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7b4cdeb0
  464. 14/01/19 17:47:17 INFO mapred.LocalJobRunner:
  465. 14/01/19 17:47:17 INFO mapred.Merger: Merging 1 sorted segments
  466. 14/01/19 17:47:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 16804860 bytes
  467. 14/01/19 17:47:17 INFO mapred.LocalJobRunner:
  468. 14/01/19 17:47:18 INFO mapred.Task: Task:attempt_local1646191673_0008_r_000000_0 is done. And is in the process of commiting
  469. 14/01/19 17:47:18 INFO mapred.LocalJobRunner:
  470. 14/01/19 17:47:18 INFO mapred.Task: Task attempt_local1646191673_0008_r_000000_0 is allowed to commit now
  471. 14/01/19 17:47:18 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1646191673_0008_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/partial-vectors-0
  472. 14/01/19 17:47:18 INFO mapred.LocalJobRunner: reduce > reduce
  473. 14/01/19 17:47:18 INFO mapred.Task: Task 'attempt_local1646191673_0008_r_000000_0' done.
  474. 14/01/19 17:47:18 INFO mapred.JobClient: map 100% reduce 100%
  475. 14/01/19 17:47:18 INFO mapred.JobClient: Job complete: job_local1646191673_0008
  476. 14/01/19 17:47:18 INFO mapred.JobClient: Counters: 20
  477. 14/01/19 17:47:18 INFO mapred.JobClient: File Output Format Counters
  478. 14/01/19 17:47:18 INFO mapred.JobClient: Bytes Written=17156391
  479. 14/01/19 17:47:18 INFO mapred.JobClient: File Input Format Counters
  480. 14/01/19 17:47:18 INFO mapred.JobClient: Bytes Read=17156391
  481. 14/01/19 17:47:18 INFO mapred.JobClient: FileSystemCounters
  482. 14/01/19 17:47:18 INFO mapred.JobClient: FILE_BYTES_READ=808591969
  483. 14/01/19 17:47:18 INFO mapred.JobClient: FILE_BYTES_WRITTEN=742519851
  484. 14/01/19 17:47:18 INFO mapred.JobClient: Map-Reduce Framework
  485. 14/01/19 17:47:18 INFO mapred.JobClient: Reduce input groups=21578
  486. 14/01/19 17:47:18 INFO mapred.JobClient: Map output materialized bytes=16804864
  487. 14/01/19 17:47:18 INFO mapred.JobClient: Combine output records=0
  488. 14/01/19 17:47:18 INFO mapred.JobClient: Map input records=21578
  489. 14/01/19 17:47:18 INFO mapred.JobClient: Reduce shuffle bytes=0
  490. 14/01/19 17:47:18 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  491. 14/01/19 17:47:18 INFO mapred.JobClient: Reduce output records=21578
  492. 14/01/19 17:47:18 INFO mapred.JobClient: Spilled Records=43156
  493. 14/01/19 17:47:18 INFO mapred.JobClient: Map output bytes=16722273
  494. 14/01/19 17:47:18 INFO mapred.JobClient: Total committed heap usage (bytes)=1168769024
  495. 14/01/19 17:47:18 INFO mapred.JobClient: CPU time spent (ms)=0
  496. 14/01/19 17:47:18 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  497. 14/01/19 17:47:18 INFO mapred.JobClient: SPLIT_RAW_BYTES=158
  498. 14/01/19 17:47:18 INFO mapred.JobClient: Map output records=21578
  499. 14/01/19 17:47:18 INFO mapred.JobClient: Combine input records=0
  500. 14/01/19 17:47:18 INFO mapred.JobClient: Reduce input records=21578
  501. 14/01/19 17:47:18 INFO input.FileInputFormat: Total input paths to process : 1
  502. 14/01/19 17:47:18 INFO mapred.JobClient: Running job: job_local1705633954_0009
  503. 14/01/19 17:47:18 INFO mapred.LocalJobRunner: Waiting for map tasks
  504. 14/01/19 17:47:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1705633954_0009_m_000000_0
  505. 14/01/19 17:47:18 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7a409fa2
  506. 14/01/19 17:47:18 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/partial-vectors-0/part-r-00000:0+17023387
  507. 14/01/19 17:47:18 INFO mapred.MapTask: io.sort.mb = 100
  508. 14/01/19 17:47:18 INFO mapred.MapTask: data buffer = 79691776/99614720
  509. 14/01/19 17:47:18 INFO mapred.MapTask: record buffer = 262144/327680
  510. 14/01/19 17:47:19 INFO mapred.MapTask: Starting flush of map output
  511. 14/01/19 17:47:19 INFO mapred.MapTask: Finished spill 0
  512. 14/01/19 17:47:19 INFO mapred.Task: Task:attempt_local1705633954_0009_m_000000_0 is done. And is in the process of commiting
  513. 14/01/19 17:47:19 INFO mapred.LocalJobRunner:
  514. 14/01/19 17:47:19 INFO mapred.Task: Task 'attempt_local1705633954_0009_m_000000_0' done.
  515. 14/01/19 17:47:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local1705633954_0009_m_000000_0
  516. 14/01/19 17:47:19 INFO mapred.LocalJobRunner: Map task executor complete.
  517. 14/01/19 17:47:19 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1a2e808a
  518. 14/01/19 17:47:19 INFO mapred.LocalJobRunner:
  519. 14/01/19 17:47:19 INFO mapred.Merger: Merging 1 sorted segments
  520. 14/01/19 17:47:19 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 16804860 bytes
  521. 14/01/19 17:47:19 INFO mapred.LocalJobRunner:
  522. 14/01/19 17:47:19 INFO mapred.Task: Task:attempt_local1705633954_0009_r_000000_0 is done. And is in the process of commiting
  523. 14/01/19 17:47:19 INFO mapred.LocalJobRunner:
  524. 14/01/19 17:47:19 INFO mapred.Task: Task attempt_local1705633954_0009_r_000000_0 is allowed to commit now
  525. 14/01/19 17:47:19 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1705633954_0009_r_000000_0' to /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tfidf-vectors
  526. 14/01/19 17:47:19 INFO mapred.LocalJobRunner: reduce > reduce
  527. 14/01/19 17:47:19 INFO mapred.Task: Task 'attempt_local1705633954_0009_r_000000_0' done.
  528. 14/01/19 17:47:19 INFO mapred.JobClient: map 100% reduce 100%
  529. 14/01/19 17:47:19 INFO mapred.JobClient: Job complete: job_local1705633954_0009
  530. 14/01/19 17:47:19 INFO mapred.JobClient: Counters: 20
  531. 14/01/19 17:47:19 INFO mapred.JobClient: File Output Format Counters
  532. 14/01/19 17:47:19 INFO mapred.JobClient: Bytes Written=17156391
  533. 14/01/19 17:47:19 INFO mapred.JobClient: File Input Format Counters
  534. 14/01/19 17:47:19 INFO mapred.JobClient: Bytes Read=17156391
  535. 14/01/19 17:47:19 INFO mapred.JobClient: FileSystemCounters
  536. 14/01/19 17:47:19 INFO mapred.JobClient: FILE_BYTES_READ=925482228
  537. 14/01/19 17:47:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=859041139
  538. 14/01/19 17:47:19 INFO mapred.JobClient: Map-Reduce Framework
  539. 14/01/19 17:47:19 INFO mapred.JobClient: Reduce input groups=21578
  540. 14/01/19 17:47:19 INFO mapred.JobClient: Map output materialized bytes=16804864
  541. 14/01/19 17:47:19 INFO mapred.JobClient: Combine output records=0
  542. 14/01/19 17:47:19 INFO mapred.JobClient: Map input records=21578
  543. 14/01/19 17:47:19 INFO mapred.JobClient: Reduce shuffle bytes=0
  544. 14/01/19 17:47:19 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  545. 14/01/19 17:47:19 INFO mapred.JobClient: Reduce output records=21578
  546. 14/01/19 17:47:19 INFO mapred.JobClient: Spilled Records=43156
  547. 14/01/19 17:47:19 INFO mapred.JobClient: Map output bytes=16722273
  548. 14/01/19 17:47:19 INFO mapred.JobClient: Total committed heap usage (bytes)=1226571776
  549. 14/01/19 17:47:19 INFO mapred.JobClient: CPU time spent (ms)=0
  550. 14/01/19 17:47:19 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  551. 14/01/19 17:47:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=165
  552. 14/01/19 17:47:19 INFO mapred.JobClient: Map output records=21578
  553. 14/01/19 17:47:19 INFO mapred.JobClient: Combine input records=0
  554. 14/01/19 17:47:19 INFO mapred.JobClient: Reduce input records=21578
  555. 14/01/19 17:47:19 INFO common.HadoopUtil: Deleting /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/partial-vectors-0
  556. 14/01/19 17:47:19 INFO driver.MahoutDriver: Program took 16376 ms (Minutes: 0.2729333333333333)
  557. MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
  558. Warning: $HADOOP_HOME is deprecated.
  559.  
  560. Running on hadoop, using /opt/hadoop/bin/hadoop and HADOOP_CONF_DIR=$HADOOP_HOME/conf
  561. MAHOUT-JOB: /home/frank/Desktop/mahout-distribution-0.9/examples/target/mahout-examples-0.9-job.jar
  562. Warning: $HADOOP_HOME is deprecated.
  563.  
  564. 14/01/19 17:47:21 WARN driver.MahoutDriver: No streamingkmeans.props found on classpath, will use command-line arguments only
  565. 14/01/19 17:47:21 INFO common.AbstractJob: Command line arguments: {--distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --estimatedDistanceCutoff=[-1.0], --estimatedNumMapClusters=[100], --input=[/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tfidf-vectors/], --maxNumIterations=[10], --method=[mapreduce], --numBallKMeansRuns=[4], --numClusters=[10], --numProjections=[3], --output=[/tmp/mahout-work-frank/reuters-streamingkmeans], --overwrite=null, --reduceStreamingKMeans=null, --searchSize=[2], --searcherClass=[org.apache.mahout.math.neighborhood.FastProjectionSearch], --startPhase=[0], --tempDir=[/tmp/mahout-work-frank/tmp], --testProbability=[0.1], --trimFraction=[0.9]}
  566. 14/01/19 17:47:21 INFO mapreduce.StreamingKMeansDriver: Starting to configure options for workers
  567. 14/01/19 17:47:21 INFO mapreduce.StreamingKMeansDriver: Parameters are: [k] numClusters 10; [SKM] estimatedNumMapClusters 100; estimatedDistanceCutoff -1.0 [BKM] maxNumIterations 10; trimFraction 0.9; randomInit false; ignoreWeights false; testProbability 0.1; numBallKMeansRuns 4; [S] measureClass org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure; searcherClass org.apache.mahout.math.neighborhood.FastProjectionSearch; searcherSize 2; numProjections 3; method mapreduce; reduceStreamingKMeans true
  568. 14/01/19 17:47:21 INFO mapreduce.StreamingKMeansDriver: Starting StreamingKMeans clustering for vectors in /tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tfidf-vectors; results are output to /tmp/mahout-work-frank/reuters-streamingkmeans
  569. 14/01/19 17:47:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  570. 14/01/19 17:47:21 INFO input.FileInputFormat: Total input paths to process : 1
  571. 14/01/19 17:47:21 INFO mapred.JobClient: Running job: job_local599981549_0001
  572. 14/01/19 17:47:21 INFO mapred.LocalJobRunner: Waiting for map tasks
  573. 14/01/19 17:47:21 INFO mapred.LocalJobRunner: Starting task: attempt_local599981549_0001_m_000000_0
  574. 14/01/19 17:47:21 INFO util.ProcessTree: setsid exited with exit code 0
  575. 14/01/19 17:47:21 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@292eeec3
  576. 14/01/19 17:47:21 INFO mapred.MapTask: Processing split: file:/tmp/mahout-work-frank/reuters-out-seqdir-sparse-streamingkmeans/tfidf-vectors/part-r-00000:0+17023387
  577. 14/01/19 17:47:21 INFO mapred.MapTask: io.sort.mb = 100
  578. 14/01/19 17:47:21 INFO mapred.MapTask: data buffer = 79691776/99614720
  579. 14/01/19 17:47:21 INFO mapred.MapTask: record buffer = 262144/327680
  580. 14/01/19 17:47:22 INFO mapred.JobClient: map 0% reduce 0%
  581. 14/01/19 17:47:27 INFO mapred.LocalJobRunner:
  582. 14/01/19 17:47:28 INFO mapred.JobClient: map 5% reduce 0%
  583. 14/01/19 17:47:30 INFO mapred.LocalJobRunner:
  584. 14/01/19 17:47:31 INFO mapred.JobClient: map 7% reduce 0%
  585. 14/01/19 17:47:33 INFO mapred.LocalJobRunner:
  586. 14/01/19 17:47:34 INFO mapred.JobClient: map 9% reduce 0%
  587. 14/01/19 17:47:36 INFO mapred.LocalJobRunner:
  588. 14/01/19 17:47:37 INFO mapred.JobClient: map 11% reduce 0%
  589. 14/01/19 17:47:39 INFO mapred.LocalJobRunner:
  590. 14/01/19 17:47:40 INFO mapred.JobClient: map 13% reduce 0%
  591. 14/01/19 17:47:42 INFO mapred.LocalJobRunner:
  592. 14/01/19 17:47:43 INFO mapred.JobClient: map 15% reduce 0%
  593. 14/01/19 17:47:45 INFO mapred.LocalJobRunner:
  594. 14/01/19 17:47:46 INFO mapred.JobClient: map 16% reduce 0%
  595. 14/01/19 17:47:48 INFO mapred.LocalJobRunner:
  596. 14/01/19 17:47:49 INFO mapred.JobClient: map 17% reduce 0%
  597. 14/01/19 17:47:51 INFO mapred.LocalJobRunner:
  598. 14/01/19 17:47:52 INFO mapred.JobClient: map 19% reduce 0%
  599. 14/01/19 17:47:54 INFO mapred.LocalJobRunner:
  600. 14/01/19 17:47:55 INFO mapred.JobClient: map 20% reduce 0%
  601. 14/01/19 17:47:57 INFO mapred.LocalJobRunner:
  602. 14/01/19 17:47:58 INFO mapred.JobClient: map 21% reduce 0%
  603. 14/01/19 17:48:00 INFO mapred.LocalJobRunner:
  604. 14/01/19 17:48:01 INFO mapred.JobClient: map 22% reduce 0%
  605. 14/01/19 17:48:03 INFO mapred.LocalJobRunner:
  606. 14/01/19 17:48:04 INFO mapred.JobClient: map 25% reduce 0%
  607. 14/01/19 17:48:06 INFO mapred.LocalJobRunner:
  608. 14/01/19 17:48:07 INFO mapred.JobClient: map 26% reduce 0%
  609. 14/01/19 17:48:09 INFO mapred.LocalJobRunner:
  610. 14/01/19 17:48:10 INFO mapred.JobClient: map 27% reduce 0%
  611. 14/01/19 17:48:12 INFO mapred.LocalJobRunner:
  612. 14/01/19 17:48:13 INFO mapred.JobClient: map 28% reduce 0%
  613. 14/01/19 17:48:15 INFO mapred.LocalJobRunner:
  614. 14/01/19 17:48:16 INFO mapred.JobClient: map 29% reduce 0%
  615. 14/01/19 17:48:18 INFO mapred.LocalJobRunner:
  616. 14/01/19 17:48:19 INFO mapred.JobClient: map 31% reduce 0%
  617. 14/01/19 17:48:21 INFO mapred.LocalJobRunner:
  618. 14/01/19 17:48:22 INFO mapred.JobClient: map 32% reduce 0%
  619. 14/01/19 17:48:24 INFO mapred.LocalJobRunner:
  620. 14/01/19 17:48:25 INFO mapred.JobClient: map 33% reduce 0%
  621. 14/01/19 17:48:27 INFO mapred.LocalJobRunner:
  622. 14/01/19 17:48:28 INFO mapred.JobClient: map 34% reduce 0%
  623. 14/01/19 17:48:30 INFO mapred.LocalJobRunner:
  624. 14/01/19 17:48:31 INFO mapred.JobClient: map 35% reduce 0%
  625. 14/01/19 17:48:33 INFO mapred.LocalJobRunner:
  626. 14/01/19 17:48:34 INFO mapred.JobClient: map 36% reduce 0%
  627. 14/01/19 17:48:36 INFO mapred.LocalJobRunner:
  628. 14/01/19 17:48:37 INFO mapred.JobClient: map 37% reduce 0%
  629. 14/01/19 17:48:39 INFO mapred.LocalJobRunner:
  630. 14/01/19 17:48:40 INFO mapred.JobClient: map 38% reduce 0%
  631. 14/01/19 17:48:42 INFO mapred.LocalJobRunner:
  632. 14/01/19 17:48:43 INFO mapred.JobClient: map 39% reduce 0%
  633. 14/01/19 17:48:45 INFO mapred.LocalJobRunner:
  634. 14/01/19 17:48:46 INFO mapred.JobClient: map 40% reduce 0%
  635. 14/01/19 17:48:48 INFO mapred.LocalJobRunner:
  636. 14/01/19 17:48:49 INFO mapred.JobClient: map 41% reduce 0%
  637. 14/01/19 17:48:51 INFO mapred.LocalJobRunner:
  638. 14/01/19 17:48:52 INFO mapred.JobClient: map 42% reduce 0%
  639. 14/01/19 17:48:54 INFO mapred.LocalJobRunner:
  640. 14/01/19 17:48:55 INFO mapred.JobClient: map 43% reduce 0%
  641. 14/01/19 17:48:57 INFO mapred.LocalJobRunner:
  642. 14/01/19 17:48:58 INFO mapred.JobClient: map 44% reduce 0%
  643. 14/01/19 17:49:00 INFO mapred.LocalJobRunner:
  644. 14/01/19 17:49:03 INFO mapred.LocalJobRunner:
  645. 14/01/19 17:49:04 INFO mapred.JobClient: map 45% reduce 0%
  646. 14/01/19 17:49:06 INFO mapred.LocalJobRunner:
  647. 14/01/19 17:49:07 INFO mapred.JobClient: map 47% reduce 0%
  648. 14/01/19 17:49:09 INFO mapred.LocalJobRunner:
  649. 14/01/19 17:49:12 INFO mapred.LocalJobRunner:
  650. 14/01/19 17:49:13 INFO mapred.JobClient: map 48% reduce 0%
  651. 14/01/19 17:49:15 INFO mapred.LocalJobRunner:
  652. 14/01/19 17:49:16 INFO mapred.JobClient: map 49% reduce 0%
  653. 14/01/19 17:49:18 INFO mapred.LocalJobRunner:
  654. 14/01/19 17:49:19 INFO mapred.JobClient: map 50% reduce 0%
  655. 14/01/19 17:49:21 INFO mapred.LocalJobRunner:
  656. 14/01/19 17:49:24 INFO mapred.LocalJobRunner:
  657. 14/01/19 17:49:25 INFO mapred.JobClient: map 52% reduce 0%
  658. 14/01/19 17:49:27 INFO mapred.LocalJobRunner:
  659. 14/01/19 17:49:30 INFO mapred.LocalJobRunner:
  660. 14/01/19 17:49:31 INFO mapred.JobClient: map 54% reduce 0%
  661. 14/01/19 17:49:33 INFO mapred.LocalJobRunner:
  662. 14/01/19 17:49:36 INFO mapred.LocalJobRunner:
  663. 14/01/19 17:49:37 INFO mapred.JobClient: map 55% reduce 0%
  664. 14/01/19 17:49:39 INFO mapred.LocalJobRunner:
  665. 14/01/19 17:49:40 INFO mapred.JobClient: map 56% reduce 0%
  666. 14/01/19 17:49:42 INFO mapred.LocalJobRunner:
  667. 14/01/19 17:49:45 INFO mapred.LocalJobRunner:
  668. 14/01/19 17:49:46 INFO mapred.JobClient: map 57% reduce 0%
  669. 14/01/19 17:49:48 INFO mapred.LocalJobRunner:
  670. 14/01/19 17:49:51 INFO mapred.LocalJobRunner:
  671. 14/01/19 17:49:52 INFO mapred.JobClient: map 58% reduce 0%
  672. 14/01/19 17:49:54 INFO mapred.LocalJobRunner:
  673. 14/01/19 17:49:55 INFO mapred.JobClient: map 59% reduce 0%
  674. 14/01/19 17:49:57 INFO mapred.LocalJobRunner:
  675. 14/01/19 17:49:58 INFO mapred.JobClient: map 60% reduce 0%
  676. 14/01/19 17:50:00 INFO mapred.LocalJobRunner:
  677. 14/01/19 17:50:03 INFO mapred.LocalJobRunner:
  678. 14/01/19 17:50:04 INFO mapred.JobClient: map 61% reduce 0%
  679. 14/01/19 17:50:06 INFO mapred.LocalJobRunner:
  680. 14/01/19 17:50:07 INFO mapred.JobClient: map 62% reduce 0%
  681. 14/01/19 17:50:09 INFO mapred.LocalJobRunner:
  682. 14/01/19 17:50:12 INFO mapred.LocalJobRunner:
  683. 14/01/19 17:50:13 INFO mapred.JobClient: map 63% reduce 0%
  684. 14/01/19 17:50:15 INFO mapred.LocalJobRunner:
  685. 14/01/19 17:50:18 INFO mapred.LocalJobRunner:
  686. 14/01/19 17:50:19 INFO mapred.JobClient: map 64% reduce 0%
  687. 14/01/19 17:50:21 INFO mapred.LocalJobRunner:
  688. 14/01/19 17:50:24 INFO mapred.LocalJobRunner:
  689. 14/01/19 17:50:27 INFO mapred.LocalJobRunner:
  690. 14/01/19 17:50:28 INFO mapred.JobClient: map 65% reduce 0%
  691. 14/01/19 17:50:30 INFO mapred.LocalJobRunner:
  692. 14/01/19 17:50:33 INFO mapred.LocalJobRunner:
  693. 14/01/19 17:50:34 INFO mapred.JobClient: map 66% reduce 0%
  694. 14/01/19 17:50:36 INFO mapred.LocalJobRunner:
  695. 14/01/19 17:50:39 INFO mapred.LocalJobRunner:
  696. 14/01/19 17:50:42 INFO mapred.LocalJobRunner:
  697. 14/01/19 17:50:43 INFO mapred.JobClient: map 67% reduce 0%
  698. 14/01/19 17:50:45 INFO mapred.LocalJobRunner:
  699. 14/01/19 17:50:48 INFO mapred.LocalJobRunner:
  700. 14/01/19 17:50:51 INFO mapred.LocalJobRunner:
  701. 14/01/19 17:50:52 INFO mapred.JobClient: map 68% reduce 0%
  702. 14/01/19 17:50:54 INFO mapred.LocalJobRunner:
  703. 14/01/19 17:50:57 INFO mapred.LocalJobRunner:
  704. 14/01/19 17:50:58 INFO mapred.JobClient: map 69% reduce 0%
  705. 14/01/19 17:51:00 INFO mapred.LocalJobRunner:
  706. 14/01/19 17:51:01 INFO mapred.JobClient: map 70% reduce 0%
  707. 14/01/19 17:51:03 INFO mapred.LocalJobRunner:
  708. 14/01/19 17:51:06 INFO mapred.LocalJobRunner:
  709. 14/01/19 17:51:07 INFO mapred.JobClient: map 71% reduce 0%
  710. 14/01/19 17:51:09 INFO mapred.LocalJobRunner:
  711. 14/01/19 17:51:12 INFO mapred.LocalJobRunner:
  712. 14/01/19 17:51:13 INFO mapred.JobClient: map 72% reduce 0%
  713. 14/01/19 17:51:15 INFO mapred.LocalJobRunner:
  714. 14/01/19 17:51:18 INFO mapred.LocalJobRunner:
  715. 14/01/19 17:51:19 INFO mapred.JobClient: map 73% reduce 0%
  716. 14/01/19 17:51:21 INFO mapred.LocalJobRunner:
  717. 14/01/19 17:51:24 INFO mapred.LocalJobRunner:
  718. 14/01/19 17:51:25 INFO mapred.JobClient: map 74% reduce 0%
  719. 14/01/19 17:51:27 INFO mapred.LocalJobRunner:
  720. 14/01/19 17:51:28 INFO mapred.JobClient: map 75% reduce 0%
  721. 14/01/19 17:51:31 INFO mapred.LocalJobRunner:
  722. 14/01/19 17:51:34 INFO mapred.LocalJobRunner:
  723. 14/01/19 17:51:34 INFO mapred.JobClient: map 76% reduce 0%
  724. 14/01/19 17:51:37 INFO mapred.LocalJobRunner:
  725. 14/01/19 17:51:40 INFO mapred.LocalJobRunner:
  726. 14/01/19 17:51:40 INFO mapred.JobClient: map 77% reduce 0%
  727. 14/01/19 17:51:43 INFO mapred.LocalJobRunner:
  728. 14/01/19 17:51:43 INFO mapred.JobClient: map 78% reduce 0%
  729. 14/01/19 17:51:46 INFO mapred.LocalJobRunner:
  730. 14/01/19 17:51:49 INFO mapred.LocalJobRunner:
  731. 14/01/19 17:51:49 INFO mapred.JobClient: map 79% reduce 0%
  732. 14/01/19 17:51:52 INFO mapred.LocalJobRunner:
  733. 14/01/19 17:51:55 INFO mapred.LocalJobRunner:
  734. 14/01/19 17:51:55 INFO mapred.JobClient: map 80% reduce 0%
  735. 14/01/19 17:51:58 INFO mapred.LocalJobRunner:
  736. 14/01/19 17:51:58 INFO mapred.JobClient: map 82% reduce 0%
  737. 14/01/19 17:52:01 INFO mapred.LocalJobRunner:
  738. 14/01/19 17:52:04 INFO mapred.LocalJobRunner:
  739. 14/01/19 17:52:04 INFO mapred.JobClient: map 83% reduce 0%
  740. 14/01/19 17:52:07 INFO mapred.LocalJobRunner:
  741. 14/01/19 17:52:10 INFO mapred.LocalJobRunner:
  742. 14/01/19 17:52:10 INFO mapred.JobClient: map 84% reduce 0%
  743. 14/01/19 17:52:13 INFO mapred.LocalJobRunner:
  744. 14/01/19 17:52:13 INFO mapred.JobClient: map 85% reduce 0%
  745. 14/01/19 17:52:16 INFO mapred.LocalJobRunner:
  746. 14/01/19 17:52:19 INFO mapred.LocalJobRunner:
  747. 14/01/19 17:52:19 INFO mapred.JobClient: map 86% reduce 0%
  748. 14/01/19 17:52:22 INFO mapred.LocalJobRunner:
  749. 14/01/19 17:52:22 INFO mapred.JobClient: map 87% reduce 0%
  750. 14/01/19 17:52:25 INFO mapred.LocalJobRunner:
  751. 14/01/19 17:52:28 INFO mapred.LocalJobRunner:
  752. 14/01/19 17:52:28 INFO mapred.JobClient: map 88% reduce 0%
  753. 14/01/19 17:52:31 INFO mapred.LocalJobRunner:
  754. 14/01/19 17:52:34 INFO mapred.LocalJobRunner:
  755. 14/01/19 17:52:37 INFO mapred.LocalJobRunner:
  756. 14/01/19 17:52:37 INFO mapred.JobClient: map 89% reduce 0%
  757. 14/01/19 17:52:40 INFO mapred.LocalJobRunner:
  758. 14/01/19 17:52:43 INFO mapred.LocalJobRunner:
  759. 14/01/19 17:52:43 INFO mapred.JobClient: map 90% reduce 0%
  760. 14/01/19 17:52:46 INFO mapred.LocalJobRunner:
  761. 14/01/19 17:52:46 INFO mapred.JobClient: map 91% reduce 0%
  762. 14/01/19 17:52:49 INFO mapred.LocalJobRunner:
  763. 14/01/19 17:52:52 INFO mapred.LocalJobRunner:
  764. 14/01/19 17:52:52 INFO mapred.JobClient: map 92% reduce 0%
  765. 14/01/19 17:52:55 INFO mapred.LocalJobRunner:
  766. 14/01/19 17:52:55 INFO mapred.JobClient: map 93% reduce 0%
  767. 14/01/19 17:52:58 INFO mapred.LocalJobRunner:
  768. 14/01/19 17:53:01 INFO mapred.LocalJobRunner:
  769. 14/01/19 17:53:01 INFO mapred.JobClient: map 94% reduce 0%
  770. 14/01/19 17:53:04 INFO mapred.LocalJobRunner:
  771. 14/01/19 17:53:07 INFO mapred.LocalJobRunner:
  772. 14/01/19 17:53:07 INFO mapred.JobClient: map 95% reduce 0%
  773. 14/01/19 17:53:10 INFO mapred.LocalJobRunner:
  774. 14/01/19 17:53:13 INFO mapred.LocalJobRunner:
  775. 14/01/19 17:53:13 INFO mapred.JobClient: map 96% reduce 0%
  776. 14/01/19 17:53:16 INFO mapred.LocalJobRunner:
  777. 14/01/19 17:53:19 INFO mapred.LocalJobRunner:
  778. 14/01/19 17:53:19 INFO mapred.JobClient: map 97% reduce 0%
  779. 14/01/19 17:53:22 INFO mapred.LocalJobRunner:
  780. 14/01/19 17:53:25 INFO mapred.LocalJobRunner:
  781. 14/01/19 17:53:25 INFO mapred.JobClient: map 98% reduce 0%
  782. 14/01/19 17:53:28 INFO mapred.LocalJobRunner:
  783. 14/01/19 17:53:31 INFO mapred.LocalJobRunner:
  784. 14/01/19 17:53:34 INFO mapred.LocalJobRunner:
  785. 14/01/19 17:53:34 INFO mapred.JobClient: map 99% reduce 0%
  786. 14/01/19 17:53:37 INFO mapred.LocalJobRunner:
  787. 14/01/19 17:53:39 INFO mapred.MapTask: Starting flush of map output
  788. 14/01/19 17:53:39 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  789. 14/01/19 17:53:39 INFO compress.CodecPool: Got brand-new compressor
  790. 14/01/19 17:53:40 INFO mapred.LocalJobRunner:
  791. 14/01/19 17:53:40 INFO mapred.MapTask: Finished spill 0
  792. 14/01/19 17:53:40 INFO mapred.Task: Task:attempt_local599981549_0001_m_000000_0 is done. And is in the process of commiting
  793. 14/01/19 17:53:40 INFO mapred.LocalJobRunner:
  794. 14/01/19 17:53:40 INFO mapred.Task: Task 'attempt_local599981549_0001_m_000000_0' done.
  795. 14/01/19 17:53:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local599981549_0001_m_000000_0
  796. 14/01/19 17:53:40 INFO mapred.LocalJobRunner: Map task executor complete.
  797. 14/01/19 17:53:40 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@b1e1e37
  798. 14/01/19 17:53:40 INFO mapred.LocalJobRunner:
  799. 14/01/19 17:53:40 INFO mapred.Merger: Merging 1 sorted segments
  800. 14/01/19 17:53:40 INFO compress.CodecPool: Got brand-new decompressor
  801. 14/01/19 17:53:40 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 863539 bytes
  802. 14/01/19 17:53:40 INFO mapred.LocalJobRunner:
  803. 14/01/19 17:53:40 INFO mapreduce.StreamingKMeansThread: Estimated Points: 144
  804. 14/01/19 17:53:40 INFO mapreduce.StreamingKMeansReducer: Number of Centroids: 0
  805. 14/01/19 17:53:40 WARN mapred.LocalJobRunner: job_local599981549_0001
  806. java.lang.IllegalArgumentException: Must have nonzero number of training and test vectors. Asked for %.1f %% of %d vectors for test [10.000000149011612, 0]
  807. at com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
  808. at org.apache.mahout.clustering.streaming.cluster.BallKMeans.splitTrainTest(BallKMeans.java:176)
  809. at org.apache.mahout.clustering.streaming.cluster.BallKMeans.cluster(BallKMeans.java:192)
  810. at org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansReducer.getBestCentroids(StreamingKMeansReducer.java:107)
  811. at org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansReducer.reduce(StreamingKMeansReducer.java:73)
  812. at org.apache.mahout.clustering.streaming.mapreduce.StreamingKMeansReducer.reduce(StreamingKMeansReducer.java:37)
  813. at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
  814. at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
  815. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
  816. at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
  817. 14/01/19 17:53:40 INFO mapred.JobClient: map 100% reduce 0%
  818. 14/01/19 17:53:40 INFO mapred.JobClient: Job complete: job_local599981549_0001
  819. 14/01/19 17:53:40 INFO mapred.JobClient: Counters: 19
  820. 14/01/19 17:53:40 INFO mapred.JobClient: File Input Format Counters
  821. 14/01/19 17:53:40 INFO mapred.JobClient: Bytes Read=17156391
  822. 14/01/19 17:53:40 INFO mapred.JobClient: FileSystemCounters
  823. 14/01/19 17:53:40 INFO mapred.JobClient: FILE_BYTES_READ=41214661
  824. 14/01/19 17:53:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=25165420
  825. 14/01/19 17:53:40 INFO mapred.JobClient: Map-Reduce Framework
  826. 14/01/19 17:53:40 INFO mapred.JobClient: Reduce input groups=0
  827. 14/01/19 17:53:40 INFO mapred.JobClient: Map output materialized bytes=863543
  828. 14/01/19 17:53:40 INFO mapred.JobClient: Combine output records=0
  829. 14/01/19 17:53:40 INFO mapred.JobClient: Map input records=21578
  830. 14/01/19 17:53:40 INFO mapred.JobClient: Reduce shuffle bytes=0
  831. 14/01/19 17:53:40 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
  832. 14/01/19 17:53:40 INFO mapred.JobClient: Reduce output records=0
  833. 14/01/19 17:53:40 INFO mapred.JobClient: Spilled Records=144
  834. 14/01/19 17:53:40 INFO mapred.JobClient: Map output bytes=1553226
  835. 14/01/19 17:53:40 INFO mapred.JobClient: Total committed heap usage (bytes)=265289728
  836. 14/01/19 17:53:40 INFO mapred.JobClient: CPU time spent (ms)=0
  837. 14/01/19 17:53:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
  838. 14/01/19 17:53:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=161
  839. 14/01/19 17:53:40 INFO mapred.JobClient: Map output records=144
  840. 14/01/19 17:53:40 INFO mapred.JobClient: Combine input records=0
  841. 14/01/19 17:53:40 INFO mapred.JobClient: Reduce input records=0
  842. 14/01/19 17:53:40 INFO driver.MahoutDriver: Program took 379635 ms (Minutes: 6.32725)
  843. MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
  844. Warning: $HADOOP_HOME is deprecated.
  845.  
  846. Running on hadoop, using /opt/hadoop/bin/hadoop and HADOOP_CONF_DIR=$HADOOP_HOME/conf
  847. MAHOUT-JOB: /home/frank/Desktop/mahout-distribution-0.9/examples/target/mahout-examples-0.9-job.jar
  848. Warning: $HADOOP_HOME is deprecated.
  849.  
  850. 14/01/19 17:53:42 WARN driver.MahoutDriver: No qualcluster.props found on classpath, will use command-line arguments only
  851. Num clusters: 0; maxDistance: 0.000000
  852. [Dunn Index] First: Infinity
  853. [Davies-Bouldin Index] First: NaN
  854. 14/01/19 17:53:42 INFO driver.MahoutDriver: Program took 254 ms (Minutes: 0.004233333333333334)
  855. cluster,distance.mean,distance.sd,distance.q0,distance.q1,distance.q2,distance.q3,distance.q4,count,is.train
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!