API tools faq
paste
Guest User

Silvia Adduci

a guest
Oct 31st, 2014
195
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 14.38 KB | None | 0 0
raw download clone embed print report
  1. [training@localhost ~]$ ls
  2. Desktop Downloads Pictures src workspace
  3. Documents eclipse Public udacity_training
  4. [training@localhost ~]$ cd udacity_training/
  5. [training@localhost udacity_training]$ ls
  6. code data
  7. [training@localhost udacity_training]$ cd data
  8. [training@localhost data]$ ls
  9. access_log.gz purchases.txt
  10. [training@localhost data]$ hadoop fs -ls
  11. [training@localhost data]$ hadoop fs -put purchases.txt
  12. [training@localhost data]$ hadoop fs -ls
  13. Found 1 items
  14. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:44 purchases.txt
  15. [training@localhost data]$ hadoop fs -tail purchases.txt
  16. 31 17:59 Norfolk Toys 164.34 MasterCard
  17. 2012-12-31 17:59 Chula Vista Music 380.67 Visa
  18. 2012-12-31 17:59 Hialeah Toys 115.21 MasterCard
  19. 2012-12-31 17:59 Indianapolis Men's Clothing 158.28 MasterCard
  20. 2012-12-31 17:59 Norfolk Garden 414.09 MasterCard
  21. 2012-12-31 17:59 Baltimore DVDs 467.3 Visa
  22. 2012-12-31 17:59 Santa Ana Video Games 144.73 Visa
  23. 2012-12-31 17:59 Gilbert Consumer Electronics 354.66 Discover
  24. 2012-12-31 17:59 Memphis Sporting Goods 124.79 Amex
  25. 2012-12-31 17:59 Chicago Men's Clothing 386.54 MasterCard
  26. 2012-12-31 17:59 Birmingham CDs 118.04 Cash
  27. 2012-12-31 17:59 Las Vegas Health and Beauty 420.46 Amex
  28. 2012-12-31 17:59 Wichita Toys 383.9 Cash
  29. 2012-12-31 17:59 Tucson Pet Supplies 268.39 MasterCard
  30. 2012-12-31 17:59 Glendale Women's Clothing 68.05 Amex
  31. 2012-12-31 17:59 Albuquerque Toys 345.7 MasterCard
  32. 2012-12-31 17:59 Rochester DVDs 399.57 Amex
  33. 2012-12-31 17:59 Greensboro Baby 277.27 Discover
  34. 2012-12-31 17:59 Arlington Women's Clothing 134.95 MasterCard
  35. 2012-12-31 17:59 Corpus Christi DVDs 441.61 Discover
  36. [training@localhost data]$ hadoop fs -mkdir myinput
  37. hadoop[training@localhost data]$ hadoop fs -put purchases.txt myinput
  38. [training@localhost data]$ hadoop fs -ls
  39. Found 2 items
  40. drwxr-xr-x - training supergroup 0 2014-10-31 17:49 myinput
  41. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:44 purchases.txt
  42. [training@localhost data]$ hadoop fs -ls myinput
  43. Found 1 items
  44. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:49 myinput/purchases.txt
  45. [training@localhost data]$ ls
  46. access_log.gz purchases.txt
  47. [training@localhost data]$ cd
  48. [training@localhost ~]$ ls
  49. Desktop Downloads Pictures src workspace
  50. Documents eclipse Public udacity_training
  51. [training@localhost ~]$ cd udacity_training/
  52. [training@localhost udacity_training]$ ls
  53. code data
  54. [training@localhost udacity_training]$ cd code
  55. [training@localhost code]$ ls
  56. mapper.py reducer.py
  57. [training@localhost code]$ hadoop fs -ls
  58. Found 2 items
  59. drwxr-xr-x - training supergroup 0 2014-10-31 17:49 myinput
  60. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:44 purchases.txt
  61. [training@localhost code]$ hadoop fs -ls myinput
  62. Found 1 items
  63. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:49 myinput/purchases.txt
  64. [training@localhost code]$ hadoop jar /usr/lib
  65. lib/ libexec/
  66. [training@localhost code]$ hadoop jar /usr/lib
  67. lib/ libexec/
  68. [training@localhost code]$ hadoop jar /usr/lib
  69. lib/ libexec/
  70. [training@localhost code]$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.1.1.jar -mapper mapper.py -reducer reducer.py -file mapper.py -file reducer.py -input myinput -output joboutput
  71. packageJobJar: [mapper.py, reducer.py, /tmp/hadoop-training/hadoop-unjar3607163364207776574/] [] /tmp/streamjob5522053172829658683.jar tmpDir=null
  72. 14/10/31 18:02:12 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  73. 14/10/31 18:02:12 WARN snappy.LoadSnappy: Snappy native library is available
  74. 14/10/31 18:02:12 INFO snappy.LoadSnappy: Snappy native library loaded
  75. 14/10/31 18:02:12 INFO mapred.FileInputFormat: Total input paths to process : 1
  76. 14/10/31 18:02:12 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-hdfs/cache/training/mapred/local]
  77. 14/10/31 18:02:12 INFO streaming.StreamJob: Running job: job_201410311722_0001
  78. 14/10/31 18:02:12 INFO streaming.StreamJob: To kill this job, run:
  79. 14/10/31 18:02:12 INFO streaming.StreamJob: UNDEF/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201410311722_0001
  80. 14/10/31 18:02:12 INFO streaming.StreamJob: Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201410311722_0001
  81. 14/10/31 18:02:13 INFO streaming.StreamJob: map 0% reduce 0%
  82. 14/10/31 18:02:27 INFO streaming.StreamJob: map 8% reduce 0%
  83. 14/10/31 18:02:31 INFO streaming.StreamJob: map 13% reduce 0%
  84. 14/10/31 18:02:34 INFO streaming.StreamJob: map 17% reduce 0%
  85. 14/10/31 18:02:37 INFO streaming.StreamJob: map 22% reduce 0%
  86. 14/10/31 18:02:40 INFO streaming.StreamJob: map 26% reduce 0%
  87. 14/10/31 18:02:43 INFO streaming.StreamJob: map 31% reduce 0%
  88. 14/10/31 18:02:46 INFO streaming.StreamJob: map 36% reduce 0%
  89. 14/10/31 18:02:49 INFO streaming.StreamJob: map 41% reduce 0%
  90. 14/10/31 18:02:52 INFO streaming.StreamJob: map 43% reduce 0%
  91. 14/10/31 18:02:53 INFO streaming.StreamJob: map 45% reduce 0%
  92. 14/10/31 18:02:55 INFO streaming.StreamJob: map 47% reduce 0%
  93. 14/10/31 18:02:56 INFO streaming.StreamJob: map 49% reduce 0%
  94. 14/10/31 18:02:59 INFO streaming.StreamJob: map 50% reduce 0%
  95. 14/10/31 18:03:16 INFO streaming.StreamJob: map 73% reduce 17%
  96. 14/10/31 18:03:18 INFO streaming.StreamJob: map 78% reduce 17%
  97. 14/10/31 18:03:19 INFO streaming.StreamJob: map 81% reduce 17%
  98. 14/10/31 18:03:22 INFO streaming.StreamJob: map 86% reduce 25%
  99. 14/10/31 18:03:25 INFO streaming.StreamJob: map 91% reduce 25%
  100. 14/10/31 18:03:28 INFO streaming.StreamJob: map 96% reduce 25%
  101. 14/10/31 18:03:31 INFO streaming.StreamJob: map 100% reduce 25%
  102. 14/10/31 18:03:34 INFO streaming.StreamJob: map 100% reduce 33%
  103. 14/10/31 18:03:37 INFO streaming.StreamJob: map 100% reduce 69%
  104. 14/10/31 18:03:40 INFO streaming.StreamJob: map 100% reduce 76%
  105. 14/10/31 18:03:43 INFO streaming.StreamJob: map 100% reduce 82%
  106. 14/10/31 18:03:46 INFO streaming.StreamJob: map 100% reduce 87%
  107. 14/10/31 18:03:49 INFO streaming.StreamJob: map 100% reduce 93%
  108. 14/10/31 18:03:52 INFO streaming.StreamJob: map 100% reduce 100%
  109. 14/10/31 18:03:56 INFO streaming.StreamJob: Job complete: job_201410311722_0001
  110. 14/10/31 18:03:56 INFO streaming.StreamJob: Output: joboutput
  111. [training@localhost code]$ hadoop fs -ls
  112. Found 3 items
  113. drwxr-xr-x - training supergroup 0 2014-10-31 18:03 joboutput
  114. drwxr-xr-x - training supergroup 0 2014-10-31 17:49 myinput
  115. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:44 purchases.txt
  116. [training@localhost code]$ hadoop fs -ls joboutput
  117. Found 3 items
  118. -rw-r--r-- 1 training supergroup 0 2014-10-31 18:03 joboutput/_SUCCESS
  119. drwxr-xr-x - training supergroup 0 2014-10-31 18:02 joboutput/_logs
  120. -rw-r--r-- 1 training supergroup 2296 2014-10-31 18:03 joboutput/part-00000
  121. [training@localhost code]$ hadoop fs -cat joboutput/part 0000 | less
  122. [training@localhost code]$ hadoop fs -cat joboutput/part-0000 | less
  123. [training@localhost code]$
  124. [training@localhost code]$ hadoop fs -cat joboutput/part-00000 | less
  125. [training@localhost code]$
  126. [training@localhost code]$ hadoop fs -cat joboutput/part-00000 mylocalfile.txt
  127. Albuquerque 10052311.42
  128. Anaheim 10076416.36
  129. Anchorage 9933500.4
  130. Arlington 10072207.97
  131. Atlanta 9997146.7
  132. Aurora 9992970.92
  133. Austin 10057158.9
  134. Bakersfield 10031208.92
  135. Baltimore 10096521.45
  136. Baton Rouge 10131273.23
  137. Birmingham 10076606.52
  138. Boise 10039166.74
  139. Boston 10039473.28
  140. Buffalo 10001941.19
  141. Chandler 9919559.86
  142. Charlotte 10112531.34
  143. Chesapeake 10038504.92
  144. Chicago 10062522.07
  145. Chula Vista 9974951.34
  146. Cincinnati 10139505.74
  147. Cleveland 10067835.84
  148. Colorado Springs 10061105.87
  149. Columbus 10035241.03
  150. Corpus Christi 9976522.77
  151. Dallas 10066548.45
  152. Denver 10031534.87
  153. Detroit 9979260.76
  154. Durham 10153890.21
  155. El Paso 10016409.97
  156. Fort Wayne 10132594.02
  157. Fort Worth 10120830.65
  158. Fremont 10053242.36
  159. Fresno 9976260.26
  160. Garland 10071043.92
  161. Gilbert 10062115.19
  162. Glendale 10044493.97
  163. Greensboro 10033781.39
  164. Henderson 10053416.05
  165. Hialeah 10047052.76
  166. Honolulu 10006273.49
  167. Houston 10042106.27
  168. Indianapolis 10090272.77
  169. Irvine 10084867.45
  170. Irving 10133944.08
  171. Jacksonville 10072003.33
  172. Jersey City 9920141.87
  173. Kansas City 9968118.73
  174. Laredo 10144604.98
  175. Las Vegas 10054257.98
  176. Lexington 10084510.95
  177. Lincoln 10069485.4
  178. Long Beach 10006380.25
  179. Los Angeles 10084576.8
  180. Louisville 10008566.47
  181. Lubbock 9958119.15
  182. Madison 10032035.54
  183. Memphis 10038565.32
  184. Mesa 10053642.6
  185. Miami 9947316.07
  186. Milwaukee 10064482.65
  187. Minneapolis 10011757.78
  188. Nashville 9961450.51
  189. New Orleans 9949257.75
  190. New York 10085293.55
  191. Newark 10144052.8
  192. Norfolk 10088563.17
  193. North Las Vegas 10029652.51
  194. Oakland 9947292.52
  195. Oklahoma City 10118986.25
  196. Omaha 10026642.34
  197. Orlando 10074922.52
  198. Philadelphia 10190080.26
  199. Phoenix 10079076.7
  200. Pittsburgh 10090124.82
  201. Plano 10046103.61
  202. Portland 10007635.77
  203. Raleigh 10061442.54
  204. Reno 10079955.16
  205. Richmond 9992941.59
  206. Riverside 10006695.42
  207. Rochester 10067606.92
  208. Sacramento 10123468.18
  209. Saint Paul 10057233.57
  210. San Antonio 10014441.7
  211. San Bernardino 9965152.04
  212. San Diego 9966038.39
  213. San Francisco 9995570.54
  214. San Jose 9936721.41
  215. Santa Ana 10050309.93
  216. Scottsdale 10037929.85
  217. Seattle 9936267.37
  218. Spokane 10083362.98
  219. St. Louis 10002105.14
  220. St. Petersburg 9986495.54
  221. Stockton 10006412.64
  222. Tampa 10106428.55
  223. Toledo 10020768.88
  224. Tucson 9998252.47
  225. Tulsa 10064955.9
  226. Virginia Beach 10086553.5
  227. Washington 10139363.39
  228. Wichita 10083643.21
  229. Winston–Salem 10044011.83
  230. cat: `mylocalfile.txt': No such file or directory
  231. [training@localhost code]$ hadoop fs -get joboutput/part-00000 mylocalfile.txt
  232. [training@localhost code]$ hs mapper.py reducer.py myinput joboutput
  233. packageJobJar: [mapper.py, reducer.py, /tmp/hadoop-training/hadoop-unjar3086953528091346500/] [] /tmp/streamjob5740223419707269269.jar tmpDir=null
  234. 14/10/31 18:10:16 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  235. 14/10/31 18:10:16 INFO mapred.JobClient: Cleaning up the staging area hdfs://0.0.0.0:8020/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/training/.staging/job_201410311722_0002
  236. 14/10/31 18:10:16 ERROR security.UserGroupInformation: PriviledgedActionException as:training (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://0.0.0.0:8020/user/training/joboutput already exists
  237. 14/10/31 18:10:16 ERROR streaming.StreamJob: Error launching job , Output path already exists : Output directory hdfs://0.0.0.0:8020/user/training/joboutput already exists
  238. Streaming Command Failed!
  239. [training@localhost code]$ hs mapper.py reducer.py myinput newoutputdir
  240. packageJobJar: [mapper.py, reducer.py, /tmp/hadoop-training/hadoop-unjar2291186339325694346/] [] /tmp/streamjob1552916525720226991.jar tmpDir=null
  241. 14/10/31 18:11:39 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  242. 14/10/31 18:11:39 WARN snappy.LoadSnappy: Snappy native library is available
  243. 14/10/31 18:11:39 INFO snappy.LoadSnappy: Snappy native library loaded
  244. 14/10/31 18:11:40 INFO mapred.FileInputFormat: Total input paths to process : 1
  245. 14/10/31 18:11:40 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-hdfs/cache/training/mapred/local]
  246. 14/10/31 18:11:40 INFO streaming.StreamJob: Running job: job_201410311722_0003
  247. 14/10/31 18:11:40 INFO streaming.StreamJob: To kill this job, run:
  248. 14/10/31 18:11:40 INFO streaming.StreamJob: UNDEF/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201410311722_0003
  249. 14/10/31 18:11:40 INFO streaming.StreamJob: Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201410311722_0003
  250. 14/10/31 18:11:41 INFO streaming.StreamJob: map 0% reduce 0%
  251. 14/10/31 18:11:54 INFO streaming.StreamJob: map 9% reduce 0%
  252. 14/10/31 18:11:58 INFO streaming.StreamJob: map 12% reduce 0%
  253. 14/10/31 18:12:01 INFO streaming.StreamJob: map 16% reduce 0%
  254. 14/10/31 18:12:04 INFO streaming.StreamJob: map 20% reduce 0%
  255. 14/10/31 18:12:08 INFO streaming.StreamJob: map 23% reduce 0%
  256. 14/10/31 18:12:11 INFO streaming.StreamJob: map 27% reduce 0%
  257. 14/10/31 18:12:14 INFO streaming.StreamJob: map 32% reduce 0%
  258. 14/10/31 18:12:17 INFO streaming.StreamJob: map 37% reduce 0%
  259. 14/10/31 18:12:20 INFO streaming.StreamJob: map 41% reduce 0%
  260. 14/10/31 18:12:23 INFO streaming.StreamJob: map 47% reduce 0%
  261. 14/10/31 18:12:26 INFO streaming.StreamJob: map 50% reduce 0%
  262. 14/10/31 18:12:40 INFO streaming.StreamJob: map 53% reduce 17%
  263. 14/10/31 18:12:41 INFO streaming.StreamJob: map 71% reduce 17%
  264. 14/10/31 18:12:43 INFO streaming.StreamJob: map 81% reduce 17%
  265. 14/10/31 18:12:46 INFO streaming.StreamJob: map 86% reduce 17%
  266. 14/10/31 18:12:47 INFO streaming.StreamJob: map 86% reduce 25%
  267. 14/10/31 18:12:49 INFO streaming.StreamJob: map 91% reduce 25%
  268. 14/10/31 18:12:53 INFO streaming.StreamJob: map 96% reduce 25%
  269. 14/10/31 18:12:56 INFO streaming.StreamJob: map 100% reduce 25%
  270. 14/10/31 18:12:59 INFO streaming.StreamJob: map 100% reduce 33%
  271. 14/10/31 18:13:02 INFO streaming.StreamJob: map 100% reduce 71%
  272. 14/10/31 18:13:05 INFO streaming.StreamJob: map 100% reduce 78%
  273. 14/10/31 18:13:08 INFO streaming.StreamJob: map 100% reduce 84%
  274. 14/10/31 18:13:11 INFO streaming.StreamJob: map 100% reduce 90%
  275. 14/10/31 18:13:14 INFO streaming.StreamJob: map 100% reduce 97%
  276. 14/10/31 18:13:16 INFO streaming.StreamJob: map 100% reduce 100%
  277. 14/10/31 18:13:18 INFO streaming.StreamJob: Job complete: job_201410311722_0003
  278. 14/10/31 18:13:18 INFO streaming.StreamJob: Output: newoutputdir
  279. [training@localhost code]$ ls
  280. mapper.py mylocalfile.txt reducer.py
  281. [training@localhost code]$ hadoop fs -ls
  282. Found 4 items
  283. drwxr-xr-x - training supergroup 0 2014-10-31 18:03 joboutput
  284. drwxr-xr-x - training supergroup 0 2014-10-31 17:49 myinput
  285. drwxr-xr-x - training supergroup 0 2014-10-31 18:13 newoutputdir
  286. -rw-r--r-- 1 training supergroup 211312924 2014-10-31 17:44 purchases.txt
  287. [training@localhost code]$ cd newoputputdir
  288. bash: cd: newoputputdir: No such file or directory
  289. [training@localhost code]$ hadoop fs -cat newoutputdir
  290. cat: `newoutputdir': Is a directory
  291. [training@localhost code]$ cd newoutputdir
  292. bash: cd: newoutputdir: No such file or directory
  293. [training@localhost code]$ newoutputdir
  294. bash: newoutputdir: command not found
  295. [training@localhost code]$
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!