API tools faq
paste
Advertisement
mahmoodn

hadoop indexdata

mahmoodn
May 4th, 2015
668
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 10.45 KB | None | 0 0
raw download clone embed print report
  1. [mahmood@tiger conf]$ cat /etc/hosts
  2. 127.0.0.1       localhost.localdomain localhost
  3. 192.168.1.5    tiger.ipm.com tiger
  4. 192.168.1.100  orca
  5. 192.168.1.6    zardalou
  6.  
  7.  
  8.  
  9.  
  10. [mahmood@tiger Index]$ hadoop dfsadmin -report
  11. Warning: $HADOOP_HOME is deprecated.
  12. Configured Capacity: 2953524240384 (2.69 TB)
  13. Present Capacity: 602299000644 (560.93 GB)
  14. DFS Remaining: 602298654720 (560.93 GB)
  15. DFS Used: 345924 (337.82 KB)
  16. DFS Used%: 0%
  17. Under replicated blocks: 0
  18. Blocks with corrupt replicas: 0
  19. Missing blocks: 0
  20. -------------------------------------------------
  21. Datanodes available: 1 (1 total, 0 dead)
  22. Name: 127.0.0.1:50010
  23. Decommission Status : Normal
  24. Configured Capacity: 2953524240384 (2.69 TB)
  25. DFS Used: 345924 (337.82 KB)
  26. Non DFS Used: 2351225239740 (2.14 TB)
  27. DFS Remaining: 602298654720(560.93 GB)
  28. DFS Used%: 0%
  29. DFS Remaining%: 20.39%
  30. Last contact: Mon May 04 14:05:32 IRDT 2015
  31.  
  32.  
  33.  
  34.  
  35.  
  36.  
  37.  
  38.  
  39. [mahmood@tiger Index]$ ./genData_Index.sh
  40. + '[' '!' -e ']'
  41. + echo 'Preparing MicroBenchmarks data dir'
  42. Preparing MicroBenchmarks data dir
  43. ++ pwd
  44. + WORK_DIR=/home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index
  45. + echo 'WORK_DIR=/home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index data will be put in /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index'
  46. WORK_DIR=/home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index data will be put in /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  47. + mkdir -p /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  48. + cd ../../BigDataGeneratorSuite/Text_datagen/
  49. + echo 'print data size GB :'
  50. print data size GB :
  51. + read GB
  52. 1
  53. + a=1
  54. + let 'L=a*2'
  55. + ./gen_text_data.sh lda_wiki1w 2 8000 10000 /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  56. Mon May  4 14:05:40 IRDT 2015
  57. Mon May  4 14:08:00 IRDT 2015
  58. + cd ../../SearchEngine/Index/
  59. + /home/mahmood/bigdatabench/apache/hadoop-1.0.2/bin/hadoop fs -copyFromLocal /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  60. Warning: $HADOOP_HOME is deprecated.
  61.  
  62.  
  63.  
  64.  
  65.  
  66.  
  67.  
  68.  
  69.  
  70. [mahmood@tiger Index]$ hadoop jar indexdata.jar `pwd`/result hdfs://127.0.0.1:9000/data-Index
  71. Warning: $HADOOP_HOME is deprecated.
  72. 15/05/04 14:10:07 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 0 time(s).
  73. 15/05/04 14:10:08 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 1 time(s).
  74. 15/05/04 14:10:09 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 2 time(s).
  75. 15/05/04 14:10:10 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 3 time(s).
  76. 15/05/04 14:10:11 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 4 time(s).
  77. 15/05/04 14:10:12 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 5 time(s).
  78. 15/05/04 14:10:13 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 6 time(s).
  79. 15/05/04 14:10:14 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 7 time(s).
  80. 15/05/04 14:10:15 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 8 time(s).
  81. 15/05/04 14:10:16 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:9000. Already tried 9 time(s).
  82. Exception in thread "main" java.net.ConnectException: Call to localhost.localdomain/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
  83.         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
  84.         at org.apache.hadoop.ipc.Client.call(Client.java:1071)
  85.         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
  86.         at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
  87.         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
  88.         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
  89.         at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
  90.         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
  91.         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
  92.         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
  93.         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
  94.         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
  95.         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
  96.         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
  97.         at IndexHDFS.indexData(IndexHDFS.java:88)
  98.         at IndexHDFS.main(IndexHDFS.java:72)
  99.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  100.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  101.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  102.         at java.lang.reflect.Method.invoke(Method.java:606)
  103.         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  104. Caused by: java.net.ConnectException: Connection refused
  105.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  106.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
  107.         at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
  108.         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
  109.         at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
  110.         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
  111.         at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
  112.         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
  113.         at org.apache.hadoop.ipc.Client.call(Client.java:1046)
  114.         ... 19 more
  115.  
  116.  
  117.  
  118.  
  119.  
  120.  
  121.  
  122.  
  123.  
  124. [mahmood@tiger Index]$ hadoop dfsadmin -report
  125. Warning: $HADOOP_HOME is deprecated.
  126. Configured Capacity: 2953524240384 (2.69 TB)
  127. Present Capacity: 602298879129 (560.93 GB)
  128. DFS Remaining: 601237704704 (559.95 GB)
  129. DFS Used: 1061174425 (1012.01 MB)
  130. DFS Used%: 0.18%
  131. Under replicated blocks: 0
  132. Blocks with corrupt replicas: 0
  133. Missing blocks: 0
  134. ------------------------------------------------
  135. Datanodes available: 1 (1 total, 0 dead)
  136. Name: 127.0.0.1:50010
  137. Decommission Status : Normal
  138. Configured Capacity: 2953524240384 (2.69 TB)
  139. DFS Used: 1061174425 (1012.01 MB)
  140. Non DFS Used: 2351225361255 (2.14 TB)
  141. DFS Remaining: 601237704704(559.95 GB)
  142. DFS Used%: 0.04%
  143. DFS Remaining%: 20.36%
  144. Last contact: Mon May 04 14:10:20 IRDT 2015
  145.  
  146.  
  147.  
  148.  
  149.  
  150.  
  151.  
  152.  
  153. [mahmood@tiger Index]$ hadoop jar indexdata.jar `pwd`/result hdfs://tiger:9000/data-Index
  154. Warning: $HADOOP_HOME is deprecated.
  155. 15/05/04 14:10:33 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 0 time(s).
  156. 15/05/04 14:10:34 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 1 time(s).
  157. 15/05/04 14:10:35 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 2 time(s).
  158. 15/05/04 14:10:36 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 3 time(s).
  159. 15/05/04 14:10:37 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 4 time(s).
  160. 15/05/04 14:10:38 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 5 time(s).
  161. 15/05/04 14:10:39 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 6 time(s).
  162. 15/05/04 14:10:40 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 7 time(s).
  163. 15/05/04 14:10:41 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 8 time(s).
  164. 15/05/04 14:10:42 INFO ipc.Client: Retrying connect to server: tiger/192.168.1.5:9000. Already tried 9 time(s).
  165. Exception in thread "main" java.net.ConnectException: Call to tiger/192.168.1.5:9000 failed on connection exception: java.net.ConnectException: Connection refused
  166.         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
  167.         at org.apache.hadoop.ipc.Client.call(Client.java:1071)
  168.         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
  169.         at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
  170.         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
  171.         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
  172.         at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
  173.         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
  174.         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
  175.         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
  176.         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
  177.         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
  178.         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
  179.         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
  180.         at IndexHDFS.indexData(IndexHDFS.java:88)
  181.         at IndexHDFS.main(IndexHDFS.java:72)
  182.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  183.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  184.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  185.         at java.lang.reflect.Method.invoke(Method.java:606)
  186.         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  187. Caused by: java.net.ConnectException: Connection refused
  188.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  189.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
  190.         at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
  191.         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
  192.         at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
  193.         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
  194.         at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
  195.         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
  196.         at org.apache.hadoop.ipc.Client.call(Client.java:1046)
  197.         ... 19 more
  198. [mahmood@tiger Index]$
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!