API tools faq
paste
Advertisement
mahmoodn

hadoop indexdata

mahmoodn
May 5th, 2015
570
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 8.23 KB | None | 0 0
raw download clone embed print report
  1. [mahmood@tiger Index]$ cat ~/bigdatabench/apache/hadoop-1.0.2/conf/core-site.xml
  2. <?xml version="1.0"?>
  3. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  4. <!-- Put site-specific property overrides in this file. -->
  5. <configuration>
  6. <property>
  7.   <name>fs.default.name</name>
  8.   <value>hdfs://tiger:9000</value>
  9. </property>
  10. </configuration>
  11.  
  12.  
  13.  
  14.  
  15.  
  16.  
  17.  
  18. [root@tiger Index]# iptables -L -n
  19. Chain INPUT (policy ACCEPT)
  20. target     prot opt source               destination
  21.  
  22. Chain FORWARD (policy ACCEPT)
  23. target     prot opt source               destination
  24.  
  25. Chain OUTPUT (policy ACCEPT)
  26. target     prot opt source               destination
  27.  
  28.  
  29.  
  30.  
  31.  
  32. [mahmood@tiger Index]$ stop-all.sh
  33. Warning: $HADOOP_HOME is deprecated.
  34. stopping jobtracker
  35. localhost: Warning: $HADOOP_HOME is deprecated.
  36. localhost:
  37. localhost: stopping tasktracker
  38. stopping namenode
  39. localhost: Warning: $HADOOP_HOME is deprecated.
  40. localhost:
  41. localhost: stopping datanode
  42. localhost: Warning: $HADOOP_HOME is deprecated.
  43. localhost:
  44. localhost: stopping secondarynamenode
  45.  
  46.  
  47.  
  48.  
  49.  
  50.  
  51.  
  52. [mahmood@tiger Index]$ rm -rf /data/apache/hadoop-1.0.2/folders/*
  53. [mahmood@tiger Index]$ rm -rf ~/bigdatabench/apache/hadoop-1.0.2/logs/*
  54.  
  55.  
  56.  
  57.  
  58.  
  59. [mahmood@tiger Index]$ hadoop  namenode -format
  60. Warning: $HADOOP_HOME is deprecated.
  61. 15/05/05 13:53:37 INFO namenode.NameNode: STARTUP_MSG:
  62. /************************************************************
  63. STARTUP_MSG: Starting NameNode
  64. STARTUP_MSG:   host = tiger/192.168.1.5
  65. STARTUP_MSG:   args = [-format]
  66. STARTUP_MSG:   version = 1.0.2
  67. STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
  68. ************************************************************/
  69. 15/05/05 13:53:38 INFO util.GSet: VM type       = 64-bit
  70. 15/05/05 13:53:38 INFO util.GSet: 2% max memory = 19.17 MB
  71. 15/05/05 13:53:38 INFO util.GSet: capacity      = 2^21 = 2097152 entries
  72. 15/05/05 13:53:38 INFO util.GSet: recommended=2097152, actual=2097152
  73. 15/05/05 13:53:38 INFO namenode.FSNamesystem: fsOwner=mahmood
  74. 15/05/05 13:53:38 INFO namenode.FSNamesystem: supergroup=supergroup
  75. 15/05/05 13:53:38 INFO namenode.FSNamesystem: isPermissionEnabled=true
  76. 15/05/05 13:53:38 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
  77. 15/05/05 13:53:38 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
  78. 15/05/05 13:53:38 INFO namenode.NameNode: Caching file names occuring more than 10 times
  79. 15/05/05 13:53:38 INFO common.Storage: Image file of size 113 saved in 0 seconds.
  80. 15/05/05 13:53:39 INFO common.Storage: Storage directory /data/apache/hadoop-1.0.2/folders/name has been successfully formatted.
  81. 15/05/05 13:53:39 INFO namenode.NameNode: SHUTDOWN_MSG:
  82. /************************************************************
  83. SHUTDOWN_MSG: Shutting down NameNode at tiger/192.168.1.5
  84. ************************************************************/
  85.  
  86.  
  87.  
  88.  
  89.  
  90.  
  91.  
  92. [mahmood@tiger Index]$ start-all.sh
  93. Warning: $HADOOP_HOME is deprecated.
  94. starting namenode, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-namenode-tiger.out
  95. localhost: Warning: $HADOOP_HOME is deprecated.
  96. localhost:
  97. localhost: starting datanode, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-datanode-tiger.out
  98. localhost: Warning: $HADOOP_HOME is deprecated.
  99. localhost:
  100. localhost: starting secondarynamenode, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-secondarynamenode-tiger.out
  101. starting jobtracker, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-jobtracker-tiger.out
  102. localhost: Warning: $HADOOP_HOME is deprecated.
  103. localhost:
  104. localhost: starting tasktracker, logging to /home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-tasktracker-tiger.out
  105.  
  106.  
  107.  
  108.  
  109.  
  110.  
  111.  
  112.  
  113. [mahmood@tiger Index]$ hadoop dfsadmin -report
  114. Warning: $HADOOP_HOME is deprecated.
  115. Configured Capacity: 2953524240384 (2.69 TB)
  116. Present Capacity: 602299076623 (560.93 GB)
  117. DFS Remaining: 602299047936 (560.93 GB)
  118. DFS Used: 28687 (28.01 KB)
  119. DFS Used%: 0%
  120. Under replicated blocks: 0
  121. Blocks with corrupt replicas: 0
  122. Missing blocks: 0
  123. -------------------------------------------------
  124. Datanodes available: 1 (1 total, 0 dead)
  125. Name: 192.168.1.5:50010
  126. Decommission Status : Normal
  127. Configured Capacity: 2953524240384 (2.69 TB)
  128. DFS Used: 28687 (28.01 KB)
  129. Non DFS Used: 2351225163761 (2.14 TB)
  130. DFS Remaining: 602299047936(560.93 GB)
  131. DFS Used%: 0%
  132. DFS Remaining%: 20.39%
  133. Last contact: Tue May 05 13:53:58 IRDT 2015
  134.  
  135.  
  136.  
  137.  
  138.  
  139.  
  140.  
  141. [mahmood@tiger Index]$ ./genData_Index.sh
  142. + '[' '!' -e ']'
  143. + echo 'Preparing MicroBenchmarks data dir'
  144. Preparing MicroBenchmarks data dir
  145. ++ pwd
  146. + WORK_DIR=/home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index
  147. + echo 'WORK_DIR=/home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index data will be put in /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index'
  148. WORK_DIR=/home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index data will be put in /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  149. + mkdir -p /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  150. + cd ../../BigDataGeneratorSuite/Text_datagen/
  151. + echo 'print data size GB :'
  152. print data size GB :
  153. + read GB
  154. 1
  155. + a=1
  156. + let 'L=a*2'
  157. + ./gen_text_data.sh lda_wiki1w 2 8000 10000 /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  158. Tue May  5 13:54:05 IRDT 2015
  159. Tue May  5 13:56:25 IRDT 2015
  160. + cd ../../SearchEngine/Index/
  161. + /home/mahmood/bigdatabench/apache/hadoop-1.0.2/bin/hadoop fs -copyFromLocal /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index /home/mahmood/bigdatabench/BigDataBench_V3.1_Hadoop_Hive/SearchEngine/Index/data-Index
  162. Warning: $HADOOP_HOME is deprecated.
  163.  
  164.  
  165.  
  166.  
  167.  
  168.  
  169.  
  170.  
  171.  
  172.  
  173.  
  174. [mahmood@tiger Index]$ hadoop jar indexdata.jar result hdfs://tiger:9000/data-Index
  175. Warning: $HADOOP_HOME is deprecated.
  176. Exception in thread "main" java.lang.NullPointerException
  177.         at IndexHDFS.indexData(IndexHDFS.java:92)
  178.         at IndexHDFS.main(IndexHDFS.java:72)
  179.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  180.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  181.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  182.         at java.lang.reflect.Method.invoke(Method.java:606)
  183.         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  184.        
  185.        
  186.        
  187.        
  188.        
  189. [mahmood@tiger Index]$ hadoop jar indexdata.jar result hdfs://194.225.69.104:9000/data-Index
  190. Warning: $HADOOP_HOME is deprecated.
  191. 15/05/05 13:58:19 INFO ipc.Client: Retrying connect to server: 194.225.69.104/194.225.69.104:9000. Already tried 0 time(s).
  192. 15/05/05 13:58:20 INFO ipc.Client: Retrying connect to server: 194.225.69.104/194.225.69.104:9000. Already tried 1 time(s).
  193.  
  194.  
  195.  
  196.  
  197.  
  198. [mahmood@tiger Index]$ hadoop jar indexdata.jar result hdfs://localhost:9000/data-Index
  199. Warning: $HADOOP_HOME is deprecated.
  200. 15/05/05 13:58:37 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).
  201. 15/05/05 13:58:38 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).
  202. 15/05/05 13:58:39 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).
  203.  
  204.  
  205.  
  206.  
  207.  
  208.  
  209.  
  210. [mahmood@tiger Index]$ hadoop jar indexdata.jar result hdfs://192.168.1.5:9000/data-Index
  211. Warning: $HADOOP_HOME is deprecated.
  212. Exception in thread "main" java.lang.NullPointerException
  213.         at IndexHDFS.indexData(IndexHDFS.java:92)
  214.         at IndexHDFS.main(IndexHDFS.java:72)
  215.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  216.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  217.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  218.         at java.lang.reflect.Method.invoke(Method.java:606)
  219.         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!