Guest User

Juju problem with hadoop charms

a guest
Feb 9th, 2012
136
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. Hi all! I need some help to deploy hadoop charms. I would like to use hadoop-master and hadoop-slave formulas from https://code.launchpad.net/~charmers/. Apparently all is working fine with my juju because I can use the mysql and mediawiki charms normally. But, with the hadoop charms the problem is that the service stops at some point.
  2.  
  3. A interesting thing is that we can debug a juju deployment using some commands. According to juju docs, we can go step by step in the deployment process (install, config_change, start etc). Below, a sequence of the facts.
  4.  
  5. jorge@s:~/charms$ juju bootstrap
  6. 2012-02-09 11:58:45,765 INFO Bootstrapping environment 'sample' (type: ec2)...
  7. 2012-02-09 11:58:47,454 INFO 'bootstrap' command finished successfully
  8.  
  9. jorge@s:~/charms$ juju deploy --repository ~/charms local:hadoop-master namenode
  10. 2012-02-09 12:01:06,649 INFO Connecting to environment.
  11. 2012-02-09 12:01:07,289 INFO Charm deployed as service: 'namenode'
  12. 2012-02-09 12:01:07,290 INFO 'deploy' command finished successfully
  13.  
  14. Juju returns some succesful messages, but running "juju status" I get:
  15.  
  16. jorge@sold049:~/charms$ juju status
  17. 2012-02-09 12:02:08,958 INFO Connecting to environment.
  18. machines:
  19. 0: {dns-name: 172.16.0.2, instance-id: i-00000056}
  20. 1: {dns-name: 172.16.0.3, instance-id: i-00000057}
  21. services:
  22. namenode:
  23. charm: local:oneiric/hadoop-master-3
  24. relations: {}
  25. units:
  26. namenode/0:
  27. machine: 1
  28. public-address: server-87.novalocal
  29. relations: {}
  30. state: install_error
  31. 2012-02-09 12:02:09,765 INFO 'status' command finished successfully
  32.  
  33. The state is install_error!!
  34.  
  35. Before debug, here is the logs from /var/lib/juju/units/namenode-0/charm.log:
  36.  
  37. ____________________
  38.  
  39. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.3
  40. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@662: Client environment:host.name=server-87
  41. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@669: Client environment:os.name=Linux
  42. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@670: Client environment:os.arch=3.0.0-12-server
  43. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@671: Client environment:os.version=#20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011
  44. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@679: Client environment:user.name=(null)
  45. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@687: Client environment:user.home=/root
  46. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@699: Client environment:user.dir=/
  47. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=172.16.0.2:2181 sessionTimeout=10000 watcher=0x7ff6d8328650 sessionId=0 sessionPasswd=<null> context=0x20c8480 flags=0
  48. 2012-02-09 12:01:50,053:2606(0x7ff6d6c79700):ZOO_INFO@check_events@1585: initiated connection to server [172.16.0.2:2181]
  49. 2012-02-09 12:01:50,056:2606(0x7ff6d6c79700):ZOO_INFO@check_events@1632: session establishment complete on server [172.16.0.2:2181], sessionId=0x135626956b00006, negotiated timeout=10000
  50. 2012-02-09 12:01:50,056: hook.executor@DEBUG: started
  51. 2012-02-09 12:01:50,817: juju.agents.unit@DEBUG: Configuration Changed
  52. 2012-02-09 12:01:50,817: juju.agents.unit@DEBUG: Configuration updated on service in a non-started state
  53. 2012-02-09 12:01:50,817: statemachine@DEBUG: unitworkflowstate: transition state (None -> installed)
  54. 2012-02-09 12:01:50,817: statemachine@DEBUG: unitworkflowstate: transition install (None -> installed) {}
  55. 2012-02-09 12:01:50,817: statemachine@DEBUG: unitworkflowstate: execute action do_install
  56. 2012-02-09 12:01:50,818: hook.executor@DEBUG: Running hook: /var/lib/juju/units/namenode-0/charm/hooks/install
  57. 2012-02-09 12:01:50,825: hook.output@ERROR: + juju-log 'install script'
  58.  
  59. 2012-02-09 12:01:51,253: unit.hook.api@INFO: install script
  60. 2012-02-09 12:01:51,265: hook.output@ERROR: + export TERM=linux
  61. + TERM=linux
  62. + juju-log 'Adding ppa'
  63.  
  64. 2012-02-09 12:01:51,403: unit.hook.api@INFO: Adding ppa
  65. 2012-02-09 12:01:51,413: hook.output@ERROR: + apt-add-repository -y ppa:mark-mims/hadoop
  66.  
  67. 2012-02-09 12:01:51,941: hook.output@ERROR: Exception in thread Thread-1:
  68. Traceback (most recent call last):
  69. File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  70. self.run()
  71. File "/usr/lib/python2.7/dist-packages/softwareproperties/ppa.py", line 78, in run
  72. self.add_ppa_signing_key(self.ppa_path)
  73. File "/usr/lib/python2.7/dist-packages/softwareproperties/ppa.py", line 90, in add_ppa_signing_key
  74. print "Error reading %s: %s" % (lp_url, e)
  75. NameError: global name 'lp_url' is not defined
  76.  
  77.  
  78. 2012-02-09 12:01:51,954: hook.output@ERROR: + gpg --keyserver keyserver.ubuntu.com --recv 8564419E6DF5770B
  79.  
  80. 2012-02-09 12:01:51,958: hook.output@ERROR: gpg:
  81. 2012-02-09 12:01:51,958: hook.output@ERROR: directory `/root/.gnupg' created
  82.  
  83. 2012-02-09 12:01:51,959: hook.output@ERROR: gpg:
  84. 2012-02-09 12:01:51,960: hook.output@ERROR: new configuration file `/root/.gnupg/gpg.conf' created
  85.  
  86. 2012-02-09 12:01:51,960: hook.output@ERROR: gpg:
  87. 2012-02-09 12:01:51,960: hook.output@ERROR: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
  88.  
  89. 2012-02-09 12:01:51,961: hook.output@ERROR: gpg:
  90. 2012-02-09 12:01:51,961: hook.output@ERROR: keyring `/root/.gnupg/secring.gpg' created
  91.  
  92. 2012-02-09 12:01:51,961: hook.output@ERROR: gpg:
  93. 2012-02-09 12:01:51,962: hook.output@ERROR: keyring `/root/.gnupg/pubring.gpg' created
  94.  
  95. 2012-02-09 12:01:51,962: hook.output@ERROR: gpg:
  96. 2012-02-09 12:01:51,962: hook.output@ERROR: requesting key 6DF5770B from hkp server keyserver.ubuntu.com
  97.  
  98. 2012-02-09 12:01:52,397: hook.output@INFO: ?:
  99. 2012-02-09 12:01:52,397: hook.output@INFO: keyserver.ubuntu.com: No route to host
  100.  
  101. 2012-02-09 12:01:52,397: hook.output@INFO: gpgkeys: HTTP fetch error 7: couldn't connect: No route to host
  102.  
  103. 2012-02-09 12:01:52,398: hook.output@ERROR: gpg: no valid OpenPGP data found.
  104. gpg: Total number processed: 0
  105.  
  106. 2012-02-09 12:01:52,399: hook.output@ERROR: + sudo apt-key add -
  107.  
  108. 2012-02-09 12:01:52,402: hook.output@ERROR: + gpg --export --armor 8564419E6DF5770B
  109.  
  110. 2012-02-09 12:01:52,405: hook.output@ERROR: gpg:
  111. 2012-02-09 12:01:52,405: hook.output@ERROR: WARNING: nothing exported
  112.  
  113. 2012-02-09 12:01:52,427: hook.output@ERROR: gpg:
  114. 2012-02-09 12:01:52,427: hook.output@ERROR: no valid OpenPGP data found.
  115.  
  116. 2012-02-09 12:01:52,428: hook.output@ERROR: + juju-log 'updating cache'
  117.  
  118. 2012-02-09 12:01:52,566: unit.hook.api@INFO: updating cache
  119. 2012-02-09 12:01:52,577: hook.output@ERROR: + apt-get update
  120.  
  121. 2012-02-09 12:01:52,591: hook.output@INFO: Ign http://my.mirror... oneiric InRelease
  122.  
  123. 2012-02-09 12:01:52,591: hook.output@INFO: Ign http://my.mirror... oneiric-updates InRelease
  124.  
  125. .
  126. .
  127. .
  128.  
  129. 2012-02-09 12:01:52,802: hook.output@INFO: Ign http://ppa.launchpad.net oneiric InRelease
  130. Err http://ppa.launchpad.net oneiric Release.gpg
  131. Unable to connect to ppa.launchpad.net:http:
  132. Ign http://ppa.launchpad.net oneiric Release
  133. Ign http://ppa.launchpad.net oneiric/main TranslationIndex
  134. Err http://ppa.launchpad.net oneiric/main Sources
  135. Unable to connect to ppa.launchpad.net:http:
  136. Err http://ppa.launchpad.net oneiric/main amd64 Packages
  137. Unable to connect to ppa.launchpad.net:http:
  138. Err http://ppa.launchpad.net oneiric/main i386 Packages
  139. Unable to connect to ppa.launchpad.net:http:
  140. Err http://ppa.launchpad.net oneiric/main Translation-en
  141. Unable to connect to ppa.launchpad.net:http:
  142.  
  143. 2012-02-09 12:01:52,803: hook.output@ERROR: W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/Release.gpg Unable to connect to ppa.launchpad.net:http:
  144.  
  145. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/source/Sources Unable to connect to ppa.launchpad.net:http:
  146.  
  147. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/binary-amd64/Packages Unable to connect to ppa.launchpad.net:http:
  148.  
  149. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/binary-i386/Packages Unable to connect to ppa.launchpad.net:http:
  150.  
  151. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/i18n/Translation-en Unable to connect to ppa.launchpad.net:http:
  152.  
  153. E: Some index files failed to download. They have been ignored, or old ones used instead.
  154.  
  155. 2012-02-09 12:01:52,803: hook.output@ERROR: ++ hostname -f
  156.  
  157. 2012-02-09 12:01:52,805: hook.output@ERROR: + HOSTNAME=server-87.novalocal
  158.  
  159. 2012-02-09 12:01:52,805: hook.output@ERROR: ++ unit-get private-address
  160.  
  161. 2012-02-09 12:01:52,956: unit.hook.api@DEBUG: Get unit setting: 'private-address'
  162. 2012-02-09 12:01:52,969: hook.output@ERROR: + PRIVATE_ADDRESS=server-87.novalocal
  163. + juju-log 'Private Address: server-87.novalocal'
  164.  
  165. 2012-02-09 12:01:53,107: unit.hook.api@INFO: Private Address: server-87.novalocal
  166. 2012-02-09 12:01:53,118: hook.output@ERROR: + sed -i 's/127.0.1.1\(.*server-87.novalocal.*\)/server-87.novalocal\1/' /etc/hosts
  167.  
  168. 2012-02-09 12:01:53,120: hook.output@ERROR: + NAMENODE=server-87.novalocal
  169. + JOBTRACKER=server-87.novalocal
  170. + HDFSDATADIR=/var/lib/hadoop-0.20/dfs/data
  171. + juju-log 'Namenode: server-87.novalocal'
  172.  
  173. 2012-02-09 12:01:53,258: unit.hook.api@INFO: Namenode: server-87.novalocal
  174. 2012-02-09 12:01:53,268: hook.output@ERROR: + juju-log 'Jobtracker: server-87.novalocal'
  175.  
  176. 2012-02-09 12:01:53,406: unit.hook.api@INFO: Jobtracker: server-87.novalocal
  177. 2012-02-09 12:01:53,417: hook.output@ERROR: + juju-log 'HDFS Dir: /var/lib/hadoop-0.20/dfs/data'
  178.  
  179. 2012-02-09 12:01:53,554: unit.hook.api@INFO: HDFS Dir: /var/lib/hadoop-0.20/dfs/data
  180. 2012-02-09 12:01:53,565: hook.output@ERROR: + /usr/bin/debconf-set-selections
  181.  
  182. 2012-02-09 12:01:53,569: hook.output@ERROR: + echo debconf hadoop/namenode string server-87.novalocal
  183.  
  184. 2012-02-09 12:01:53,693: hook.output@ERROR: + /usr/bin/debconf-set-selections
  185.  
  186. 2012-02-09 12:01:53,697: hook.output@ERROR: + echo debconf hadoop/jobtracker string server-87.novalocal
  187.  
  188. 2012-02-09 12:01:53,806: hook.output@ERROR: + /usr/bin/debconf-set-selections
  189.  
  190. 2012-02-09 12:01:53,813: hook.output@ERROR: + echo debconf hadoop/hdfsdatadir string /var/lib/hadoop-0.20/dfs/data
  191.  
  192. 2012-02-09 12:01:53,918: hook.output@ERROR: + juju-log 'installing packages'
  193.  
  194. 2012-02-09 12:01:54,057: unit.hook.api@INFO: installing packages
  195. 2012-02-09 12:01:54,067: hook.output@ERROR: + apt-get install -y --force-yes hadoop-0.20-namenode
  196.  
  197. 2012-02-09 12:01:54,077: hook.output@INFO: Reading package lists...
  198. 2012-02-09 12:01:55,726: hook.output@INFO:
  199.  
  200. 2012-02-09 12:01:55,734: hook.output@INFO: Building dependency tree...
  201. 2012-02-09 12:01:55,928: hook.output@INFO:
  202.  
  203. 2012-02-09 12:01:55,928: hook.output@INFO: Reading state information...
  204. 2012-02-09 12:01:55,929: hook.output@INFO:
  205.  
  206. 2012-02-09 12:01:56,010: hook.output@ERROR: E
  207. 2012-02-09 12:01:56,010: hook.output@ERROR: :
  208. 2012-02-09 12:01:56,010: hook.output@ERROR: Unable to locate package hadoop-0.20-namenode
  209. 2012-02-09 12:01:56,011: hook.output@ERROR:
  210.  
  211. 2012-02-09 12:01:56,011: hook.output@ERROR: E
  212. 2012-02-09 12:01:56,011: hook.output@ERROR: :
  213. 2012-02-09 12:01:56,011: hook.output@ERROR: Couldn't find any package by regex 'hadoop-0.20-namenode'
  214. 2012-02-09 12:01:56,011: hook.output@ERROR:
  215.  
  216. 2012-02-09 12:01:56,012: hook.output@ERROR: + apt-get install -y --force-yes hadoop-0.20-jobtracker
  217.  
  218. 2012-02-09 12:01:56,016: hook.output@INFO: Reading package lists...
  219. 2012-02-09 12:01:56,018: hook.output@INFO:
  220.  
  221. 2012-02-09 12:01:56,022: hook.output@INFO: Building dependency tree...
  222. 2012-02-09 12:01:56,212: hook.output@INFO:
  223.  
  224. 2012-02-09 12:01:56,212: hook.output@INFO: Reading state information...
  225. 2012-02-09 12:01:56,213: hook.output@INFO:
  226.  
  227. 2012-02-09 12:01:56,294: hook.output@ERROR: E
  228. 2012-02-09 12:01:56,294: hook.output@ERROR: :
  229. 2012-02-09 12:01:56,295: hook.output@ERROR: Unable to locate package hadoop-0.20-jobtracker
  230. 2012-02-09 12:01:56,295: hook.output@ERROR:
  231.  
  232. 2012-02-09 12:01:56,295: hook.output@ERROR: E
  233. 2012-02-09 12:01:56,295: hook.output@ERROR: :
  234. 2012-02-09 12:01:56,295: hook.output@ERROR: Couldn't find any package by regex 'hadoop-0.20-jobtracker'
  235. 2012-02-09 12:01:56,295: hook.output@ERROR:
  236.  
  237. 2012-02-09 12:01:56,296: hook.output@ERROR: + '[' -x /usr/bin/open-port ']'
  238. + open-port 50010/TCP
  239.  
  240. 2012-02-09 12:01:56,444: unit.hook.api@DEBUG: opened 50010/tcp
  241. 2012-02-09 12:01:56,455: hook.output@ERROR: + open-port 50020/TCP
  242.  
  243. 2012-02-09 12:01:56,605: unit.hook.api@DEBUG: opened 50020/tcp
  244. 2012-02-09 12:01:56,616: hook.output@ERROR: + open-port 50030/TCP
  245.  
  246. 2012-02-09 12:01:56,767: unit.hook.api@DEBUG: opened 50030/tcp
  247. 2012-02-09 12:01:56,778: hook.output@ERROR: + open-port 50105/TCP
  248.  
  249. 2012-02-09 12:01:56,929: unit.hook.api@DEBUG: opened 50105/tcp
  250. 2012-02-09 12:01:56,940: hook.output@ERROR: + open-port 54310/TCP
  251.  
  252. 2012-02-09 12:01:57,091: unit.hook.api@DEBUG: opened 54310/tcp
  253. 2012-02-09 12:01:57,102: hook.output@ERROR: + open-port 54311/TCP
  254.  
  255. 2012-02-09 12:01:57,255: unit.hook.api@DEBUG: opened 54311/tcp
  256. 2012-02-09 12:01:57,267: hook.output@ERROR: + open-port 50060/TCP
  257.  
  258. 2012-02-09 12:01:57,420: unit.hook.api@DEBUG: opened 50060/tcp
  259. 2012-02-09 12:01:57,431: hook.output@ERROR: + open-port 50070/TCP
  260.  
  261. 2012-02-09 12:01:57,584: unit.hook.api@DEBUG: opened 50070/tcp
  262. 2012-02-09 12:01:57,595: hook.output@ERROR: + open-port 50075/TCP
  263.  
  264. 2012-02-09 12:01:57,584: unit.hook.api@DEBUG: opened 50070/tcp
  265. 2012-02-09 12:01:57,595: hook.output@ERROR: + open-port 50075/TCP
  266.  
  267. 2012-02-09 12:01:57,749: unit.hook.api@DEBUG: opened 50075/tcp
  268. 2012-02-09 12:01:57,760: hook.output@ERROR: + open-port 50090/TCP
  269.  
  270. 2012-02-09 12:01:57,914: unit.hook.api@DEBUG: opened 50090/tcp
  271. 2012-02-09 12:01:57,924: hook.output@ERROR: + open-port 8020/TCP
  272.  
  273. 2012-02-09 12:01:58,079: unit.hook.api@DEBUG: opened 8020/tcp
  274. 2012-02-09 12:01:58,090: hook.output@ERROR: + HADOOP_HOME=/usr/lib/hadoop-0.20
  275. + job_already_installed
  276. + [[ -f /usr/lib/hadoop-0.20/terasort.sh ]]
  277. + install_job
  278. + juju-log 'installing terasort script'
  279.  
  280. 2012-02-09 12:01:58,228: unit.hook.api@INFO: installing terasort script
  281. 2012-02-09 12:01:58,237: hook.output@ERROR: + cp ../scripts/terasort.sh /usr/lib/hadoop-0.20
  282.  
  283. 2012-02-09 12:01:58,238: hook.output@ERROR: cp:
  284. 2012-02-09 12:01:58,238: hook.output@ERROR: cannot stat `../scripts/terasort.sh'
  285. 2012-02-09 12:01:58,239: hook.output@ERROR: : No such file or directory
  286. 2012-02-09 12:01:58,239: hook.output@ERROR:
  287.  
  288. 2012-02-09 12:01:58,239: hook.output@ERROR: + chown hdfs.hdfs /usr/lib/hadoop-0.20/terasort.sh
  289.  
  290. 2012-02-09 12:01:58,240: hook.output@ERROR: chown:
  291. 2012-02-09 12:01:58,240: hook.output@ERROR: invalid user: `hdfs.hdfs'
  292. 2012-02-09 12:01:58,240: hook.output@ERROR:
  293.  
  294. 2012-02-09 12:01:58,240: hook.output@ERROR: + chmod 755 /usr/lib/hadoop-0.20/terasort.sh
  295.  
  296. 2012-02-09 12:01:58,241: hook.output@ERROR: chmod:
  297. 2012-02-09 12:01:58,241: hook.output@ERROR: cannot access `/usr/lib/hadoop-0.20/terasort.sh'
  298. 2012-02-09 12:01:58,241: hook.output@ERROR: : No such file or directory
  299. 2012-02-09 12:01:58,241: hook.output@ERROR:
  300.  
  301. 2012-02-09 12:01:58,242: hook.output@DEBUG: hook install exited, exit code Traceback (most recent call last):
  302. Failure: juju.errors.CharmInvocationError: Error processing '/var/lib/juju/units/namenode-0/charm/hooks/install': exit code 1.
  303. .
  304. 2012-02-09 12:01:58,242: hook.executor@DEBUG: Hook error: /var/lib/juju/units/namenode-0/charm/hooks/install Error processing '/var/lib/juju/units/namenode-0/charm/hooks/install': exit code 1.
  305. 2012-02-09 12:01:58,245: statemachine@DEBUG: unitworkflowstate: executing error transition error_install, Error processing '/var/lib/juju/units/namenode-0/charm/hooks/install': exit code 1.
  306. 2012-02-09 12:01:58,245: statemachine@DEBUG: unitworkflowstate: transition error_install (None -> install_error) {}
  307. 2012-02-09 12:01:58,250: statemachine@DEBUG: unitworkflowstate: transition complete error_install (state install_error) {}
  308.  
  309. ___________________________
  310.  
  311.  
  312. So, I've tried https://juju.ubuntu.com/docs/write-charm.html#debugging-hooks to debug. I could connect to the instance and run the install script from charm manually.
  313.  
  314. $ juju debug-hooks namenode/0 (open a shell at the instance)
  315. $ juju resolved --retry namenode/0 (change the shell to install step)
  316.  
  317. So,
  318.  
  319. $ cd /var/lib/juju/units/namenode-0/charm/hooks/
  320. $ ./install
  321.  
  322. and ALL works great!!
  323.  
  324. $ exit
  325. $ exit
  326.  
  327. jorge@s:~/charms/oneiric/hadoop-master/hooks$ juju status
  328. 2012-02-09 12:12:39,562 INFO Connecting to environment.
  329. machines:
  330. 0: {dns-name: 172.16.0.2, instance-id: i-00000056}
  331. 1: {dns-name: 172.16.0.3, instance-id: i-00000057}
  332. services:
  333. namenode:
  334. charm: local:oneiric/hadoop-master-3
  335. relations: {}
  336. units:
  337. namenode/0:
  338. machine: 1
  339. public-address: server-87.novalocal
  340. relations: {}
  341. state: started
  342. 2012-02-09 12:12:40,362 INFO 'status' command finished successfully
  343.  
  344.  
  345. Now, state is started!
  346.  
  347.  
  348. Doubts:
  349.  
  350. What could be happening?
  351. How does juju run the install script? If I run the same install script in a common shell, all work fine!
  352.  
  353. I don't know if the problem could be with the charms.
  354.  
  355. Please, any help is appreciated!
RAW Paste Data