API tools faq
paste
Advertisement
Guest User

Juju problem with hadoop charms

a guest
Feb 9th, 2012
149
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 17.01 KB | None | 0 0
raw download clone embed print report
  1. Hi all! I need some help to deploy hadoop charms. I would like to use hadoop-master and hadoop-slave formulas from https://code.launchpad.net/~charmers/. Apparently all is working fine with my juju because I can use the mysql and mediawiki charms normally. But, with the hadoop charms the problem is that the service stops at some point.
  2.  
  3. A interesting thing is that we can debug a juju deployment using some commands. According to juju docs, we can go step by step in the deployment process (install, config_change, start etc). Below, a sequence of the facts.
  4.  
  5. jorge@s:~/charms$ juju bootstrap
  6. 2012-02-09 11:58:45,765 INFO Bootstrapping environment 'sample' (type: ec2)...
  7. 2012-02-09 11:58:47,454 INFO 'bootstrap' command finished successfully
  8.  
  9. jorge@s:~/charms$ juju deploy --repository ~/charms local:hadoop-master namenode
  10. 2012-02-09 12:01:06,649 INFO Connecting to environment.
  11. 2012-02-09 12:01:07,289 INFO Charm deployed as service: 'namenode'
  12. 2012-02-09 12:01:07,290 INFO 'deploy' command finished successfully
  13.  
  14. Juju returns some succesful messages, but running "juju status" I get:
  15.  
  16. jorge@sold049:~/charms$ juju status
  17. 2012-02-09 12:02:08,958 INFO Connecting to environment.
  18. machines:
  19. 0: {dns-name: 172.16.0.2, instance-id: i-00000056}
  20. 1: {dns-name: 172.16.0.3, instance-id: i-00000057}
  21. services:
  22. namenode:
  23. charm: local:oneiric/hadoop-master-3
  24. relations: {}
  25. units:
  26. namenode/0:
  27. machine: 1
  28. public-address: server-87.novalocal
  29. relations: {}
  30. state: install_error
  31. 2012-02-09 12:02:09,765 INFO 'status' command finished successfully
  32.  
  33. The state is install_error!!
  34.  
  35. Before debug, here is the logs from /var/lib/juju/units/namenode-0/charm.log:
  36.  
  37. ____________________
  38.  
  39. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.3
  40. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@662: Client environment:host.name=server-87
  41. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@669: Client environment:os.name=Linux
  42. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@670: Client environment:os.arch=3.0.0-12-server
  43. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@671: Client environment:os.version=#20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011
  44. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@679: Client environment:user.name=(null)
  45. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@687: Client environment:user.home=/root
  46. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@log_env@699: Client environment:user.dir=/
  47. 2012-02-09 12:01:50,052:2606(0x7ff6db1dc720):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=172.16.0.2:2181 sessionTimeout=10000 watcher=0x7ff6d8328650 sessionId=0 sessionPasswd=<null> context=0x20c8480 flags=0
  48. 2012-02-09 12:01:50,053:2606(0x7ff6d6c79700):ZOO_INFO@check_events@1585: initiated connection to server [172.16.0.2:2181]
  49. 2012-02-09 12:01:50,056:2606(0x7ff6d6c79700):ZOO_INFO@check_events@1632: session establishment complete on server [172.16.0.2:2181], sessionId=0x135626956b00006, negotiated timeout=10000
  50. 2012-02-09 12:01:50,056: hook.executor@DEBUG: started
  51. 2012-02-09 12:01:50,817: juju.agents.unit@DEBUG: Configuration Changed
  52. 2012-02-09 12:01:50,817: juju.agents.unit@DEBUG: Configuration updated on service in a non-started state
  53. 2012-02-09 12:01:50,817: statemachine@DEBUG: unitworkflowstate: transition state (None -> installed)
  54. 2012-02-09 12:01:50,817: statemachine@DEBUG: unitworkflowstate: transition install (None -> installed) {}
  55. 2012-02-09 12:01:50,817: statemachine@DEBUG: unitworkflowstate: execute action do_install
  56. 2012-02-09 12:01:50,818: hook.executor@DEBUG: Running hook: /var/lib/juju/units/namenode-0/charm/hooks/install
  57. 2012-02-09 12:01:50,825: hook.output@ERROR: + juju-log 'install script'
  58.  
  59. 2012-02-09 12:01:51,253: unit.hook.api@INFO: install script
  60. 2012-02-09 12:01:51,265: hook.output@ERROR: + export TERM=linux
  61. + TERM=linux
  62. + juju-log 'Adding ppa'
  63.  
  64. 2012-02-09 12:01:51,403: unit.hook.api@INFO: Adding ppa
  65. 2012-02-09 12:01:51,413: hook.output@ERROR: + apt-add-repository -y ppa:mark-mims/hadoop
  66.  
  67. 2012-02-09 12:01:51,941: hook.output@ERROR: Exception in thread Thread-1:
  68. Traceback (most recent call last):
  69. File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  70. self.run()
  71. File "/usr/lib/python2.7/dist-packages/softwareproperties/ppa.py", line 78, in run
  72. self.add_ppa_signing_key(self.ppa_path)
  73. File "/usr/lib/python2.7/dist-packages/softwareproperties/ppa.py", line 90, in add_ppa_signing_key
  74. print "Error reading %s: %s" % (lp_url, e)
  75. NameError: global name 'lp_url' is not defined
  76.  
  77.  
  78. 2012-02-09 12:01:51,954: hook.output@ERROR: + gpg --keyserver keyserver.ubuntu.com --recv 8564419E6DF5770B
  79.  
  80. 2012-02-09 12:01:51,958: hook.output@ERROR: gpg:
  81. 2012-02-09 12:01:51,958: hook.output@ERROR: directory `/root/.gnupg' created
  82.  
  83. 2012-02-09 12:01:51,959: hook.output@ERROR: gpg:
  84. 2012-02-09 12:01:51,960: hook.output@ERROR: new configuration file `/root/.gnupg/gpg.conf' created
  85.  
  86. 2012-02-09 12:01:51,960: hook.output@ERROR: gpg:
  87. 2012-02-09 12:01:51,960: hook.output@ERROR: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
  88.  
  89. 2012-02-09 12:01:51,961: hook.output@ERROR: gpg:
  90. 2012-02-09 12:01:51,961: hook.output@ERROR: keyring `/root/.gnupg/secring.gpg' created
  91.  
  92. 2012-02-09 12:01:51,961: hook.output@ERROR: gpg:
  93. 2012-02-09 12:01:51,962: hook.output@ERROR: keyring `/root/.gnupg/pubring.gpg' created
  94.  
  95. 2012-02-09 12:01:51,962: hook.output@ERROR: gpg:
  96. 2012-02-09 12:01:51,962: hook.output@ERROR: requesting key 6DF5770B from hkp server keyserver.ubuntu.com
  97.  
  98. 2012-02-09 12:01:52,397: hook.output@INFO: ?:
  99. 2012-02-09 12:01:52,397: hook.output@INFO: keyserver.ubuntu.com: No route to host
  100.  
  101. 2012-02-09 12:01:52,397: hook.output@INFO: gpgkeys: HTTP fetch error 7: couldn't connect: No route to host
  102.  
  103. 2012-02-09 12:01:52,398: hook.output@ERROR: gpg: no valid OpenPGP data found.
  104. gpg: Total number processed: 0
  105.  
  106. 2012-02-09 12:01:52,399: hook.output@ERROR: + sudo apt-key add -
  107.  
  108. 2012-02-09 12:01:52,402: hook.output@ERROR: + gpg --export --armor 8564419E6DF5770B
  109.  
  110. 2012-02-09 12:01:52,405: hook.output@ERROR: gpg:
  111. 2012-02-09 12:01:52,405: hook.output@ERROR: WARNING: nothing exported
  112.  
  113. 2012-02-09 12:01:52,427: hook.output@ERROR: gpg:
  114. 2012-02-09 12:01:52,427: hook.output@ERROR: no valid OpenPGP data found.
  115.  
  116. 2012-02-09 12:01:52,428: hook.output@ERROR: + juju-log 'updating cache'
  117.  
  118. 2012-02-09 12:01:52,566: unit.hook.api@INFO: updating cache
  119. 2012-02-09 12:01:52,577: hook.output@ERROR: + apt-get update
  120.  
  121. 2012-02-09 12:01:52,591: hook.output@INFO: Ign http://my.mirror... oneiric InRelease
  122.  
  123. 2012-02-09 12:01:52,591: hook.output@INFO: Ign http://my.mirror... oneiric-updates InRelease
  124.  
  125. .
  126. .
  127. .
  128.  
  129. 2012-02-09 12:01:52,802: hook.output@INFO: Ign http://ppa.launchpad.net oneiric InRelease
  130. Err http://ppa.launchpad.net oneiric Release.gpg
  131. Unable to connect to ppa.launchpad.net:http:
  132. Ign http://ppa.launchpad.net oneiric Release
  133. Ign http://ppa.launchpad.net oneiric/main TranslationIndex
  134. Err http://ppa.launchpad.net oneiric/main Sources
  135. Unable to connect to ppa.launchpad.net:http:
  136. Err http://ppa.launchpad.net oneiric/main amd64 Packages
  137. Unable to connect to ppa.launchpad.net:http:
  138. Err http://ppa.launchpad.net oneiric/main i386 Packages
  139. Unable to connect to ppa.launchpad.net:http:
  140. Err http://ppa.launchpad.net oneiric/main Translation-en
  141. Unable to connect to ppa.launchpad.net:http:
  142.  
  143. 2012-02-09 12:01:52,803: hook.output@ERROR: W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/Release.gpg Unable to connect to ppa.launchpad.net:http:
  144.  
  145. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/source/Sources Unable to connect to ppa.launchpad.net:http:
  146.  
  147. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/binary-amd64/Packages Unable to connect to ppa.launchpad.net:http:
  148.  
  149. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/binary-i386/Packages Unable to connect to ppa.launchpad.net:http:
  150.  
  151. W: Failed to fetch http://ppa.launchpad.net/mark-mims/hadoop/ubuntu/dists/oneiric/main/i18n/Translation-en Unable to connect to ppa.launchpad.net:http:
  152.  
  153. E: Some index files failed to download. They have been ignored, or old ones used instead.
  154.  
  155. 2012-02-09 12:01:52,803: hook.output@ERROR: ++ hostname -f
  156.  
  157. 2012-02-09 12:01:52,805: hook.output@ERROR: + HOSTNAME=server-87.novalocal
  158.  
  159. 2012-02-09 12:01:52,805: hook.output@ERROR: ++ unit-get private-address
  160.  
  161. 2012-02-09 12:01:52,956: unit.hook.api@DEBUG: Get unit setting: 'private-address'
  162. 2012-02-09 12:01:52,969: hook.output@ERROR: + PRIVATE_ADDRESS=server-87.novalocal
  163. + juju-log 'Private Address: server-87.novalocal'
  164.  
  165. 2012-02-09 12:01:53,107: unit.hook.api@INFO: Private Address: server-87.novalocal
  166. 2012-02-09 12:01:53,118: hook.output@ERROR: + sed -i 's/127.0.1.1\(.*server-87.novalocal.*\)/server-87.novalocal\1/' /etc/hosts
  167.  
  168. 2012-02-09 12:01:53,120: hook.output@ERROR: + NAMENODE=server-87.novalocal
  169. + JOBTRACKER=server-87.novalocal
  170. + HDFSDATADIR=/var/lib/hadoop-0.20/dfs/data
  171. + juju-log 'Namenode: server-87.novalocal'
  172.  
  173. 2012-02-09 12:01:53,258: unit.hook.api@INFO: Namenode: server-87.novalocal
  174. 2012-02-09 12:01:53,268: hook.output@ERROR: + juju-log 'Jobtracker: server-87.novalocal'
  175.  
  176. 2012-02-09 12:01:53,406: unit.hook.api@INFO: Jobtracker: server-87.novalocal
  177. 2012-02-09 12:01:53,417: hook.output@ERROR: + juju-log 'HDFS Dir: /var/lib/hadoop-0.20/dfs/data'
  178.  
  179. 2012-02-09 12:01:53,554: unit.hook.api@INFO: HDFS Dir: /var/lib/hadoop-0.20/dfs/data
  180. 2012-02-09 12:01:53,565: hook.output@ERROR: + /usr/bin/debconf-set-selections
  181.  
  182. 2012-02-09 12:01:53,569: hook.output@ERROR: + echo debconf hadoop/namenode string server-87.novalocal
  183.  
  184. 2012-02-09 12:01:53,693: hook.output@ERROR: + /usr/bin/debconf-set-selections
  185.  
  186. 2012-02-09 12:01:53,697: hook.output@ERROR: + echo debconf hadoop/jobtracker string server-87.novalocal
  187.  
  188. 2012-02-09 12:01:53,806: hook.output@ERROR: + /usr/bin/debconf-set-selections
  189.  
  190. 2012-02-09 12:01:53,813: hook.output@ERROR: + echo debconf hadoop/hdfsdatadir string /var/lib/hadoop-0.20/dfs/data
  191.  
  192. 2012-02-09 12:01:53,918: hook.output@ERROR: + juju-log 'installing packages'
  193.  
  194. 2012-02-09 12:01:54,057: unit.hook.api@INFO: installing packages
  195. 2012-02-09 12:01:54,067: hook.output@ERROR: + apt-get install -y --force-yes hadoop-0.20-namenode
  196.  
  197. 2012-02-09 12:01:54,077: hook.output@INFO: Reading package lists...
  198. 2012-02-09 12:01:55,726: hook.output@INFO:
  199.  
  200. 2012-02-09 12:01:55,734: hook.output@INFO: Building dependency tree...
  201. 2012-02-09 12:01:55,928: hook.output@INFO:
  202.  
  203. 2012-02-09 12:01:55,928: hook.output@INFO: Reading state information...
  204. 2012-02-09 12:01:55,929: hook.output@INFO:
  205.  
  206. 2012-02-09 12:01:56,010: hook.output@ERROR: E
  207. 2012-02-09 12:01:56,010: hook.output@ERROR: :
  208. 2012-02-09 12:01:56,010: hook.output@ERROR: Unable to locate package hadoop-0.20-namenode
  209. 2012-02-09 12:01:56,011: hook.output@ERROR:
  210.  
  211. 2012-02-09 12:01:56,011: hook.output@ERROR: E
  212. 2012-02-09 12:01:56,011: hook.output@ERROR: :
  213. 2012-02-09 12:01:56,011: hook.output@ERROR: Couldn't find any package by regex 'hadoop-0.20-namenode'
  214. 2012-02-09 12:01:56,011: hook.output@ERROR:
  215.  
  216. 2012-02-09 12:01:56,012: hook.output@ERROR: + apt-get install -y --force-yes hadoop-0.20-jobtracker
  217.  
  218. 2012-02-09 12:01:56,016: hook.output@INFO: Reading package lists...
  219. 2012-02-09 12:01:56,018: hook.output@INFO:
  220.  
  221. 2012-02-09 12:01:56,022: hook.output@INFO: Building dependency tree...
  222. 2012-02-09 12:01:56,212: hook.output@INFO:
  223.  
  224. 2012-02-09 12:01:56,212: hook.output@INFO: Reading state information...
  225. 2012-02-09 12:01:56,213: hook.output@INFO:
  226.  
  227. 2012-02-09 12:01:56,294: hook.output@ERROR: E
  228. 2012-02-09 12:01:56,294: hook.output@ERROR: :
  229. 2012-02-09 12:01:56,295: hook.output@ERROR: Unable to locate package hadoop-0.20-jobtracker
  230. 2012-02-09 12:01:56,295: hook.output@ERROR:
  231.  
  232. 2012-02-09 12:01:56,295: hook.output@ERROR: E
  233. 2012-02-09 12:01:56,295: hook.output@ERROR: :
  234. 2012-02-09 12:01:56,295: hook.output@ERROR: Couldn't find any package by regex 'hadoop-0.20-jobtracker'
  235. 2012-02-09 12:01:56,295: hook.output@ERROR:
  236.  
  237. 2012-02-09 12:01:56,296: hook.output@ERROR: + '[' -x /usr/bin/open-port ']'
  238. + open-port 50010/TCP
  239.  
  240. 2012-02-09 12:01:56,444: unit.hook.api@DEBUG: opened 50010/tcp
  241. 2012-02-09 12:01:56,455: hook.output@ERROR: + open-port 50020/TCP
  242.  
  243. 2012-02-09 12:01:56,605: unit.hook.api@DEBUG: opened 50020/tcp
  244. 2012-02-09 12:01:56,616: hook.output@ERROR: + open-port 50030/TCP
  245.  
  246. 2012-02-09 12:01:56,767: unit.hook.api@DEBUG: opened 50030/tcp
  247. 2012-02-09 12:01:56,778: hook.output@ERROR: + open-port 50105/TCP
  248.  
  249. 2012-02-09 12:01:56,929: unit.hook.api@DEBUG: opened 50105/tcp
  250. 2012-02-09 12:01:56,940: hook.output@ERROR: + open-port 54310/TCP
  251.  
  252. 2012-02-09 12:01:57,091: unit.hook.api@DEBUG: opened 54310/tcp
  253. 2012-02-09 12:01:57,102: hook.output@ERROR: + open-port 54311/TCP
  254.  
  255. 2012-02-09 12:01:57,255: unit.hook.api@DEBUG: opened 54311/tcp
  256. 2012-02-09 12:01:57,267: hook.output@ERROR: + open-port 50060/TCP
  257.  
  258. 2012-02-09 12:01:57,420: unit.hook.api@DEBUG: opened 50060/tcp
  259. 2012-02-09 12:01:57,431: hook.output@ERROR: + open-port 50070/TCP
  260.  
  261. 2012-02-09 12:01:57,584: unit.hook.api@DEBUG: opened 50070/tcp
  262. 2012-02-09 12:01:57,595: hook.output@ERROR: + open-port 50075/TCP
  263.  
  264. 2012-02-09 12:01:57,584: unit.hook.api@DEBUG: opened 50070/tcp
  265. 2012-02-09 12:01:57,595: hook.output@ERROR: + open-port 50075/TCP
  266.  
  267. 2012-02-09 12:01:57,749: unit.hook.api@DEBUG: opened 50075/tcp
  268. 2012-02-09 12:01:57,760: hook.output@ERROR: + open-port 50090/TCP
  269.  
  270. 2012-02-09 12:01:57,914: unit.hook.api@DEBUG: opened 50090/tcp
  271. 2012-02-09 12:01:57,924: hook.output@ERROR: + open-port 8020/TCP
  272.  
  273. 2012-02-09 12:01:58,079: unit.hook.api@DEBUG: opened 8020/tcp
  274. 2012-02-09 12:01:58,090: hook.output@ERROR: + HADOOP_HOME=/usr/lib/hadoop-0.20
  275. + job_already_installed
  276. + [[ -f /usr/lib/hadoop-0.20/terasort.sh ]]
  277. + install_job
  278. + juju-log 'installing terasort script'
  279.  
  280. 2012-02-09 12:01:58,228: unit.hook.api@INFO: installing terasort script
  281. 2012-02-09 12:01:58,237: hook.output@ERROR: + cp ../scripts/terasort.sh /usr/lib/hadoop-0.20
  282.  
  283. 2012-02-09 12:01:58,238: hook.output@ERROR: cp:
  284. 2012-02-09 12:01:58,238: hook.output@ERROR: cannot stat `../scripts/terasort.sh'
  285. 2012-02-09 12:01:58,239: hook.output@ERROR: : No such file or directory
  286. 2012-02-09 12:01:58,239: hook.output@ERROR:
  287.  
  288. 2012-02-09 12:01:58,239: hook.output@ERROR: + chown hdfs.hdfs /usr/lib/hadoop-0.20/terasort.sh
  289.  
  290. 2012-02-09 12:01:58,240: hook.output@ERROR: chown:
  291. 2012-02-09 12:01:58,240: hook.output@ERROR: invalid user: `hdfs.hdfs'
  292. 2012-02-09 12:01:58,240: hook.output@ERROR:
  293.  
  294. 2012-02-09 12:01:58,240: hook.output@ERROR: + chmod 755 /usr/lib/hadoop-0.20/terasort.sh
  295.  
  296. 2012-02-09 12:01:58,241: hook.output@ERROR: chmod:
  297. 2012-02-09 12:01:58,241: hook.output@ERROR: cannot access `/usr/lib/hadoop-0.20/terasort.sh'
  298. 2012-02-09 12:01:58,241: hook.output@ERROR: : No such file or directory
  299. 2012-02-09 12:01:58,241: hook.output@ERROR:
  300.  
  301. 2012-02-09 12:01:58,242: hook.output@DEBUG: hook install exited, exit code Traceback (most recent call last):
  302. Failure: juju.errors.CharmInvocationError: Error processing '/var/lib/juju/units/namenode-0/charm/hooks/install': exit code 1.
  303. .
  304. 2012-02-09 12:01:58,242: hook.executor@DEBUG: Hook error: /var/lib/juju/units/namenode-0/charm/hooks/install Error processing '/var/lib/juju/units/namenode-0/charm/hooks/install': exit code 1.
  305. 2012-02-09 12:01:58,245: statemachine@DEBUG: unitworkflowstate: executing error transition error_install, Error processing '/var/lib/juju/units/namenode-0/charm/hooks/install': exit code 1.
  306. 2012-02-09 12:01:58,245: statemachine@DEBUG: unitworkflowstate: transition error_install (None -> install_error) {}
  307. 2012-02-09 12:01:58,250: statemachine@DEBUG: unitworkflowstate: transition complete error_install (state install_error) {}
  308.  
  309. ___________________________
  310.  
  311.  
  312. So, I've tried https://juju.ubuntu.com/docs/write-charm.html#debugging-hooks to debug. I could connect to the instance and run the install script from charm manually.
  313.  
  314. $ juju debug-hooks namenode/0 (open a shell at the instance)
  315. $ juju resolved --retry namenode/0 (change the shell to install step)
  316.  
  317. So,
  318.  
  319. $ cd /var/lib/juju/units/namenode-0/charm/hooks/
  320. $ ./install
  321.  
  322. and ALL works great!!
  323.  
  324. $ exit
  325. $ exit
  326.  
  327. jorge@s:~/charms/oneiric/hadoop-master/hooks$ juju status
  328. 2012-02-09 12:12:39,562 INFO Connecting to environment.
  329. machines:
  330. 0: {dns-name: 172.16.0.2, instance-id: i-00000056}
  331. 1: {dns-name: 172.16.0.3, instance-id: i-00000057}
  332. services:
  333. namenode:
  334. charm: local:oneiric/hadoop-master-3
  335. relations: {}
  336. units:
  337. namenode/0:
  338. machine: 1
  339. public-address: server-87.novalocal
  340. relations: {}
  341. state: started
  342. 2012-02-09 12:12:40,362 INFO 'status' command finished successfully
  343.  
  344.  
  345. Now, state is started!
  346.  
  347.  
  348. Doubts:
  349.  
  350. What could be happening?
  351. How does juju run the install script? If I run the same install script in a common shell, all work fine!
  352.  
  353. I don't know if the problem could be with the charms.
  354.  
  355. Please, any help is appreciated!
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!