API tools faq
paste
askb

fuel deploy failed thrid time:

askb
Mar 23rd, 2016
62
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 8.86 KB | None | 0 0
raw download clone embed print report
  1.  
  2. Error
  3. Provision has failed. Failed to execute hook 'shell' command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz", "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": [{"name": "ubuntu", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": "ubuntu-security", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-security", "type": "deb"}, {"name": "mos", "section": "main restricted", "uri": "http://10.20.0.2:8080/liberty-8.0/ubuntu/x86_64", "priority": 1050, "suite": "mos8.0", "type": "deb"}, {"name": "mos-updates", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/", "priority": 1050, "suite": "mos8.0-updates", "type": "deb"}, {"name": "mos-security", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/", "priority": 1050, "suite": "mos8.0-security", "type": "deb"}, {"name": "mos-holdback", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/", "priority": 1100, "suite": "mos8.0-holdback", "type": "deb"}, {"name": "Auxiliary", "section": "main restricted", "uri": "http://10.20.0.2:8080/liberty-8.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", "type": "deb"}], "codename": "trusty"}'
  4.  
  5. Task: 8bc04b2f-f522-4145-8470-f06bb5c1f3d2: shell timeout error: execution expired
  6. Task timeout: 3600, Retries: 1
  7.  
  8. Later encountered the following errors in astute logs:
  9. Deployment has failed. Method granular_deploy. Deployment failed on nodes 3.
  10. Inspect Astute logs for the details
  11.  
  12. 2016-03-24 02:57:00 ERROR [734] Error running RPC method granular_deploy: Deployment failed on nodes 3, trace:
  13. ["/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
  14. "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
  15. "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
  16. "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `each'",
  17. "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
  18. "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `each'",
  19. "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",
  20. "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:216:in `deploy_cluster'",
  21. "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:52:in `granular_deploy'",
  22. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:92:in `granular_deploy'",
  23. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'",
  24. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'",
  25. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
  26. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
  27. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
  28. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
  29. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'",
  30. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'",
  31. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]
  32. 2016-03-24 02:57:00 ERROR [734] No more tasks will be executed on the node 3
  33. 2016-03-24 02:57:00 ERROR [734] Task '{"priority"=>5200, "type"=>"puppet", "id"=>"glance", "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/glance/glance.pp", "timeout"=>3600, "cwd"=>"/"}, "uids"=>["3"]}' failed on node 3
  34. 2016-03-23 17:29:20 ERROR [754] Error running provisioning: Image build task failed. Please check build log here for details: /var/log/docker-logs/fuel-agent-env-1.log. Hint: restart deployment can help if no error in build log was found, trace:
  35. ["/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:109:in `rescue in block in provision'",
  36. "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:102:in `block in provision'",
  37. "/usr/share/gems/gems/astute-8.0.0/lib/astute/provision.rb:288:in `call'",
  38. "/usr/share/gems/gems/astute-8.0.0/lib/astute/provision.rb:288:in `report_image_provision'",
  39. "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:96:in `provision'",
  40. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:51:in `provision'",
  41. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:37:in `image_provision'",
  42. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'",
  43. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'",
  44. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
  45. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
  46. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
  47. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
  48. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'",
  49. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'",
  50. "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]
  51. 2016-03-23 17:29:20 ERROR [754] command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz", "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": [{"name": "ubuntu", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": "ubuntu-security", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-security", "type": "deb"}, {"name": "mos", "section": "main restricted", "uri": "http://10.20.0.2:8080/liberty-8.0/ubuntu/x86_64", "priority": 1050, "suite": "mos8.0", "type": "deb"}, {"name": "mos-updates", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/", "priority": 1050, "suite": "mos8.0-updates", "type": "deb"}, {"name": "mos-security", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/", "priority": 1050, "suite": "mos8.0-security", "type": "deb"}, {"name": "mos-holdback", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/", "priority": 1100, "suite": "mos8.0-holdback", "type": "deb"}, {"name": "Auxiliary", "section": "main restricted", "uri": "http://10.20.0.2:8080/liberty-8.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", "type": "deb"}], "codename": "trusty"}'
  52. Task: 8bc04b2f-f522-4145-8470-f06bb5c1f3d2: shell timeout error: execution expired
  53. Task timeout: 3600, Retries: 1
  54.  
  55. http://comments.gmane.org/gmane.comp.cloud.openstack.general/12550
  56. https://bugs.launchpad.net/fuel/+bug/1528498/
  57.  
  58. >>>
  59. This is potentially happening because of time outs (3600 seconds).
  60.  
  61. A) There is a timer which runs for 3600 sec and this timer expires after an hour of you starting the process but the image creation task does not get completed by then ,so when the FUEL master nodes starts looking for the image to install on the slave nodes it does not find them and hence the error .
  62.  
  63. When you hit the error, don't do anything. Don't delete the environment. Wait for 40-60 mins and just hit deploy button again. I did same and in middle of deployment again. Fingers crossed this time :D.
  64.  
  65. B) Or it may be bug as well as described below. This is mainly because of older docker version.
  66.  
  67. https://bugs.launchpad.net/fuel/+bug/1528498/comments/8
  68.  
  69. There is work around for that as well, as mentioned in comment above
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!