API tools faq
paste
Advertisement
Guest User

Openstack QPID Exception

a guest
Feb 14th, 2013
197
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 51.90 KB | None | 0 0
raw download clone embed print report
  1. 2013-02-01 00:23:35 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=22224) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:172
  2. 2013-02-01 00:23:35 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=22224) _publish_service_capabilities /usr/lib/python2.6/site-packages/nova/manager.py:231
  3. 2013-02-01 00:23:35 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous fanout cast... from (pid=22224) fanout_cast /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:384
  4. 2013-02-01 00:23:35 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=22224) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:172
  5. 2013-02-01 00:23:35 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=22224) _publish_service_capabilities /usr/lib/python2.6/site-packages/nova/manager.py:231
  6. 2013-02-01 00:23:35 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous fanout cast... from (pid=22224) fanout_cast /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:384
  7. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ ensure EXCEPTION Traceback (most recent call last):
  8. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 382, in ensure
  9. return method(*args, **kwargs)
  10. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 438, in _consume
  11. nxt_receiver = self.session.next_receiver(timeout=timeout)
  12. File "<string>", line 6, in next_receiver
  13. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
  14. raise Empty
  15. Empty: None
  16.  
  17. 2013-02-01 00:23:35 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None
  18. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):
  19. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 382, in ensure
  20. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)
  21. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 438, in _consume
  22. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)
  23. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver
  24. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
  25. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid raise Empty
  26. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid Empty: None
  27. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid
  28. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ reset
  29. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ cancel_consumer_thread
  30. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ ensure EXCEPTION Traceback (most recent call last):
  31. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 382, in ensure
  32. return method(*args, **kwargs)
  33. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 438, in _consume
  34. nxt_receiver = self.session.next_receiver(timeout=timeout)
  35. File "<string>", line 6, in next_receiver
  36. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
  37. raise Empty
  38. Empty: None
  39.  
  40. 2013-02-01 00:23:35 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None
  41. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):
  42. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 382, in ensure
  43. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)
  44. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 438, in _consume
  45. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)
  46. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver
  47. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
  48. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid raise Empty
  49. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid Empty: None
  50. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid
  51. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ reset
  52. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ cancel_consumer_thread
  53. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ ensure EXCEPTION Traceback (most recent call last):
  54. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 382, in ensure
  55. return method(*args, **kwargs)
  56. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 438, in _consume
  57. nxt_receiver = self.session.next_receiver(timeout=timeout)
  58. File "<string>", line 6, in next_receiver
  59. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
  60. raise Empty
  61. Empty: None
  62.  
  63. 2013-02-01 00:23:35 ERROR nova.openstack.common.rpc.impl_qpid [-] Timed out waiting for RPC response: None
  64. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid Traceback (most recent call last):
  65. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 382, in ensure
  66. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid return method(*args, **kwargs)
  67. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 438, in _consume
  68. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout)
  69. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver
  70. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 663, in next_receiver
  71. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid raise Empty
  72. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid Empty: None
  73. 2013-02-01 00:23:35 TRACE nova.openstack.common.rpc.impl_qpid
  74. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ reset
  75. 2013-02-01 00:23:35 INFO nova.openstack.common.rpc.impl_qpid [-] prc_ cancel_consumer_thread
  76. 2013-02-01 00:23:35 DEBUG nova.manager [-] Running periodic task ComputeManager._check_instance_build_time from (pid=22224) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:172
  77. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-f1d422d8-5036-451e-bf3d-1913dc06badd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] < 200: {u'virtualServersUri': u'http://localhost:8080/cloud/api/workloads/793816/virtualServers', u'cloudGroupId': u'18851', u'name': u'cb-yongwen-OPSTACK-vm715-tinyvm', u'timestampsUri': u'http://localhost:8080/cloud/api/workloads/793816/timestamps', u'cloudGroupName': u'skcm2', u'hypervisor': u'VMware', u'description': u'cb-yongwen-OPSTACK-vm715-tinyvm', u'cloudId': u'cloud://18851/a9d35dbf-f897-4f68-baf3-677a9ef51bee', u'uri': u'http://localhost:8080/cloud/api/workloads/793816', u'targetUri': u'http://localhost:8080/cloud/api/workloads/793816/target', u'state': {u'id': u'EXECUTING', u'label': u'Deploying'}, u'customizationUri': u'http://localhost:8080/cloud/api/workloads/793816/customization', u'architecture': u'x86', u'executionDate': 1359699487176, u'isHidden': False, u'id': u'793816', u'logsUri': u'http://localhost:8080/cloud/api/workloads/793816/log', u'projectUri': u'http://localhost:8080/cloud/api/projects/1'} from (pid=22224) _do_request /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:256
  78. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.driver [req-f1d422d8-5036-451e-bf3d-1913dc06badd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Waiting for deployment 793816 to complete... from (pid=22224) spawn /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/driver.py:542
  79. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-76f3cb44-a966-4499-9522-d5d7edba41f8 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] < 200: {u'virtualServersUri': u'http://localhost:8080/cloud/api/workloads/793805/virtualServers', u'cloudGroupId': u'18851', u'name': u'cb-yongwen-OPSTACK-vm705-tinyvm', u'timestampsUri': u'http://localhost:8080/cloud/api/workloads/793805/timestamps', u'cloudName': u'cb-yongwen-OPSTACK-vm705-tinyvm', u'hypervisor': u'VMware', u'description': u'cb-yongwen-OPSTACK-vm705-tinyvm', u'cloudId': u'cloud://18851/9db6a9d1-cc3c-4d1f-9dec-13a9d64163cf', u'uri': u'http://localhost:8080/cloud/api/workloads/793805', u'targetUri': u'http://localhost:8080/cloud/api/workloads/793805/target', u'state': {u'id': u'OK', u'label': u'OK'}, u'customizationUri': u'http://localhost:8080/cloud/api/workloads/793805/customization', u'architecture': u'x86', u'cloudGroupName': u'skcm2', u'executionDate': 1359699322644, u'isHidden': False, u'id': u'793805', u'logsUri': u'http://localhost:8080/cloud/api/workloads/793805/log', u'projectUri': u'http://localhost:8080/cloud/api/projects/1'} from (pid=22224) _do_request /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:256
  80. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.driver [req-76f3cb44-a966-4499-9522-d5d7edba41f8 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Deployment 'cb-yongwen-OPSTACK-vm705-tinyvm' finished executing: OK from (pid=22224) spawn /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/driver.py:550
  81. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-76f3cb44-a966-4499-9522-d5d7edba41f8 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Constructed URL: http://localhost:8080/cloud/api/workloads/793805 from (pid=22224) _construct_url /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:223
  82. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-76f3cb44-a966-4499-9522-d5d7edba41f8 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] > GET /cloud/api/workloads/793805 from (pid=22224) _do_request /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:244
  83. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-8a51cf3f-796a-404b-87e6-d8bc1cad78dd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] < 200: CYX1009I: SUCCESS: Workload '793828' saved. from (pid=22224) _do_request /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:256
  84. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.client [req-8a51cf3f-796a-404b-87e6-d8bc1cad78dd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] JSON request: {"name": "cb-yongwen-OPSTACK-vm729-tinyvm", "description": "cb-yongwen-OPSTACK-vm729-tinyvm"} from (pid=22224) update_workload /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/client.py:293
  85. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-8a51cf3f-796a-404b-87e6-d8bc1cad78dd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Constructed URL: http://localhost:8080/cloud/api/workloads/793828 from (pid=22224) _construct_url /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:223
  86. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-8a51cf3f-796a-404b-87e6-d8bc1cad78dd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] > PUT /cloud/api/workloads/793828 from (pid=22224) _do_request /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:244
  87. 2013-02-01 00:23:35 DEBUG smartcloud.virt.sce.baseclient [req-8a51cf3f-796a-404b-87e6-d8bc1cad78dd 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Body: {"name": "cb-yongwen-OPSTACK-vm729-tinyvm", "description": "cb-yongwen-OPSTACK-vm729-tinyvm"} from (pid=22224) _do_request /opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/baseclient.py:246
  88. 2013-02-01 00:23:36 ERROR nova.compute.manager [req-bf088d1e-5090-464e-a195-b15088e66b50 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] [instance: d01d050e-42de-4e43-90db-183917950b20] Instance failed network setup
  89. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] Traceback (most recent call last):
  90. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 709, in _allocate_network
  91. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] requested_networks=requested_networks)
  92. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 44, in wrapper
  93. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] res = f(self, context, *args, **kwargs)
  94. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 249, in allocate_for_instance
  95. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] 'args': args})
  96. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 102, in call
  97. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  98. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 575, in call
  99. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] rpc_amqp.get_connection_pool(conf, Connection))
  100. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 368, in call
  101. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] rv = list(rv)
  102. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 329, in __iter__
  103. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] self.done()
  104. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 306, in done
  105. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] self._connection.close()
  106. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 132, in close
  107. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] self._done()
  108. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 113, in _done
  109. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] self.connection.reset()
  110. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 403, in reset
  111. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] self.session.close()
  112. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "<string>", line 6, in close
  113. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 742, in close
  114. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] link.close(timeout=timeout)
  115. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "<string>", line 6, in close
  116. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1050, in close
  117. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] if not self.session._ewait(lambda: self.closed, timeout=timeout):
  118. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
  119. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  120. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
  121. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] self.check_error()
  122. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
  123. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] raise self.error
  124. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] InternalError: Traceback (most recent call last):
  125. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 654, in write
  126. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] op.dispatch(self)
  127. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
  128. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] getattr(target, handler)(self, *args)
  129. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1295, in do_message_transfer
  130. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] assert rcv.received < rcv.impending, "%s, %s" % (rcv.received, rcv.impending)
  131. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20] AssertionError: 0, 0
  132. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20]
  133. 2013-02-01 00:23:36 TRACE nova.compute.manager [instance: d01d050e-42de-4e43-90db-183917950b20]
  134. 2013-02-01 00:23:36 DEBUG nova.utils [req-bf088d1e-5090-464e-a195-b15088e66b50 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Got semaphore "compute_resources" for method "update_usage"... from (pid=22224) inner /usr/lib/python2.6/site-packages/nova/utils.py:713
  135. 2013-02-01 00:23:39 DEBUG nova.compute.manager [req-bf088d1e-5090-464e-a195-b15088e66b50 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] REVERTING TASK STATE TO NONE FOR d01d050e-42de-4e43-90db-183917950b20 from (pid=22224) decorated_function /usr/lib/python2.6/site-packages/nova/compute/manager.py:177
  136. 2013-02-01 00:23:39 DEBUG nova.utils [req-bf088d1e-5090-464e-a195-b15088e66b50 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Got semaphore "compute_resources" for method "update_usage"... from (pid=22224) inner /usr/lib/python2.6/site-packages/nova/utils.py:713
  137. 2013-02-01 00:23:39 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
  138. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  139. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
  140. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
  141. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
  142. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
  143. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
  144. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
  145. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  146. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  147. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
  148. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
  149. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 182, in decorated_function
  150. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp pass
  151. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  152. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  153. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 167, in decorated_function
  154. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  155. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 203, in decorated_function
  156. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
  157. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  158. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  159. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 197, in decorated_function
  160. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  161. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 829, in run_instance
  162. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp do_run_instance()
  163. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 752, in inner
  164. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
  165. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 828, in do_run_instance
  166. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, instance)
  167. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 517, in _run_instance
  168. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
  169. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  170. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  171. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 483, in _run_instance
  172. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp requested_networks)
  173. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 709, in _allocate_network
  174. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp requested_networks=requested_networks)
  175. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 44, in wrapper
  176. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp res = f(self, context, *args, **kwargs)
  177. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 249, in allocate_for_instance
  178. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp 'args': args})
  179. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 102, in call
  180. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  181. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 575, in call
  182. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
  183. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 368, in call
  184. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp rv = list(rv)
  185. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 329, in __iter__
  186. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.done()
  187. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 306, in done
  188. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self._connection.close()
  189. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 132, in close
  190. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self._done()
  191. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 113, in _done
  192. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.connection.reset()
  193. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 403, in reset
  194. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.session.close()
  195. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "<string>", line 6, in close
  196. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 742, in close
  197. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp link.close(timeout=timeout)
  198. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "<string>", line 6, in close
  199. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1050, in close
  200. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp if not self.session._ewait(lambda: self.closed, timeout=timeout):
  201. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
  202. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  203. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
  204. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp self.check_error()
  205. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
  206. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp raise self.error
  207. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp InternalError: Traceback (most recent call last):
  208. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 654, in write
  209. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp op.dispatch(self)
  210. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
  211. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp getattr(target, handler)(self, *args)
  212. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1295, in do_message_transfer
  213. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp assert rcv.received < rcv.impending, "%s, %s" % (rcv.received, rcv.impending)
  214. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp AssertionError: 0, 0
  215. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp
  216. 2013-02-01 00:23:39 TRACE nova.openstack.common.rpc.amqp
  217. 2013-02-01 00:23:39 ERROR nova.compute.manager [req-38264bcd-ef34-4a1e-a4af-5d8ad88136a1 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] Instance failed network setup
  218. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] Traceback (most recent call last):
  219. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 709, in _allocate_network
  220. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] requested_networks=requested_networks)
  221. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 44, in wrapper
  222. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] res = f(self, context, *args, **kwargs)
  223. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 249, in allocate_for_instance
  224. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] 'args': args})
  225. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 102, in call
  226. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  227. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 575, in call
  228. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] rpc_amqp.get_connection_pool(conf, Connection))
  229. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 368, in call
  230. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] rv = list(rv)
  231. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 329, in __iter__
  232. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] self.done()
  233. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 306, in done
  234. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] self._connection.close()
  235. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 132, in close
  236. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] self._done()
  237. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 113, in _done
  238. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] self.connection.reset()
  239. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 403, in reset
  240. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] self.session.close()
  241. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "<string>", line 6, in close
  242. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 742, in close
  243. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] link.close(timeout=timeout)
  244. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "<string>", line 6, in close
  245. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1050, in close
  246. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] if not self.session._ewait(lambda: self.closed, timeout=timeout):
  247. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
  248. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  249. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
  250. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] self.check_error()
  251. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
  252. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] raise self.error
  253. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] InternalError: Traceback (most recent call last):
  254. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 654, in write
  255. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] op.dispatch(self)
  256. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
  257. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] getattr(target, handler)(self, *args)
  258. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1295, in do_message_transfer
  259. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] assert rcv.received < rcv.impending, "%s, %s" % (rcv.received, rcv.impending)
  260. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a] AssertionError: 0, 0
  261. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a]
  262. 2013-02-01 00:23:39 TRACE nova.compute.manager [instance: a49c75f4-de0c-4fc4-a42d-562a9a7e857a]
  263. 2013-02-01 00:23:40 DEBUG nova.utils [req-38264bcd-ef34-4a1e-a4af-5d8ad88136a1 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Got semaphore "compute_resources" for method "update_usage"... from (pid=22224) inner /usr/lib/python2.6/site-packages/nova/utils.py:713
  264. 2013-02-01 00:23:40 DEBUG nova.compute.manager [req-38264bcd-ef34-4a1e-a4af-5d8ad88136a1 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] REVERTING TASK STATE TO NONE FOR a49c75f4-de0c-4fc4-a42d-562a9a7e857a from (pid=22224) decorated_function /usr/lib/python2.6/site-packages/nova/compute/manager.py:177
  265. 2013-02-01 00:23:40 DEBUG nova.utils [req-38264bcd-ef34-4a1e-a4af-5d8ad88136a1 31fcb75ece584c7296dab83fc81b64d4 34b8c022b1304291bd9f122f3b4a50cb] Got semaphore "compute_resources" for method "update_usage"... from (pid=22224) inner /usr/lib/python2.6/site-packages/nova/utils.py:713
  266. 2013-02-01 00:23:40 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
  267. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  268. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
  269. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
  270. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
  271. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
  272. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
  273. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
  274. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  275. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  276. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
  277. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
  278. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 182, in decorated_function
  279. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp pass
  280. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  281. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  282. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 167, in decorated_function
  283. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  284. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 203, in decorated_function
  285. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
  286. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  287. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  288. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 197, in decorated_function
  289. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  290. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 829, in run_instance
  291. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp do_run_instance()
  292. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 752, in inner
  293. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
  294. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 828, in do_run_instance
  295. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, instance)
  296. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 517, in _run_instance
  297. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
  298. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  299. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.gen.next()
  300. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 483, in _run_instance
  301. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp requested_networks)
  302. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 709, in _allocate_network
  303. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp requested_networks=requested_networks)
  304. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 44, in wrapper
  305. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp res = f(self, context, *args, **kwargs)
  306. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 249, in allocate_for_instance
  307. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp 'args': args})
  308. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 102, in call
  309. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  310. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 575, in call
  311. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
  312. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 368, in call
  313. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp rv = list(rv)
  314. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 329, in __iter__
  315. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.done()
  316. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 306, in done
  317. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self._connection.close()
  318. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 132, in close
  319. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self._done()
  320. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 113, in _done
  321. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.connection.reset()
  322. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 403, in reset
  323. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.session.close()
  324. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "<string>", line 6, in close
  325. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 742, in close
  326. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp link.close(timeout=timeout)
  327. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "<string>", line 6, in close
  328. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1050, in close
  329. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp if not self.session._ewait(lambda: self.closed, timeout=timeout):
  330. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
  331. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  332. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
  333. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp self.check_error()
  334. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
  335. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp raise self.error
  336. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp InternalError: Traceback (most recent call last):
  337. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 654, in write
  338. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp op.dispatch(self)
  339. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
  340. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp getattr(target, handler)(self, *args)
  341. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1295, in do_message_transfer
  342. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp assert rcv.received < rcv.impending, "%s, %s" % (rcv.received, rcv.impending)
  343. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp AssertionError: 0, 0
  344. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp
  345. 2013-02-01 00:23:40 TRACE nova.openstack.common.rpc.amqp
  346. 2013-02-01 00:23:40 ERROR smartcloud.virt.sce.messaging_common [-] The message cannot be handled. The reason is: <type 'exceptions.Exception'>: Traceback (most recent call last):
  347. File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 654, in write
  348. op.dispatch(self)
  349. File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
  350. getattr(target, handler)(self, *args)
  351. File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1295, in do_message_transfer
  352. assert rcv.received < rcv.impending, "%s, %s" % (rcv.received, rcv.impending)
  353. AssertionError: 0, 0
  354. . ResourceType: DEPLOYMENT, Action: DEPLOYMENT_EXECUTION_SUCCESSFUL, Contents: {"cloudName":"cb-yongwen-OPSTACK-vm180-tinyvm","cloudGroupId":"18851","executionDate":1359691380830,"virtualServersUri":"/cloud/api/workloads/754276/virtualServers","name":"cb-yongwen-OPSTACK-vm180-tinyvm","uri":"/cloud/api/workloads/754276","timestampsUri":"/cloud/api/workloads/754276/timestamps","cloudId":"cloud://18851/95dadacb-510c-4d9c-ab39-c48327e589cd","state":{"label":"OK","id":"OK"},"architecture":"x86","projectUri":"/cloud/api/projects/1","isHidden":false,"cloudGroupName":"skcm2","hypervisor":"VMware","customizationUri":"/cloud/api/workloads/754276/customization","targetUri":"/cloud/api/workloads/754276/target","id":"754276","description":"cb-yongwen-OPSTACK-vm180-tinyvm","logsUri":"/cloud/api/workloads/754276/log","instances":1}
  355. 2013-02-01 00:23:40 ERROR smartcloud.virt.sce.messaging_common [-] error type <class 'qpid.messaging.exceptions.InternalError'>
  356. 2013-02-01 00:23:40 ERROR smartcloud.virt.sce.messaging_common [-] error object Traceback (most recent call last):
  357. File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 654, in write
  358. op.dispatch(self)
  359. File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
  360. getattr(target, handler)(self, *args)
  361. File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1295, in do_message_transfer
  362. assert rcv.received < rcv.impending, "%s, %s" % (rcv.received, rcv.impending)
  363. AssertionError: 0, 0
  364.  
  365. 2013-02-01 00:23:40 ERROR smartcloud.virt.sce.messaging_common [-] File "/opt/build/ws/iaas/300/driver/smartcloud/smartcloud/virt/sce/messaging_common.py", line 120, in _dispatch
  366. File "/opt/build/ws/iaas/300/driver/smartcloud/smartcloud/compute/manager.py", line 1710, in _handle_instance_messages
  367. File "/opt/build/ws/iaas/300/driver/smartcloud/smartcloud/compute/manager.py", line 2156, in _update_ip_of_instance
  368. File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 44, in wrapper
  369. res = f(self, context, *args, **kwargs)
  370. File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 292, in get_instance_nw_info
  371. return self._get_instance_nw_info(context, instance)
  372. File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 303, in _get_instance_nw_info
  373. 'args': args})
  374. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 102, in call
  375. return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  376. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 575, in call
  377. rpc_amqp.get_connection_pool(conf, Connection))
  378. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 368, in call
  379. rv = list(rv)
  380. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 329, in __iter__
  381. self.done()
  382. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 306, in done
  383. self._connection.close()
  384. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 132, in close
  385. self._done()
  386. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 113, in _done
  387. self.connection.reset()
  388. File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 403, in reset
  389. self.session.close()
  390. File "<string>", line 6, in close
  391. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 742, in close
  392. link.close(timeout=timeout)
  393. File "<string>", line 6, in close
  394. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1050, in close
  395. if not self.session._ewait(lambda: self.closed, timeout=timeout):
  396. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
  397. result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  398. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
  399. self.check_error()
  400. File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
  401. raise self.error
  402.  
  403. 2013-02-01 00:23:40 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=22224) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:172
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!