API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Apr 20th, 2015
229
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 253.04 KB | None | 0 0
raw download clone embed print report
  1. dsvm-functional develop-inst-nodeps: /opt/openstack/neutron
  2. dsvm-functional runtests: PYTHONHASHSEED='910089220'
  3. dsvm-functional runtests: commands[0] | /opt/openstack/neutron/tools/deploy_rootwrap.sh /opt/openstack/neutron /opt/openstack/neutron/.tox/dsvm-functional/etc /opt/openstack/neutron/.tox/dsvm-functional/bin
  4. dsvm-functional runtests: commands[1] | sh tools/pretty_tox.sh neutron.tests.functional.agent.test_l3_agent neutron.tests.fullstack
  5. running testr
  6. running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --list
  7. running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --load-list /tmp/tmpeUiRlH
  8. running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --load-list /tmp/tmphmMWxs
  9. running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --load-list /tmp/tmpo49oJT
  10. running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --load-list /tmp/tmpIP5gge
  11. {3} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_agent_notifications_for_router_events [3.440450s] ... ok
  12.  
  13. Captured stdout:
  14. ~~~~~~~~~~~~~~~~
  15. 2015-04-20 09:24:08.904 10268 INFO neutron.common.config [-] Logging enabled!
  16. 2015-04-20 09:24:08.905 10268 INFO neutron.common.config [-] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  17. 2015-04-20 09:24:08.912 10268 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=10455
  18. 2015-04-20 09:24:09.115 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Notify callbacks for router, before_create
  19. 2015-04-20 09:24:09.115 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback mock.Mock
  20. 2015-04-20 09:24:09.134 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  21. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  22. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  23. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent self.use_call)
  24. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  25. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  26. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  27. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  28. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  29. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  30. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  31. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  32. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  33. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  34. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  35. 2015-04-20 09:24:09.134 10268 TRACE neutron.agent.l3.agent
  36. 2015-04-20 09:24:09.137 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  37. 2015-04-20 09:24:10.323 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Notify callbacks for router, after_create
  38. 2015-04-20 09:24:10.324 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback mock.Mock
  39. 2015-04-20 09:24:10.324 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  40. 2015-04-20 09:24:11.382 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Notify callbacks for router, before_update
  41. 2015-04-20 09:24:11.382 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback mock.Mock
  42. 2015-04-20 09:24:11.599 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Notify callbacks for router, after_update
  43. 2015-04-20 09:24:11.599 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback mock.Mock
  44. 2015-04-20 09:24:11.600 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Notify callbacks for router, before_delete
  45. 2015-04-20 09:24:11.600 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback mock.Mock
  46. 2015-04-20 09:24:11.600 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  47. 2015-04-20 09:24:12.062 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Notify callbacks for router, after_delete
  48. 2015-04-20 09:24:12.062 10268 INFO neutron.callbacks.manager [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Calling callback mock.Mock
  49. 2015-04-20 09:24:12.063 10268 WARNING neutron.agent.l3.agent [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Info for router a6589ed5-c471-441f-a0c0-c692703016fc were not found. Skipping router removal
  50.  
  51. {0} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_conntrack_disassociate_fip [11.457945s] ... ok
  52.  
  53. Captured stdout:
  54. ~~~~~~~~~~~~~~~~
  55. 2015-04-20 09:24:08.550 10265 INFO neutron.common.config [-] Logging enabled!
  56. 2015-04-20 09:24:08.551 10265 INFO neutron.common.config [-] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  57. 2015-04-20 09:24:08.567 10265 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=10327
  58. 2015-04-20 09:24:08.770 10265 INFO neutron.callbacks.manager [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Notify callbacks for router, before_create
  59. 2015-04-20 09:24:08.784 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  60. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  61. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  62. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent self.use_call)
  63. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  64. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  65. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  66. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  67. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  68. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  69. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  70. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  71. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  72. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  73. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  74. 2015-04-20 09:24:08.784 10265 TRACE neutron.agent.l3.agent
  75. 2015-04-20 09:24:08.785 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  76. 2015-04-20 09:24:09.942 10265 INFO neutron.callbacks.manager [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Notify callbacks for router, after_create
  77. 2015-04-20 09:24:09.942 10265 INFO neutron.callbacks.manager [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  78. 2015-04-20 09:24:19.245 10265 INFO neutron.callbacks.manager [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Notify callbacks for router, before_delete
  79. 2015-04-20 09:24:19.245 10265 INFO neutron.callbacks.manager [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  80. 2015-04-20 09:24:19.749 10265 INFO neutron.callbacks.manager [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Notify callbacks for router, after_delete
  81.  
  82. {2} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_ha_router_conf_on_restarted_agent [14.027622s] ... ok
  83.  
  84. Captured stdout:
  85. ~~~~~~~~~~~~~~~~
  86. 2015-04-20 09:24:08.460 10267 INFO neutron.common.config [-] Logging enabled!
  87. 2015-04-20 09:24:08.460 10267 INFO neutron.common.config [-] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  88. 2015-04-20 09:24:08.473 10267 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=10303
  89. 2015-04-20 09:24:08.704 10267 INFO neutron.callbacks.manager [req-490804be-5094-44b2-815c-c67ea63123f1 - - - - -] Notify callbacks for router, before_create
  90. 2015-04-20 09:24:08.733 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  91. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  92. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  93. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent self.use_call)
  94. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  95. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  96. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  97. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  98. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  99. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  100. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  101. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  102. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  103. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  104. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  105. 2015-04-20 09:24:08.733 10267 TRACE neutron.agent.l3.agent
  106. 2015-04-20 09:24:08.738 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  107. 2015-04-20 09:24:09.216 10267 INFO neutron.agent.l3.ha [req-490804be-5094-44b2-815c-c67ea63123f1 - - - - -] Router 7fcb11f4-c425-460e-9c28-5cf9a67351be transitioned to backup
  108. 2015-04-20 09:24:11.070 10267 INFO neutron.callbacks.manager [req-490804be-5094-44b2-815c-c67ea63123f1 - - - - -] Notify callbacks for router, after_create
  109. 2015-04-20 09:24:11.071 10267 INFO neutron.callbacks.manager [req-490804be-5094-44b2-815c-c67ea63123f1 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  110. 2015-04-20 09:24:11.203 10267 INFO neutron.callbacks.manager [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Notify callbacks for router, before_create
  111. 2015-04-20 09:24:11.205 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  112. 2015-04-20 09:24:11.205 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  113. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  114. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  115. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent self.use_call)
  116. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  117. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  118. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  119. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  120. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  121. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  122. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  123. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  124. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  125. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  126. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  127. 2015-04-20 09:24:11.205 10267 TRACE neutron.agent.l3.agent
  128. 2015-04-20 09:24:11.300 10267 INFO neutron.agent.linux.interface [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Device ha-dbb18843-33 already exists
  129. 2015-04-20 09:24:11.427 10267 INFO neutron.agent.l3.ha [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Router 7fcb11f4-c425-460e-9c28-5cf9a67351be transitioned to backup
  130. 2015-04-20 09:24:11.740 10267 INFO neutron.callbacks.manager [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Notify callbacks for router, after_create
  131. 2015-04-20 09:24:11.740 10267 INFO neutron.callbacks.manager [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  132. 2015-04-20 09:24:20.558 10267 INFO eventlet.wsgi.server [-] (10267) accepted ''
  133. 2015-04-20 09:24:20.559 10267 INFO neutron.agent.l3.ha [-] Router 7fcb11f4-c425-460e-9c28-5cf9a67351be transitioned to master
  134. 2015-04-20 09:24:21.475 10267 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:21] "GET / HTTP/1.1" 200 115 0.915617
  135. 2015-04-20 09:24:21.611 10267 INFO neutron.callbacks.manager [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Notify callbacks for router, before_delete
  136. 2015-04-20 09:24:21.612 10267 INFO neutron.callbacks.manager [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  137. 2015-04-20 09:24:22.151 10267 INFO neutron.callbacks.manager [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Notify callbacks for router, after_delete
  138.  
  139. {1} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_ipv6_ha_router_lifecycle [14.545210s] ... ok
  140.  
  141. Captured stdout:
  142. ~~~~~~~~~~~~~~~~
  143. 2015-04-20 09:24:08.369 10266 INFO neutron.common.config [-] Logging enabled!
  144. 2015-04-20 09:24:08.369 10266 INFO neutron.common.config [-] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  145. 2015-04-20 09:24:08.377 10266 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=10295
  146. 2015-04-20 09:24:08.593 10266 INFO neutron.callbacks.manager [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Notify callbacks for router, before_create
  147. 2015-04-20 09:24:08.624 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  148. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  149. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  150. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent self.use_call)
  151. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  152. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  153. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  154. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  155. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  156. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  157. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  158. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  159. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  160. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  161. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  162. 2015-04-20 09:24:08.624 10266 TRACE neutron.agent.l3.agent
  163. 2015-04-20 09:24:08.632 10266 INFO eventlet.wsgi.server [-] (10266) wsgi starting up on http:///:t/
  164. 2015-04-20 09:24:09.133 10266 INFO neutron.agent.l3.ha [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Router dfee100c-05e0-4b03-b184-16151dc4624f transitioned to backup
  165. 2015-04-20 09:24:11.149 10266 INFO neutron.callbacks.manager [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Notify callbacks for router, after_create
  166. 2015-04-20 09:24:11.150 10266 INFO neutron.callbacks.manager [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  167. 2015-04-20 09:24:20.529 10266 INFO eventlet.wsgi.server [-] (10266) accepted ''
  168. 2015-04-20 09:24:20.530 10266 INFO neutron.agent.l3.ha [-] Router dfee100c-05e0-4b03-b184-16151dc4624f transitioned to master
  169. 2015-04-20 09:24:21.396 10266 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:21] "GET / HTTP/1.1" 200 115 0.865834
  170. 2015-04-20 09:24:22.076 10266 INFO neutron.callbacks.manager [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Notify callbacks for router, before_delete
  171. 2015-04-20 09:24:22.076 10266 INFO neutron.callbacks.manager [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  172. 2015-04-20 09:24:22.646 10266 INFO neutron.callbacks.manager [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Notify callbacks for router, after_delete
  173. 2015-04-20 09:24:22.666 10266 WARNING neutron.agent.l3.agent [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Info for router dfee100c-05e0-4b03-b184-16151dc4624f were not found. Skipping router removal
  174.  
  175. {2} neutron.tests.functional.agent.test_l3_agent.MetadataL3AgentTestCase.test_access_to_metadata_proxy [3.188793s] ... ok
  176.  
  177. Captured stdout:
  178. ~~~~~~~~~~~~~~~~
  179. 2015-04-20 09:24:22.215 10267 INFO neutron.common.config [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] Logging enabled!
  180. 2015-04-20 09:24:22.215 10267 INFO neutron.common.config [req-2c9aac79-3935-426f-ad6c-0d9e8eb242f0 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  181. 2015-04-20 09:24:22.293 10267 INFO neutron.callbacks.manager [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Notify callbacks for router, before_create
  182. 2015-04-20 09:24:22.295 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  183. 2015-04-20 09:24:22.295 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  184. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  185. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  186. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent self.use_call)
  187. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  188. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  189. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  190. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  191. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  192. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  193. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  194. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  195. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  196. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  197. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  198. 2015-04-20 09:24:22.295 10267 TRACE neutron.agent.l3.agent
  199. 2015-04-20 09:24:23.570 10267 INFO neutron.callbacks.manager [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Notify callbacks for router, after_create
  200. 2015-04-20 09:24:23.570 10267 INFO neutron.callbacks.manager [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  201. 2015-04-20 09:24:24.585 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  202. 2015-04-20 09:24:24.829 10267 INFO eventlet.wsgi.server [-] (10267) accepted ''
  203. 2015-04-20 09:24:24.830 10267 INFO eventlet.wsgi.server [-] 35.4.0.5,<local> - - [20/Apr/2015 09:24:24] "GET / HTTP/1.1" 200 115 0.000268
  204. 2015-04-20 09:24:24.876 10267 INFO eventlet.wsgi.server [-] (10267) wsgi exited, is_accepting=True
  205. 2015-04-20 09:24:24.877 10267 INFO neutron.callbacks.manager [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Notify callbacks for router, before_delete
  206. 2015-04-20 09:24:24.877 10267 INFO neutron.callbacks.manager [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  207. 2015-04-20 09:24:25.347 10267 INFO neutron.callbacks.manager [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Notify callbacks for router, after_delete
  208.  
  209. {3} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_ha_router_lifecycle [14.035255s] ... ok
  210.  
  211. Captured stdout:
  212. ~~~~~~~~~~~~~~~~
  213. 2015-04-20 09:24:12.125 10268 INFO neutron.common.config [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] Logging enabled!
  214. 2015-04-20 09:24:12.125 10268 INFO neutron.common.config [req-5152d0dd-f037-4d5d-bec0-c92b67046686 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  215. 2015-04-20 09:24:12.183 10268 INFO neutron.callbacks.manager [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Notify callbacks for router, before_create
  216. 2015-04-20 09:24:12.186 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  217. 2015-04-20 09:24:12.186 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  218. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  219. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  220. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent self.use_call)
  221. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  222. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  223. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  224. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  225. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  226. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  227. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  228. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  229. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  230. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  231. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  232. 2015-04-20 09:24:12.186 10268 TRACE neutron.agent.l3.agent
  233. 2015-04-20 09:24:12.821 10268 INFO neutron.agent.l3.ha [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Router bc03630d-fc27-4f0e-84e6-befeeb1c5f66 transitioned to backup
  234. 2015-04-20 09:24:14.412 10268 INFO neutron.callbacks.manager [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Notify callbacks for router, after_create
  235. 2015-04-20 09:24:14.413 10268 INFO neutron.callbacks.manager [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  236. 2015-04-20 09:24:23.795 10268 INFO eventlet.wsgi.server [-] (10268) accepted ''
  237. 2015-04-20 09:24:23.796 10268 INFO neutron.agent.l3.ha [-] Router bc03630d-fc27-4f0e-84e6-befeeb1c5f66 transitioned to master
  238. 2015-04-20 09:24:24.805 10268 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:24] "GET / HTTP/1.1" 200 115 1.009545
  239. 2015-04-20 09:24:25.448 10268 INFO neutron.callbacks.manager [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Notify callbacks for router, before_delete
  240. 2015-04-20 09:24:25.448 10268 INFO neutron.callbacks.manager [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  241. 2015-04-20 09:24:26.060 10268 INFO neutron.callbacks.manager [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Notify callbacks for router, after_delete
  242. 2015-04-20 09:24:26.078 10268 WARNING neutron.agent.l3.agent [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Info for router bc03630d-fc27-4f0e-84e6-befeeb1c5f66 were not found. Skipping router removal
  243.  
  244. {0} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_fip_connection_from_same_subnet [6.462774s] ... ok
  245.  
  246. Captured stdout:
  247. ~~~~~~~~~~~~~~~~
  248. 2015-04-20 09:24:19.818 10265 INFO neutron.common.config [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] Logging enabled!
  249. 2015-04-20 09:24:19.818 10265 INFO neutron.common.config [req-98bec038-fd63-4f79-b7b0-1bb0bf48d97f - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  250. 2015-04-20 09:24:19.867 10265 INFO neutron.callbacks.manager [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Notify callbacks for router, before_create
  251. 2015-04-20 09:24:19.870 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  252. 2015-04-20 09:24:19.871 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  253. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  254. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  255. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent self.use_call)
  256. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  257. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  258. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  259. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  260. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  261. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  262. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  263. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  264. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  265. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  266. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  267. 2015-04-20 09:24:19.871 10265 TRACE neutron.agent.l3.agent
  268. 2015-04-20 09:24:20.916 10265 INFO neutron.callbacks.manager [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Notify callbacks for router, after_create
  269. 2015-04-20 09:24:20.916 10265 INFO neutron.callbacks.manager [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  270. 2015-04-20 09:24:25.743 10265 INFO neutron.callbacks.manager [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Notify callbacks for router, before_delete
  271. 2015-04-20 09:24:25.743 10265 INFO neutron.callbacks.manager [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  272. 2015-04-20 09:24:26.206 10265 INFO neutron.callbacks.manager [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Notify callbacks for router, after_delete
  273.  
  274. {3} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_keepalived_configuration [3.421230s] ... ok
  275.  
  276. Captured stdout:
  277. ~~~~~~~~~~~~~~~~
  278. 2015-04-20 09:24:26.162 10268 INFO neutron.common.config [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] Logging enabled!
  279. 2015-04-20 09:24:26.162 10268 INFO neutron.common.config [req-c52b0f88-82c1-4780-b317-6f942f111cd3 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  280. 2015-04-20 09:24:26.224 10268 INFO neutron.callbacks.manager [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Notify callbacks for router, before_create
  281. 2015-04-20 09:24:26.226 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  282. 2015-04-20 09:24:26.226 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  283. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  284. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  285. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent self.use_call)
  286. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  287. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  288. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  289. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  290. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  291. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  292. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  293. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  294. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  295. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  296. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  297. 2015-04-20 09:24:26.226 10268 TRACE neutron.agent.l3.agent
  298. 2015-04-20 09:24:26.830 10268 INFO neutron.agent.l3.ha [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Router 322e0bc8-e61a-43cc-832a-fe96eebb73c9 transitioned to backup
  299. 2015-04-20 09:24:28.657 10268 INFO neutron.callbacks.manager [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Notify callbacks for router, after_create
  300. 2015-04-20 09:24:28.657 10268 INFO neutron.callbacks.manager [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  301. 2015-04-20 09:24:29.022 10268 INFO neutron.callbacks.manager [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Notify callbacks for router, before_delete
  302. 2015-04-20 09:24:29.023 10268 INFO neutron.callbacks.manager [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  303. 2015-04-20 09:24:29.531 10268 INFO neutron.callbacks.manager [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Notify callbacks for router, after_delete
  304.  
  305. {0} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_legacy_router_lifecycle [3.788498s] ... ok
  306.  
  307. Captured stdout:
  308. ~~~~~~~~~~~~~~~~
  309. 2015-04-20 09:24:26.278 10265 INFO neutron.common.config [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] Logging enabled!
  310. 2015-04-20 09:24:26.278 10265 INFO neutron.common.config [req-0b88800b-f74f-4cd2-b34d-00cc594a00a1 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  311. 2015-04-20 09:24:26.338 10265 INFO neutron.callbacks.manager [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Notify callbacks for router, before_create
  312. 2015-04-20 09:24:26.339 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  313. 2015-04-20 09:24:26.340 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  314. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  315. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  316. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent self.use_call)
  317. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  318. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  319. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  320. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  321. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  322. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  323. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  324. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  325. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  326. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  327. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  328. 2015-04-20 09:24:26.340 10265 TRACE neutron.agent.l3.agent
  329. 2015-04-20 09:24:27.566 10265 INFO neutron.callbacks.manager [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Notify callbacks for router, after_create
  330. 2015-04-20 09:24:27.566 10265 INFO neutron.callbacks.manager [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  331. 2015-04-20 09:24:29.465 10265 INFO neutron.callbacks.manager [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Notify callbacks for router, before_delete
  332. 2015-04-20 09:24:29.465 10265 INFO neutron.callbacks.manager [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  333. 2015-04-20 09:24:29.985 10265 INFO neutron.callbacks.manager [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Notify callbacks for router, after_delete
  334. 2015-04-20 09:24:30.005 10265 WARNING neutron.agent.l3.agent [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Info for router 73f71201-aa7d-47c5-a7aa-c3a9f903851b were not found. Skipping router removal
  335.  
  336. {2} neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_fips_for_multiple_ext_networks [8.994708s] ... ok
  337.  
  338. Captured stdout:
  339. ~~~~~~~~~~~~~~~~
  340. 2015-04-20 09:24:25.401 10267 INFO neutron.common.config [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] Logging enabled!
  341. 2015-04-20 09:24:25.401 10267 INFO neutron.common.config [req-eb481af5-1c51-426d-8539-950b5501f9ea - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  342. 2015-04-20 09:24:25.454 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, before_create
  343. 2015-04-20 09:24:25.455 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  344. 2015-04-20 09:24:25.456 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  345. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  346. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  347. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent self.use_call)
  348. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  349. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  350. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  351. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  352. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  353. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  354. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  355. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  356. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  357. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  358. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  359. 2015-04-20 09:24:25.456 10267 TRACE neutron.agent.l3.agent
  360. 2015-04-20 09:24:27.836 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, after_create
  361. 2015-04-20 09:24:27.836 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  362. 2015-04-20 09:24:29.015 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, before_create
  363. 2015-04-20 09:24:31.398 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, after_create
  364. 2015-04-20 09:24:31.398 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  365. 2015-04-20 09:24:32.526 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, before_delete
  366. 2015-04-20 09:24:32.527 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  367. 2015-04-20 09:24:33.410 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, after_delete
  368. 2015-04-20 09:24:33.410 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, before_delete
  369. 2015-04-20 09:24:33.411 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  370. 2015-04-20 09:24:34.334 10267 INFO neutron.callbacks.manager [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Notify callbacks for router, after_delete
  371.  
  372. {2} neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_rem_fips_on_restarted_agent [6.422733s] ... ok
  373.  
  374. Captured stdout:
  375. ~~~~~~~~~~~~~~~~
  376. 2015-04-20 09:24:34.398 10267 INFO neutron.common.config [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] Logging enabled!
  377. 2015-04-20 09:24:34.398 10267 INFO neutron.common.config [req-0a2af93c-2591-434f-ac13-90fc40f73ac6 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  378. 2015-04-20 09:24:34.454 10267 INFO neutron.callbacks.manager [req-fce1cacf-bb83-4c9d-8a6b-445304c86619 - - - - -] Notify callbacks for router, before_create
  379. 2015-04-20 09:24:34.457 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  380. 2015-04-20 09:24:34.457 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  381. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  382. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  383. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent self.use_call)
  384. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  385. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  386. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  387. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  388. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  389. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  390. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  391. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  392. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  393. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  394. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  395. 2015-04-20 09:24:34.457 10267 TRACE neutron.agent.l3.agent
  396. 2015-04-20 09:24:37.896 10267 INFO neutron.callbacks.manager [req-fce1cacf-bb83-4c9d-8a6b-445304c86619 - - - - -] Notify callbacks for router, after_create
  397. 2015-04-20 09:24:37.896 10267 INFO neutron.callbacks.manager [req-fce1cacf-bb83-4c9d-8a6b-445304c86619 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  398. 2015-04-20 09:24:38.733 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  399. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  400. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  401. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent self.use_call)
  402. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  403. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  404. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  405. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  406. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  407. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  408. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  409. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  410. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  411. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  412. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  413. 2015-04-20 09:24:38.733 10267 TRACE neutron.agent.l3.agent
  414. 2015-04-20 09:24:38.828 10267 INFO neutron.callbacks.manager [req-a7c48d83-296e-4d93-9e13-66d200ff1387 - - - - -] Notify callbacks for router, before_create
  415. 2015-04-20 09:24:38.834 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  416. 2015-04-20 09:24:38.835 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  417. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  418. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  419. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent self.use_call)
  420. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  421. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  422. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  423. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  424. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  425. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  426. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  427. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  428. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  429. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  430. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  431. 2015-04-20 09:24:38.835 10267 TRACE neutron.agent.l3.agent
  432. 2015-04-20 09:24:40.551 10267 INFO neutron.callbacks.manager [req-a7c48d83-296e-4d93-9e13-66d200ff1387 - - - - -] Notify callbacks for router, after_create
  433. 2015-04-20 09:24:40.551 10267 INFO neutron.callbacks.manager [req-a7c48d83-296e-4d93-9e13-66d200ff1387 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  434.  
  435. {1} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_keepalived_state_change_bulk_rpc [18.457791s] ... ok
  436.  
  437. Captured stdout:
  438. ~~~~~~~~~~~~~~~~
  439. 2015-04-20 09:24:22.736 10266 INFO neutron.common.config [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] Logging enabled!
  440. 2015-04-20 09:24:22.737 10266 INFO neutron.common.config [req-4fd74d52-8802-4d41-8af5-cc4ffd58f2da - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  441. 2015-04-20 09:24:22.794 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, before_create
  442. 2015-04-20 09:24:22.796 10266 INFO eventlet.wsgi.server [-] (10266) wsgi starting up on http:///:t/
  443. 2015-04-20 09:24:22.797 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  444. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  445. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  446. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent self.use_call)
  447. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  448. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  449. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  450. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  451. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  452. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  453. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  454. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  455. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  456. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  457. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  458. 2015-04-20 09:24:22.797 10266 TRACE neutron.agent.l3.agent
  459. 2015-04-20 09:24:23.389 10266 INFO neutron.agent.l3.ha [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Router 5c54357d-11a6-4d9c-b366-56c7740fc2d3 transitioned to backup
  460. 2015-04-20 09:24:25.203 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, after_create
  461. 2015-04-20 09:24:25.203 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  462. 2015-04-20 09:24:25.368 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, before_create
  463. 2015-04-20 09:24:26.082 10266 INFO neutron.agent.l3.ha [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Router 3f127564-6297-440b-b4b0-1922b2d9de4c transitioned to backup
  464. 2015-04-20 09:24:27.943 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, after_create
  465. 2015-04-20 09:24:27.944 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  466. 2015-04-20 09:24:36.751 10266 INFO eventlet.wsgi.server [-] (10266) accepted ''
  467. 2015-04-20 09:24:36.751 10266 INFO neutron.agent.l3.ha [-] Router 3f127564-6297-440b-b4b0-1922b2d9de4c transitioned to master
  468. 2015-04-20 09:24:37.484 10266 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:37] "GET / HTTP/1.1" 200 115 0.732797
  469. 2015-04-20 09:24:38.624 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  470. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  471. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  472. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent self.use_call)
  473. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  474. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  475. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  476. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  477. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  478. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  479. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  480. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  481. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  482. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  483. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  484. 2015-04-20 09:24:38.624 10266 TRACE neutron.agent.l3.agent
  485. 2015-04-20 09:24:40.076 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, before_delete
  486. 2015-04-20 09:24:40.077 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  487. 2015-04-20 09:24:40.603 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, after_delete
  488. 2015-04-20 09:24:40.603 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, before_delete
  489. 2015-04-20 09:24:40.604 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  490. 2015-04-20 09:24:41.120 10266 INFO neutron.callbacks.manager [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Notify callbacks for router, after_delete
  491.  
  492. {3} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_keepalived_state_change_notification [14.225464s] ... ok
  493.  
  494. Captured stdout:
  495. ~~~~~~~~~~~~~~~~
  496. 2015-04-20 09:24:29.584 10268 INFO neutron.common.config [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] Logging enabled!
  497. 2015-04-20 09:24:29.585 10268 INFO neutron.common.config [req-27e3748f-1b3b-4029-b7e5-6e77d8553f24 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  498. 2015-04-20 09:24:29.641 10268 INFO neutron.callbacks.manager [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Notify callbacks for router, before_create
  499. 2015-04-20 09:24:29.644 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  500. 2015-04-20 09:24:29.644 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  501. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  502. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  503. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent self.use_call)
  504. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  505. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  506. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  507. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  508. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  509. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  510. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  511. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  512. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  513. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  514. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  515. 2015-04-20 09:24:29.644 10268 TRACE neutron.agent.l3.agent
  516. 2015-04-20 09:24:32.037 10268 INFO neutron.callbacks.manager [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Notify callbacks for router, after_create
  517. 2015-04-20 09:24:32.037 10268 INFO neutron.callbacks.manager [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  518. 2015-04-20 09:24:39.135 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  519. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  520. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  521. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent self.use_call)
  522. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  523. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  524. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  525. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  526. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  527. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  528. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  529. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  530. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  531. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  532. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  533. 2015-04-20 09:24:39.135 10268 TRACE neutron.agent.l3.agent
  534. 2015-04-20 09:24:40.848 10268 INFO eventlet.wsgi.server [-] (10268) accepted ''
  535. 2015-04-20 09:24:40.853 10268 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:40] "GET / HTTP/1.1" 200 115 0.003926
  536. 2015-04-20 09:24:42.186 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  537. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  538. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  539. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent self.use_call)
  540. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  541. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  542. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  543. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  544. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  545. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  546. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  547. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  548. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  549. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  550. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  551. 2015-04-20 09:24:42.186 10268 TRACE neutron.agent.l3.agent
  552. 2015-04-20 09:24:42.849 10268 INFO eventlet.wsgi.server [-] (10268) accepted ''
  553. 2015-04-20 09:24:42.851 10268 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:42] "GET / HTTP/1.1" 200 115 0.001393
  554. 2015-04-20 09:24:43.196 10268 INFO neutron.callbacks.manager [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Notify callbacks for router, before_delete
  555. 2015-04-20 09:24:43.196 10268 INFO neutron.callbacks.manager [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  556. 2015-04-20 09:24:43.752 10268 INFO neutron.callbacks.manager [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Notify callbacks for router, after_delete
  557.  
  558. {2} neutron.tests.functional.agent.test_l3_agent.UnprivilegedUserGroupMetadataL3AgentTestCase.test_access_to_metadata_proxy [3.415611s] ... ok
  559.  
  560. Captured stdout:
  561. ~~~~~~~~~~~~~~~~
  562. 2015-04-20 09:24:40.822 10267 INFO neutron.common.config [req-a7c48d83-296e-4d93-9e13-66d200ff1387 - - - - -] Logging enabled!
  563. 2015-04-20 09:24:40.823 10267 INFO neutron.common.config [req-a7c48d83-296e-4d93-9e13-66d200ff1387 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  564. 2015-04-20 09:24:40.883 10267 INFO neutron.callbacks.manager [req-2e28d56e-c99c-45a8-ab3d-cbd293b84019 - - - - -] Notify callbacks for router, before_create
  565. 2015-04-20 09:24:40.889 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  566. 2015-04-20 09:24:40.890 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  567. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  568. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  569. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent self.use_call)
  570. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  571. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  572. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  573. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  574. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  575. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  576. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  577. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  578. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  579. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  580. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  581. 2015-04-20 09:24:40.890 10267 TRACE neutron.agent.l3.agent
  582. 2015-04-20 09:24:41.205 10267 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  583. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  584. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  585. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent self.use_call)
  586. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  587. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  588. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  589. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent retry=self.retry)
  590. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  591. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  592. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  593. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  594. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  595. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  596. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  597. 2015-04-20 09:24:41.205 10267 TRACE neutron.agent.l3.agent
  598. 2015-04-20 09:24:42.334 10267 INFO neutron.callbacks.manager [req-2e28d56e-c99c-45a8-ab3d-cbd293b84019 - - - - -] Notify callbacks for router, after_create
  599. 2015-04-20 09:24:42.335 10267 INFO neutron.callbacks.manager [req-2e28d56e-c99c-45a8-ab3d-cbd293b84019 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  600. 2015-04-20 09:24:43.308 10267 INFO eventlet.wsgi.server [-] (10267) wsgi starting up on http:///:t/
  601. 2015-04-20 09:24:43.549 10267 INFO eventlet.wsgi.server [-] (10267) accepted ''
  602. 2015-04-20 09:24:43.550 10267 INFO eventlet.wsgi.server [-] 35.4.0.5,<local> - - [20/Apr/2015 09:24:43] "GET / HTTP/1.1" 200 115 0.000348
  603. 2015-04-20 09:24:43.605 10267 INFO eventlet.wsgi.server [-] (10267) wsgi exited, is_accepting=True
  604. 2015-04-20 09:24:43.605 10267 INFO neutron.callbacks.manager [req-2e28d56e-c99c-45a8-ab3d-cbd293b84019 - - - - -] Notify callbacks for router, before_delete
  605. 2015-04-20 09:24:43.605 10267 INFO neutron.callbacks.manager [req-2e28d56e-c99c-45a8-ab3d-cbd293b84019 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  606. 2015-04-20 09:24:44.159 10267 INFO neutron.callbacks.manager [req-2e28d56e-c99c-45a8-ab3d-cbd293b84019 - - - - -] Notify callbacks for router, after_delete
  607.  
  608. {1} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_legacy_router_lifecycle_with_no_gateway_subnet [4.124216s] ... ok
  609.  
  610. Captured stdout:
  611. ~~~~~~~~~~~~~~~~
  612. 2015-04-20 09:24:41.194 10266 INFO neutron.common.config [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] Logging enabled!
  613. 2015-04-20 09:24:41.194 10266 INFO neutron.common.config [req-98e265ce-decb-461e-9305-1296c5ff1841 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  614. 2015-04-20 09:24:41.268 10266 INFO neutron.callbacks.manager [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Notify callbacks for router, before_create
  615. 2015-04-20 09:24:41.270 10266 INFO eventlet.wsgi.server [-] (10266) wsgi starting up on http:///:t/
  616. 2015-04-20 09:24:41.271 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  617. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  618. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  619. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent self.use_call)
  620. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  621. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  622. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  623. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  624. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  625. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  626. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  627. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  628. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  629. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  630. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  631. 2015-04-20 09:24:41.271 10266 TRACE neutron.agent.l3.agent
  632. 2015-04-20 09:24:42.685 10266 INFO neutron.callbacks.manager [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Notify callbacks for router, after_create
  633. 2015-04-20 09:24:42.686 10266 INFO neutron.callbacks.manager [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  634. 2015-04-20 09:24:44.684 10266 INFO neutron.callbacks.manager [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Notify callbacks for router, before_delete
  635. 2015-04-20 09:24:44.684 10266 INFO neutron.callbacks.manager [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  636. 2015-04-20 09:24:45.236 10266 INFO neutron.callbacks.manager [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Notify callbacks for router, after_delete
  637. 2015-04-20 09:24:45.260 10266 WARNING neutron.agent.l3.agent [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Info for router f3e161c9-7672-4855-b981-2cb6eadcb875 were not found. Skipping router removal
  638.  
  639. {1} neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_with_snat_with_fips [6.765972s] ... ok
  640.  
  641. Captured stdout:
  642. ~~~~~~~~~~~~~~~~
  643. 2015-04-20 09:24:45.327 10266 INFO neutron.common.config [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] Logging enabled!
  644. 2015-04-20 09:24:45.327 10266 INFO neutron.common.config [req-b39a223d-e785-4a19-a4cc-f55ac1d1e2db - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  645. 2015-04-20 09:24:45.392 10266 INFO neutron.callbacks.manager [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Notify callbacks for router, before_create
  646. 2015-04-20 09:24:45.395 10266 INFO eventlet.wsgi.server [-] (10266) wsgi starting up on http:///:t/
  647. 2015-04-20 09:24:45.395 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  648. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  649. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  650. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent self.use_call)
  651. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  652. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  653. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  654. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  655. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  656. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  657. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  658. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  659. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  660. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  661. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  662. 2015-04-20 09:24:45.395 10266 TRACE neutron.agent.l3.agent
  663. 2015-04-20 09:24:48.971 10266 INFO neutron.callbacks.manager [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Notify callbacks for router, after_create
  664. 2015-04-20 09:24:48.971 10266 INFO neutron.callbacks.manager [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  665. 2015-04-20 09:24:50.389 10266 INFO neutron.callbacks.manager [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Notify callbacks for router, before_delete
  666. 2015-04-20 09:24:50.389 10266 INFO neutron.callbacks.manager [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  667. 2015-04-20 09:24:51.940 10266 INFO neutron.callbacks.manager [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Notify callbacks for router, after_delete
  668. 2015-04-20 09:24:51.988 10266 WARNING neutron.agent.l3.agent [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Info for router 1c1ef2a8-dfb0-4cc7-b7bd-a9e349fdc7de were not found. Skipping router removal
  669.  
  670. {0} neutron.tests.functional.agent.test_l3_agent.L3HATestFramework.test_ha_router_failover [22.563040s] ... ok
  671.  
  672. Captured stdout:
  673. ~~~~~~~~~~~~~~~~
  674. 2015-04-20 09:24:30.082 10265 INFO neutron.common.config [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] Logging enabled!
  675. 2015-04-20 09:24:30.082 10265 INFO neutron.common.config [req-b476c215-658b-4d23-9d45-bc2e2c7688c2 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  676. 2015-04-20 09:24:30.142 10265 INFO neutron.common.config [req-9ec6aa93-e19f-4989-b32b-623bccdd540d - - - - -] Logging enabled!
  677. 2015-04-20 09:24:30.142 10265 INFO neutron.common.config [req-9ec6aa93-e19f-4989-b32b-623bccdd540d - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  678. 2015-04-20 09:24:30.146 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  679. 2015-04-20 09:24:30.146 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  680. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  681. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  682. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent self.use_call)
  683. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  684. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  685. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  686. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  687. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  688. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  689. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  690. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  691. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  692. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  693. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  694. 2015-04-20 09:24:30.146 10265 TRACE neutron.agent.l3.agent
  695. 2015-04-20 09:24:30.204 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  696. 2015-04-20 09:24:30.205 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  697. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  698. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  699. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent self.use_call)
  700. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  701. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  702. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  703. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  704. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  705. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  706. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  707. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  708. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  709. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  710. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  711. 2015-04-20 09:24:30.205 10265 TRACE neutron.agent.l3.agent
  712. 2015-04-20 09:24:30.286 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, before_create
  713. 2015-04-20 09:24:30.698 10265 INFO neutron.agent.l3.ha [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Router 6c125be5-6f97-4e6f-ab86-73b31bd21fcc transitioned to backup
  714. 2015-04-20 09:24:32.361 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, after_create
  715. 2015-04-20 09:24:32.362 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  716. 2015-04-20 09:24:32.508 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, before_create
  717. 2015-04-20 09:24:32.930 10265 INFO neutron.agent.l3.ha [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Router 6c125be5-6f97-4e6f-ab86-73b31bd21fcc transitioned to backup
  718. 2015-04-20 09:24:34.612 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, after_create
  719. 2015-04-20 09:24:34.612 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  720. 2015-04-20 09:24:38.784 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  721. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  722. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  723. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent self.use_call)
  724. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  725. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  726. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  727. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  728. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  729. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  730. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  731. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  732. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  733. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  734. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  735. 2015-04-20 09:24:38.784 10265 TRACE neutron.agent.l3.agent
  736. 2015-04-20 09:24:41.170 10265 INFO eventlet.wsgi.server [-] (10265) accepted ''
  737. 2015-04-20 09:24:41.171 10265 INFO neutron.agent.l3.ha [-] Router 6c125be5-6f97-4e6f-ab86-73b31bd21fcc transitioned to master
  738. 2015-04-20 09:24:42.030 10265 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:42] "GET / HTTP/1.1" 200 115 0.858801
  739. 2015-04-20 09:24:43.179 10265 INFO eventlet.wsgi.server [-] (10265) accepted ''
  740. 2015-04-20 09:24:43.180 10265 INFO neutron.agent.l3.ha [-] Router 6c125be5-6f97-4e6f-ab86-73b31bd21fcc transitioned to backup
  741. 2015-04-20 09:24:43.190 10265 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:43] "GET / HTTP/1.1" 200 115 0.009292
  742. 2015-04-20 09:24:49.871 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  743. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  744. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  745. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent self.use_call)
  746. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  747. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  748. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  749. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  750. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  751. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  752. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  753. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  754. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  755. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  756. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  757. 2015-04-20 09:24:49.871 10265 TRACE neutron.agent.l3.agent
  758. 2015-04-20 09:24:49.984 10265 INFO eventlet.wsgi.server [-] (10265) accepted ''
  759. 2015-04-20 09:24:49.985 10265 INFO neutron.agent.l3.ha [-] Router 6c125be5-6f97-4e6f-ab86-73b31bd21fcc transitioned to master
  760. 2015-04-20 09:24:50.787 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, before_delete
  761. 2015-04-20 09:24:50.788 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  762. 2015-04-20 09:24:51.220 10265 INFO eventlet.wsgi.server [-] <local> - - [20/Apr/2015 09:24:51] "GET / HTTP/1.1" 200 115 1.235606
  763. 2015-04-20 09:24:51.771 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, after_delete
  764. 2015-04-20 09:24:51.771 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, before_delete
  765. 2015-04-20 09:24:51.772 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  766. 2015-04-20 09:24:52.438 10265 INFO neutron.callbacks.manager [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Notify callbacks for router, after_delete
  767.  
  768. {3} neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_periodic_sync_routers_task [10.431793s] ... FAILED
  769.  
  770. Captured traceback-1:
  771. ~~~~~~~~~~~~~~~~~~~~~
  772. Traceback (most recent call last):
  773. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  774. agent._router_removed(router_id)
  775. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  776. self, router=ri)
  777. File "neutron/callbacks/registry.py", line 44, in notify
  778. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  779. File "neutron/callbacks/manager.py", line 123, in notify
  780. raise exceptions.CallbackFailure(errors=errors)
  781. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  782. Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  783. Exit code: 1
  784. Stdin:
  785. Stdout:
  786. Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  787. "
  788.  
  789.  
  790. Captured traceback-2:
  791. ~~~~~~~~~~~~~~~~~~~~~
  792. Traceback (most recent call last):
  793. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  794. agent._router_removed(router_id)
  795. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  796. self, router=ri)
  797. File "neutron/callbacks/registry.py", line 44, in notify
  798. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  799. File "neutron/callbacks/manager.py", line 123, in notify
  800. raise exceptions.CallbackFailure(errors=errors)
  801. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  802. Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  803. Exit code: 1
  804. Stdin:
  805. Stdout:
  806. Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  807. "
  808.  
  809.  
  810. Captured traceback-3:
  811. ~~~~~~~~~~~~~~~~~~~~~
  812. Traceback (most recent call last):
  813. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  814. agent._router_removed(router_id)
  815. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  816. self, router=ri)
  817. File "neutron/callbacks/registry.py", line 44, in notify
  818. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  819. File "neutron/callbacks/manager.py", line 123, in notify
  820. raise exceptions.CallbackFailure(errors=errors)
  821. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  822. Command: ['ip', 'netns', 'exec', 'qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0', 'iptables-restore', '-c']
  823. Exit code: 1
  824. Stdin: # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  825. *raw
  826. :PREROUTING ACCEPT [0:0]
  827. :OUTPUT ACCEPT [0:0]
  828. :run.py-OUTPUT - [0:0]
  829. :run.py-PREROUTING - [0:0]
  830. [0:0] -A PREROUTING -j run.py-PREROUTING
  831. [0:0] -A OUTPUT -j run.py-OUTPUT
  832. COMMIT
  833. # Completed on Mon Apr 20 09:24:54 2015
  834. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  835. *nat
  836. :PREROUTING ACCEPT [0:0]
  837. :INPUT ACCEPT [0:0]
  838. :OUTPUT ACCEPT [0:0]
  839. :POSTROUTING ACCEPT [0:0]
  840. :neutron-postrouting-bottom - [0:0]
  841. :run.py-OUTPUT - [0:0]
  842. :run.py-POSTROUTING - [0:0]
  843. :run.py-PREROUTING - [0:0]
  844. :run.py-float-snat - [0:0]
  845. :run.py-snat - [0:0]
  846. [0:0] -A PREROUTING -j run.py-PREROUTING
  847. [0:0] -A OUTPUT -j run.py-OUTPUT
  848. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  849. [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  850. [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  851. [0:0] -A run.py-snat -j run.py-float-snat
  852. [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  853. [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  854. [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  855. [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  856. [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  857. [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  858. COMMIT
  859. # Completed on Mon Apr 20 09:24:54 2015
  860. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  861. *mangle
  862. :PREROUTING ACCEPT [0:0]
  863. :INPUT ACCEPT [0:0]
  864. :FORWARD ACCEPT [0:0]
  865. :OUTPUT ACCEPT [0:0]
  866. :POSTROUTING ACCEPT [0:0]
  867. :run.py-FORWARD - [0:0]
  868. :run.py-INPUT - [0:0]
  869. :run.py-OUTPUT - [0:0]
  870. :run.py-POSTROUTING - [0:0]
  871. :run.py-PREROUTING - [0:0]
  872. :run.py-mark - [0:0]
  873. [0:0] -A PREROUTING -j run.py-PREROUTING
  874. [0:0] -A INPUT -j run.py-INPUT
  875. [0:0] -A FORWARD -j run.py-FORWARD
  876. [0:0] -A OUTPUT -j run.py-OUTPUT
  877. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  878. [0:0] -A run.py-PREROUTING -j run.py-mark
  879. [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  880. COMMIT
  881. # Completed on Mon Apr 20 09:24:54 2015
  882. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  883. *filter
  884. :INPUT ACCEPT [0:0]
  885. :FORWARD ACCEPT [0:0]
  886. :neutron-filter-top - [0:0]
  887. :run.py-FORWARD - [0:0]
  888. :run.py-INPUT - [0:0]
  889. :run.py-OUTPUT - [0:0]
  890. :run.py-local - [0:0]
  891. [0:0] -A FORWARD -j neutron-filter-top
  892. [0:0] -A OUTPUT -j neutron-filter-top
  893. [0:0] -A neutron-filter-top -j run.py-local
  894. [0:0] -A INPUT -j run.py-INPUT
  895. [0:0] -A OUTPUT -j run.py-OUTPUT
  896. [0:0] -A FORWARD -j run.py-FORWARD
  897. :OUTPUT ACCEPT [0:0]
  898. COMMIT
  899. # Completed on Mon Apr 20 09:24:54 2015
  900.  
  901. Stdout:
  902. Stderr: Cannot open network namespace "qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0": No such file or directory
  903. "
  904.  
  905.  
  906. Captured traceback-4:
  907. ~~~~~~~~~~~~~~~~~~~~~
  908. Traceback (most recent call last):
  909. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  910. agent._router_removed(router_id)
  911. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  912. self, router=ri)
  913. File "neutron/callbacks/registry.py", line 44, in notify
  914. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  915. File "neutron/callbacks/manager.py", line 123, in notify
  916. raise exceptions.CallbackFailure(errors=errors)
  917. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  918. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'iptables-save', '-c']
  919. Exit code: 1
  920. Stdin:
  921. Stdout:
  922. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  923. "
  924.  
  925.  
  926. Captured traceback:
  927. ~~~~~~~~~~~~~~~~~~~
  928. Traceback (most recent call last):
  929. File "neutron/tests/functional/agent/test_l3_agent.py", line 551, in test_periodic_sync_routers_task
  930. routers_to_keep[i]['id']))
  931. File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/unittest2/case.py", line 678, in assertTrue
  932. raise self.failureException(msg)
  933. AssertionError: False is not true
  934.  
  935.  
  936. Captured stdout:
  937. ~~~~~~~~~~~~~~~~
  938. 2015-04-20 09:24:43.818 10268 INFO neutron.common.config [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Logging enabled!
  939. 2015-04-20 09:24:43.819 10268 INFO neutron.common.config [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  940. 2015-04-20 09:24:43.921 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  941. 2015-04-20 09:24:43.925 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  942. 2015-04-20 09:24:43.925 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  943. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  944. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  945. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent self.use_call)
  946. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  947. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  948. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  949. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  950. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  951. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  952. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  953. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  954. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  955. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  956. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  957. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent
  958. 2015-04-20 09:24:45.326 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  959. 2015-04-20 09:24:45.326 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  960. 2015-04-20 09:24:46.395 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  961. 2015-04-20 09:24:47.527 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  962. 2015-04-20 09:24:47.527 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  963. 2015-04-20 09:24:48.624 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  964. 2015-04-20 09:24:49.760 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  965. 2015-04-20 09:24:49.761 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  966. 2015-04-20 09:24:50.958 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  967. 2015-04-20 09:24:52.405 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  968. 2015-04-20 09:24:52.406 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  969. 2015-04-20 09:24:53.700 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_update
  970. 2015-04-20 09:24:53.945 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_update
  971. 2015-04-20 09:24:53.956 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_update
  972. 2015-04-20 09:24:53.970 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  973. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  974. Exit code: 1
  975. Stdin:
  976. Stdout:
  977. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  978.  
  979. 2015-04-20 09:24:53.971 10268 ERROR neutron.agent.l3.router_info [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  980. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  981. Exit code: 1
  982. Stdin:
  983. Stdout:
  984. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  985. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  986. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/common/utils.py", line 330, in call
  987. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
  988. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 605, in process
  989. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info self._process_internal_ports()
  990. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 385, in _process_internal_ports
  991. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info existing_devices = self._get_existing_devices()
  992. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 323, in _get_existing_devices
  993. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  994. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 102, in get_devices
  995. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error
  996. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/utils.py", line 137, in execute
  997. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info raise RuntimeError(m)
  998. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info RuntimeError:
  999. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  1000. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Exit code: 1
  1001. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Stdin:
  1002. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Stdout:
  1003. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1004. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info
  1005. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info
  1006. 2015-04-20 09:24:53.975 10268 ERROR neutron.agent.l3.agent [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Failed to process compatible router '8ba02e38-13ad-493b-be0e-cf33d6a6f815'
  1007. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1008. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 452, in _process_router_update
  1009. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
  1010. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 406, in _process_router_if_compatible
  1011. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self._process_updated_router(router)
  1012. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 420, in _process_updated_router
  1013. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent ri.process(self)
  1014. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/common/utils.py", line 333, in call
  1015. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self.logger(e)
  1016. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  1017. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
  1018. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/common/utils.py", line 330, in call
  1019. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
  1020. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/router_info.py", line 605, in process
  1021. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self._process_internal_ports()
  1022. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/router_info.py", line 385, in _process_internal_ports
  1023. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent existing_devices = self._get_existing_devices()
  1024. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/router_info.py", line 323, in _get_existing_devices
  1025. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  1026. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/linux/ip_lib.py", line 102, in get_devices
  1027. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error
  1028. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/linux/utils.py", line 137, in execute
  1029. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent raise RuntimeError(m)
  1030. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent RuntimeError:
  1031. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  1032. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Exit code: 1
  1033. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Stdin:
  1034. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Stdout:
  1035. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1036. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent
  1037. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent
  1038. 2015-04-20 09:24:53.977 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  1039. 2015-04-20 09:24:53.978 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1040. 2015-04-20 09:24:53.982 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1041. Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  1042. Exit code: 1
  1043. Stdin:
  1044. Stdout:
  1045. Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  1046.  
  1047. 2015-04-20 09:24:53.982 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1048. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1049. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1050. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1051. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1052. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1053. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1054. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager self._apply()
  1055. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1056. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1057. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  1058. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  1059. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1060. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1061. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager RuntimeError:
  1062. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  1063. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Exit code: 1
  1064. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Stdin:
  1065. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Stdout:
  1066. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  1067. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager
  1068. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager
  1069. 2015-04-20 09:24:53.983 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  1070. 2015-04-20 09:24:53.983 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  1071. 2015-04-20 09:24:53.984 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1072. 2015-04-20 09:24:53.987 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1073. Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  1074. Exit code: 1
  1075. Stdin:
  1076. Stdout:
  1077. Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  1078.  
  1079. 2015-04-20 09:24:53.988 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1080. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1081. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1082. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1083. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1084. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1085. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1086. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager self._apply()
  1087. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1088. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1089. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  1090. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  1091. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1092. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1093. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager RuntimeError:
  1094. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  1095. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Exit code: 1
  1096. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Stdin:
  1097. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Stdout:
  1098. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  1099. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager
  1100. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager
  1101. 2015-04-20 09:24:53.988 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  1102. 2015-04-20 09:24:54.000 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  1103. 2015-04-20 09:24:54.005 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1104. 2015-04-20 09:24:54.006 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-m mark --mark 0x1 -j ACCEPT' True False
  1105. 2015-04-20 09:24:54.006 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-p tcp -m tcp --dport 9697 -j DROP' True False
  1106. 2015-04-20 09:24:54.006 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff' True False
  1107. 2015-04-20 09:24:54.007 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-port 9697' True False
  1108. 2015-04-20 09:24:54.015 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1109. Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  1110. Exit code: 1
  1111. Stdin:
  1112. Stdout:
  1113. Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  1114.  
  1115. 2015-04-20 09:24:54.016 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1116. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1117. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1118. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1119. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1120. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1121. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1122. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager self._apply()
  1123. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1124. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1125. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  1126. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  1127. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1128. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1129. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager RuntimeError:
  1130. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  1131. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Exit code: 1
  1132. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Stdin:
  1133. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Stdout:
  1134. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  1135. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager
  1136. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager
  1137. 2015-04-20 09:24:54.017 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  1138. 2015-04-20 09:24:54.018 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  1139. 2015-04-20 09:24:54.020 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1140. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-m mark --mark 0x1 -j ACCEPT' True False
  1141. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-p tcp -m tcp --dport 9697 -j DROP' True False
  1142. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff' True False
  1143. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-port 9697' True False
  1144. 2015-04-20 09:24:54.034 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1145. Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  1146. Exit code: 1
  1147. Stdin:
  1148. Stdout:
  1149. Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  1150.  
  1151. 2015-04-20 09:24:54.035 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1152. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1153. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1154. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1155. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1156. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1157. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1158. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager self._apply()
  1159. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1160. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1161. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  1162. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  1163. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1164. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1165. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager RuntimeError:
  1166. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  1167. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Exit code: 1
  1168. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Stdin:
  1169. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Stdout:
  1170. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  1171. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager
  1172. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager
  1173. 2015-04-20 09:24:54.036 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  1174. 2015-04-20 09:24:54.037 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  1175. 2015-04-20 09:24:54.038 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1176. 2015-04-20 09:24:54.083 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1177. Command: ['ip', 'netns', 'exec', 'qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0', 'iptables-restore', '-c']
  1178. Exit code: 1
  1179. Stdin: # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1180. *raw
  1181. :PREROUTING ACCEPT [0:0]
  1182. :OUTPUT ACCEPT [0:0]
  1183. :run.py-OUTPUT - [0:0]
  1184. :run.py-PREROUTING - [0:0]
  1185. [0:0] -A PREROUTING -j run.py-PREROUTING
  1186. [0:0] -A OUTPUT -j run.py-OUTPUT
  1187. COMMIT
  1188. # Completed on Mon Apr 20 09:24:54 2015
  1189. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1190. *nat
  1191. :PREROUTING ACCEPT [0:0]
  1192. :INPUT ACCEPT [0:0]
  1193. :OUTPUT ACCEPT [0:0]
  1194. :POSTROUTING ACCEPT [0:0]
  1195. :neutron-postrouting-bottom - [0:0]
  1196. :run.py-OUTPUT - [0:0]
  1197. :run.py-POSTROUTING - [0:0]
  1198. :run.py-PREROUTING - [0:0]
  1199. :run.py-float-snat - [0:0]
  1200. :run.py-snat - [0:0]
  1201. [0:0] -A PREROUTING -j run.py-PREROUTING
  1202. [0:0] -A OUTPUT -j run.py-OUTPUT
  1203. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1204. [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  1205. [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  1206. [0:0] -A run.py-snat -j run.py-float-snat
  1207. [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  1208. [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  1209. [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  1210. [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1211. [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1212. [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  1213. COMMIT
  1214. # Completed on Mon Apr 20 09:24:54 2015
  1215. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1216. *mangle
  1217. :PREROUTING ACCEPT [0:0]
  1218. :INPUT ACCEPT [0:0]
  1219. :FORWARD ACCEPT [0:0]
  1220. :OUTPUT ACCEPT [0:0]
  1221. :POSTROUTING ACCEPT [0:0]
  1222. :run.py-FORWARD - [0:0]
  1223. :run.py-INPUT - [0:0]
  1224. :run.py-OUTPUT - [0:0]
  1225. :run.py-POSTROUTING - [0:0]
  1226. :run.py-PREROUTING - [0:0]
  1227. :run.py-mark - [0:0]
  1228. [0:0] -A PREROUTING -j run.py-PREROUTING
  1229. [0:0] -A INPUT -j run.py-INPUT
  1230. [0:0] -A FORWARD -j run.py-FORWARD
  1231. [0:0] -A OUTPUT -j run.py-OUTPUT
  1232. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1233. [0:0] -A run.py-PREROUTING -j run.py-mark
  1234. [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  1235. COMMIT
  1236. # Completed on Mon Apr 20 09:24:54 2015
  1237. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1238. *filter
  1239. :INPUT ACCEPT [0:0]
  1240. :FORWARD ACCEPT [0:0]
  1241. :neutron-filter-top - [0:0]
  1242. :run.py-FORWARD - [0:0]
  1243. :run.py-INPUT - [0:0]
  1244. :run.py-OUTPUT - [0:0]
  1245. :run.py-local - [0:0]
  1246. [0:0] -A FORWARD -j neutron-filter-top
  1247. [0:0] -A OUTPUT -j neutron-filter-top
  1248. [0:0] -A neutron-filter-top -j run.py-local
  1249. [0:0] -A INPUT -j run.py-INPUT
  1250. [0:0] -A OUTPUT -j run.py-OUTPUT
  1251. [0:0] -A FORWARD -j run.py-FORWARD
  1252. :OUTPUT ACCEPT [0:0]
  1253. COMMIT
  1254. # Completed on Mon Apr 20 09:24:54 2015
  1255.  
  1256. Stdout:
  1257. Stderr: Cannot open network namespace "qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0": No such file or directory
  1258.  
  1259. 2015-04-20 09:24:54.085 10268 ERROR neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] IPTablesManager.apply failed to apply the following set of iptables rules:
  1260. 1. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1261. 2. *raw
  1262. 3. :PREROUTING ACCEPT [0:0]
  1263. 4. :OUTPUT ACCEPT [0:0]
  1264. 5. :run.py-OUTPUT - [0:0]
  1265. 6. :run.py-PREROUTING - [0:0]
  1266. 7. [0:0] -A PREROUTING -j run.py-PREROUTING
  1267. 8. [0:0] -A OUTPUT -j run.py-OUTPUT
  1268. 9. COMMIT
  1269. 10. # Completed on Mon Apr 20 09:24:54 2015
  1270. 11. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1271. 12. *nat
  1272. 13. :PREROUTING ACCEPT [0:0]
  1273. 14. :INPUT ACCEPT [0:0]
  1274. 15. :OUTPUT ACCEPT [0:0]
  1275. 16. :POSTROUTING ACCEPT [0:0]
  1276. 17. :neutron-postrouting-bottom - [0:0]
  1277. 18. :run.py-OUTPUT - [0:0]
  1278. 19. :run.py-POSTROUTING - [0:0]
  1279. 20. :run.py-PREROUTING - [0:0]
  1280. 21. :run.py-float-snat - [0:0]
  1281. 22. :run.py-snat - [0:0]
  1282. 23. [0:0] -A PREROUTING -j run.py-PREROUTING
  1283. 24. [0:0] -A OUTPUT -j run.py-OUTPUT
  1284. 25. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1285. 26. [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  1286. 27. [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  1287. 28. [0:0] -A run.py-snat -j run.py-float-snat
  1288. 29. [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  1289. 30. [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  1290. 31. [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  1291. 32. [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1292. 33. [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1293. 34. [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  1294. 35. COMMIT
  1295. 36. # Completed on Mon Apr 20 09:24:54 2015
  1296. 37. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1297. 38. *mangle
  1298. 39. :PREROUTING ACCEPT [0:0]
  1299. 40. :INPUT ACCEPT [0:0]
  1300. 41. :FORWARD ACCEPT [0:0]
  1301. 42. :OUTPUT ACCEPT [0:0]
  1302. 43. :POSTROUTING ACCEPT [0:0]
  1303. 44. :run.py-FORWARD - [0:0]
  1304. 45. :run.py-INPUT - [0:0]
  1305. 46. :run.py-OUTPUT - [0:0]
  1306. 47. :run.py-POSTROUTING - [0:0]
  1307. 48. :run.py-PREROUTING - [0:0]
  1308. 49. :run.py-mark - [0:0]
  1309. 50. [0:0] -A PREROUTING -j run.py-PREROUTING
  1310. 51. [0:0] -A INPUT -j run.py-INPUT
  1311. 52. [0:0] -A FORWARD -j run.py-FORWARD
  1312. 53. [0:0] -A OUTPUT -j run.py-OUTPUT
  1313. 54. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1314. 55. [0:0] -A run.py-PREROUTING -j run.py-mark
  1315. 56. [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  1316. 57. COMMIT
  1317. 58. # Completed on Mon Apr 20 09:24:54 2015
  1318. 59. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1319. 60. *filter
  1320. 61. :INPUT ACCEPT [0:0]
  1321. 62. :FORWARD ACCEPT [0:0]
  1322. 63. :neutron-filter-top - [0:0]
  1323. 64. :run.py-FORWARD - [0:0]
  1324. 65. :run.py-INPUT - [0:0]
  1325. 66. :run.py-OUTPUT - [0:0]
  1326. 67. :run.py-local - [0:0]
  1327. 68. [0:0] -A FORWARD -j neutron-filter-top
  1328. 69. [0:0] -A OUTPUT -j neutron-filter-top
  1329. 70. [0:0] -A neutron-filter-top -j run.py-local
  1330. 71. [0:0] -A INPUT -j run.py-INPUT
  1331. 72. [0:0] -A OUTPUT -j run.py-OUTPUT
  1332. 73. [0:0] -A FORWARD -j run.py-FORWARD
  1333. 74. :OUTPUT ACCEPT [0:0]
  1334. 75. COMMIT
  1335. 76. # Completed on Mon Apr 20 09:24:54 2015
  1336. 77.
  1337. 2015-04-20 09:24:54.086 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1338. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1339. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1340. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1341. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1342. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1343. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1344. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager self._apply()
  1345. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1346. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1347. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 473, in _apply_synchronized
  1348. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager '\n'.join(log_lines))
  1349. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  1350. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager six.reraise(self.type_, self.value, self.tb)
  1351. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 452, in _apply_synchronized
  1352. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager run_as_root=True)
  1353. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1354. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1355. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager RuntimeError:
  1356. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0', 'iptables-restore', '-c']
  1357. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Exit code: 1
  1358. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Stdin: # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1359. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *raw
  1360. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :PREROUTING ACCEPT [0:0]
  1361. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  1362. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  1363. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-PREROUTING - [0:0]
  1364. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A PREROUTING -j run.py-PREROUTING
  1365. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  1366. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  1367. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  1368. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1369. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *nat
  1370. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :PREROUTING ACCEPT [0:0]
  1371. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :INPUT ACCEPT [0:0]
  1372. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  1373. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :POSTROUTING ACCEPT [0:0]
  1374. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :neutron-postrouting-bottom - [0:0]
  1375. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  1376. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-POSTROUTING - [0:0]
  1377. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-PREROUTING - [0:0]
  1378. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-float-snat - [0:0]
  1379. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-snat - [0:0]
  1380. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A PREROUTING -j run.py-PREROUTING
  1381. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  1382. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1383. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  1384. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  1385. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-snat -j run.py-float-snat
  1386. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  1387. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  1388. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  1389. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1390. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1391. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  1392. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  1393. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  1394. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1395. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *mangle
  1396. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :PREROUTING ACCEPT [0:0]
  1397. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :INPUT ACCEPT [0:0]
  1398. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :FORWARD ACCEPT [0:0]
  1399. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  1400. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :POSTROUTING ACCEPT [0:0]
  1401. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-FORWARD - [0:0]
  1402. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-INPUT - [0:0]
  1403. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  1404. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-POSTROUTING - [0:0]
  1405. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-PREROUTING - [0:0]
  1406. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-mark - [0:0]
  1407. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A PREROUTING -j run.py-PREROUTING
  1408. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A INPUT -j run.py-INPUT
  1409. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A FORWARD -j run.py-FORWARD
  1410. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  1411. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1412. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-PREROUTING -j run.py-mark
  1413. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  1414. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  1415. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  1416. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1417. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *filter
  1418. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :INPUT ACCEPT [0:0]
  1419. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :FORWARD ACCEPT [0:0]
  1420. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :neutron-filter-top - [0:0]
  1421. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-FORWARD - [0:0]
  1422. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-INPUT - [0:0]
  1423. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  1424. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-local - [0:0]
  1425. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A FORWARD -j neutron-filter-top
  1426. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j neutron-filter-top
  1427. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A neutron-filter-top -j run.py-local
  1428. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A INPUT -j run.py-INPUT
  1429. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  1430. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A FORWARD -j run.py-FORWARD
  1431. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  1432. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  1433. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  1434. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager
  1435. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Stdout:
  1436. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0": No such file or directory
  1437. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager
  1438. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager
  1439. 2015-04-20 09:24:54.087 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  1440. 2015-04-20 09:24:54.088 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  1441. 2015-04-20 09:24:54.089 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1442. 2015-04-20 09:24:54.094 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1443. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'iptables-save', '-c']
  1444. Exit code: 1
  1445. Stdin:
  1446. Stdout:
  1447. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1448.  
  1449. 2015-04-20 09:24:54.095 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1450. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1451. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1452. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1453. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1454. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1455. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1456. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager self._apply()
  1457. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1458. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1459. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  1460. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  1461. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1462. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1463. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager RuntimeError:
  1464. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'iptables-save', '-c']
  1465. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Exit code: 1
  1466. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Stdin:
  1467. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Stdout:
  1468. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1469. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager
  1470. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager
  1471. 2015-04-20 09:24:54.095 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  1472.  
  1473. {1} neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_without_snat_with_fips [2.244684s] ... FAILED
  1474.  
  1475. Captured traceback-1:
  1476. ~~~~~~~~~~~~~~~~~~~~~
  1477. Traceback (most recent call last):
  1478. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  1479. agent._router_removed(router_id)
  1480. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  1481. self, router=ri)
  1482. File "neutron/callbacks/registry.py", line 44, in notify
  1483. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  1484. File "neutron/callbacks/manager.py", line 123, in notify
  1485. raise exceptions.CallbackFailure(errors=errors)
  1486. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  1487. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'iptables-save', '-c']
  1488. Exit code: 1
  1489. Stdin:
  1490. Stdout:
  1491. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1492. "
  1493.  
  1494.  
  1495. Captured traceback:
  1496. ~~~~~~~~~~~~~~~~~~~
  1497. Traceback (most recent call last):
  1498. File "neutron/tests/functional/agent/test_l3_agent.py", line 878, in test_dvr_router_lifecycle_without_ha_without_snat_with_fips
  1499. self._dvr_router_lifecycle(enable_ha=False, enable_snat=False)
  1500. File "neutron/tests/functional/agent/test_l3_agent.py", line 963, in _dvr_router_lifecycle
  1501. router = self.manage_router(self.agent, router_info)
  1502. File "neutron/tests/functional/agent/test_l3_agent.py", line 136, in manage_router
  1503. ri = self._create_router(agent, router)
  1504. File "neutron/tests/functional/agent/test_l3_agent.py", line 140, in _create_router
  1505. agent._process_added_router(router)
  1506. File "neutron/agent/l3/agent.py", line 412, in _process_added_router
  1507. ri.process(self)
  1508. File "neutron/agent/l3/dvr_router.py", line 523, in process
  1509. super(DvrRouter, self).process(agent)
  1510. File "neutron/common/utils.py", line 333, in call
  1511. self.logger(e)
  1512. File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  1513. six.reraise(self.type_, self.value, self.tb)
  1514. File "neutron/common/utils.py", line 330, in call
  1515. return func(*args, **kwargs)
  1516. File "neutron/agent/l3/router_info.py", line 606, in process
  1517. self.process_external(agent)
  1518. File "neutron/agent/l3/dvr_router.py", line 492, in process_external
  1519. self.create_dvr_fip_interfaces(ex_gw_port)
  1520. File "neutron/agent/l3/dvr_router.py", line 511, in create_dvr_fip_interfaces
  1521. self.fip_ns.create_rtr_2_fip_link(self)
  1522. File "neutron/agent/l3/dvr_fip_ns.py", line 195, in create_rtr_2_fip_link
  1523. fip_ns_name)
  1524. File "neutron/agent/linux/ip_lib.py", line 130, in add_veth
  1525. self._as_root([], 'link', tuple(args))
  1526. File "neutron/agent/linux/ip_lib.py", line 69, in _as_root
  1527. log_fail_as_error=self.log_fail_as_error)
  1528. File "neutron/agent/linux/ip_lib.py", line 78, in _execute
  1529. log_fail_as_error=log_fail_as_error)
  1530. File "neutron/agent/linux/utils.py", line 137, in execute
  1531. raise RuntimeError(m)
  1532. RuntimeError:
  1533. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  1534. Exit code: 1
  1535. Stdin:
  1536. Stdout:
  1537. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1538.  
  1539.  
  1540.  
  1541. Captured stdout:
  1542. ~~~~~~~~~~~~~~~~
  1543. 2015-04-20 09:24:52.139 10266 INFO neutron.common.config [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Logging enabled!
  1544. 2015-04-20 09:24:52.139 10266 INFO neutron.common.config [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  1545. 2015-04-20 09:24:52.268 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Notify callbacks for router, before_create
  1546. 2015-04-20 09:24:52.273 10266 INFO eventlet.wsgi.server [-] (10266) wsgi starting up on http:///:t/
  1547. 2015-04-20 09:24:52.273 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  1548. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1549. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  1550. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent self.use_call)
  1551. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  1552. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  1553. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  1554. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  1555. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  1556. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  1557. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  1558. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  1559. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  1560. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  1561. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  1562. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent
  1563. 2015-04-20 09:24:52.797 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  1564. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1565. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  1566. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent self.use_call)
  1567. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  1568. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  1569. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  1570. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  1571. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  1572. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  1573. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  1574. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  1575. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  1576. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  1577. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  1578. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent
  1579. 2015-04-20 09:24:54.232 10266 ERROR neutron.agent.linux.utils [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -]
  1580. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  1581. Exit code: 1
  1582. Stdin:
  1583. Stdout:
  1584. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1585.  
  1586. 2015-04-20 09:24:54.233 10266 ERROR neutron.agent.l3.router_info [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -]
  1587. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  1588. Exit code: 1
  1589. Stdin:
  1590. Stdout:
  1591. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1592. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  1593. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/common/utils.py", line 330, in call
  1594. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
  1595. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 606, in process
  1596. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self.process_external(agent)
  1597. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/dvr_router.py", line 492, in process_external
  1598. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self.create_dvr_fip_interfaces(ex_gw_port)
  1599. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/dvr_router.py", line 511, in create_dvr_fip_interfaces
  1600. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self.fip_ns.create_rtr_2_fip_link(self)
  1601. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/dvr_fip_ns.py", line 195, in create_rtr_2_fip_link
  1602. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info fip_ns_name)
  1603. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 130, in add_veth
  1604. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self._as_root([], 'link', tuple(args))
  1605. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 69, in _as_root
  1606. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
  1607. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 78, in _execute
  1608. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
  1609. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/utils.py", line 137, in execute
  1610. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info raise RuntimeError(m)
  1611. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info RuntimeError:
  1612. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  1613. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Exit code: 1
  1614. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Stdin:
  1615. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Stdout:
  1616. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1617. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info
  1618. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info
  1619. 2015-04-20 09:24:54.246 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Notify callbacks for router, before_delete
  1620. 2015-04-20 09:24:54.246 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1621. 2015-04-20 09:24:54.246 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'INPUT' '-m mark --mark 0x1 -j ACCEPT' True False
  1622. 2015-04-20 09:24:54.246 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'INPUT' '-p tcp -m tcp --dport 9697 -j DROP' True False
  1623. 2015-04-20 09:24:54.247 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff' True False
  1624. 2015-04-20 09:24:54.247 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-port 9697' True False
  1625. 2015-04-20 09:24:54.252 10266 ERROR neutron.agent.linux.utils [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -]
  1626. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'iptables-save', '-c']
  1627. Exit code: 1
  1628. Stdin:
  1629. Stdout:
  1630. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1631.  
  1632. 2015-04-20 09:24:54.252 10266 ERROR neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  1633. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Traceback (most recent call last):
  1634. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  1635. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  1636. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  1637. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  1638. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  1639. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager self._apply()
  1640. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  1641. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager return self._apply_synchronized()
  1642. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  1643. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  1644. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  1645. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager raise RuntimeError(m)
  1646. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager RuntimeError:
  1647. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'iptables-save', '-c']
  1648. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Exit code: 1
  1649. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Stdin:
  1650. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Stdout:
  1651. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  1652. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager
  1653. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager
  1654. 2015-04-20 09:24:54.255 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Notify callbacks for router, abort_delete
  1655.  
  1656. {0} neutron.tests.functional.agent.test_l3_agent.UnprivilegedUserMetadataL3AgentTestCase.test_access_to_metadata_proxy [3.519196s] ... ok
  1657.  
  1658. Captured stdout:
  1659. ~~~~~~~~~~~~~~~~
  1660. 2015-04-20 09:24:52.652 10265 INFO neutron.common.config [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] Logging enabled!
  1661. 2015-04-20 09:24:52.652 10265 INFO neutron.common.config [req-58a95b58-fe9d-4857-ae86-95aa43f029ae - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  1662. 2015-04-20 09:24:52.754 10265 INFO neutron.callbacks.manager [req-87c31dc5-c4a3-4bf1-9f4c-eb02b38ee03f - - - - -] Notify callbacks for router, before_create
  1663. 2015-04-20 09:24:52.757 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  1664. 2015-04-20 09:24:52.757 10265 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  1665. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1666. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  1667. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent self.use_call)
  1668. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  1669. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  1670. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  1671. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent retry=self.retry)
  1672. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  1673. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  1674. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  1675. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  1676. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  1677. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  1678. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  1679. 2015-04-20 09:24:52.757 10265 TRACE neutron.agent.l3.agent
  1680. 2015-04-20 09:24:54.096 10265 INFO neutron.callbacks.manager [req-87c31dc5-c4a3-4bf1-9f4c-eb02b38ee03f - - - - -] Notify callbacks for router, after_create
  1681. 2015-04-20 09:24:54.097 10265 INFO neutron.callbacks.manager [req-87c31dc5-c4a3-4bf1-9f4c-eb02b38ee03f - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  1682. 2015-04-20 09:24:55.232 10265 INFO eventlet.wsgi.server [-] (10265) wsgi starting up on http:///:t/
  1683. 2015-04-20 09:24:55.519 10265 INFO eventlet.wsgi.server [-] (10265) accepted ''
  1684. 2015-04-20 09:24:55.521 10265 INFO eventlet.wsgi.server [-] 35.4.0.5,<local> - - [20/Apr/2015 09:24:55] "GET / HTTP/1.1" 200 115 0.001658
  1685. 2015-04-20 09:24:55.568 10265 INFO eventlet.wsgi.server [-] (10265) wsgi exited, is_accepting=True
  1686. 2015-04-20 09:24:55.568 10265 INFO neutron.callbacks.manager [req-87c31dc5-c4a3-4bf1-9f4c-eb02b38ee03f - - - - -] Notify callbacks for router, before_delete
  1687. 2015-04-20 09:24:55.569 10265 INFO neutron.callbacks.manager [req-87c31dc5-c4a3-4bf1-9f4c-eb02b38ee03f - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  1688. 2015-04-20 09:24:56.076 10265 INFO neutron.callbacks.manager [req-87c31dc5-c4a3-4bf1-9f4c-eb02b38ee03f - - - - -] Notify callbacks for router, after_delete
  1689.  
  1690. {3} neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_add_internal_network_set_arp_cache [2.436717s] ... ok
  1691.  
  1692. Captured stdout:
  1693. ~~~~~~~~~~~~~~~~
  1694. 2015-04-20 09:24:54.311 10268 INFO neutron.common.config [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Logging enabled!
  1695. 2015-04-20 09:24:54.311 10268 INFO neutron.common.config [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  1696. 2015-04-20 09:24:54.424 10268 INFO neutron.callbacks.manager [req-28d6a2b3-0509-48fc-b9af-77ba631f0ce7 - - - - -] Notify callbacks for router, before_create
  1697. 2015-04-20 09:24:54.429 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  1698. 2015-04-20 09:24:54.430 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  1699. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1700. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  1701. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent self.use_call)
  1702. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  1703. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  1704. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  1705. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  1706. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  1707. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  1708. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  1709. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  1710. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  1711. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  1712. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  1713. 2015-04-20 09:24:54.430 10268 TRACE neutron.agent.l3.agent
  1714. 2015-04-20 09:24:55.055 10268 ERROR neutron.agent.l3.dvr_router [req-28d6a2b3-0509-48fc-b9af-77ba631f0ce7 - - - - -] DVR: no map match_port found!
  1715. 2015-04-20 09:24:55.120 10268 ERROR neutron.agent.l3.dvr_router [req-28d6a2b3-0509-48fc-b9af-77ba631f0ce7 - - - - -] DVR: no map match_port found!
  1716. 2015-04-20 09:24:55.314 10268 INFO neutron.callbacks.manager [req-28d6a2b3-0509-48fc-b9af-77ba631f0ce7 - - - - -] Notify callbacks for router, after_create
  1717. 2015-04-20 09:24:55.315 10268 INFO neutron.callbacks.manager [req-28d6a2b3-0509-48fc-b9af-77ba631f0ce7 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  1718. 2015-04-20 09:24:56.227 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  1719. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1720. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  1721. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent self.use_call)
  1722. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  1723. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  1724. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  1725. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  1726. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  1727. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  1728. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  1729. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  1730. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  1731. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  1732. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  1733. 2015-04-20 09:24:56.227 10268 TRACE neutron.agent.l3.agent
  1734.  
  1735. {2} neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_namespace_exists [14.808632s] ... ok
  1736.  
  1737. Captured stderr:
  1738. ~~~~~~~~~~~~~~~~
  1739. /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/sqlalchemy/engine/default.py:442: Warning: Cannot load from mysql.proc. The table is probably corrupted
  1740. cursor.execute(statement, parameters)
  1741. /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/sqlalchemy/engine/default.py:442: Warning: Failed to open mysql.event
  1742. cursor.execute(statement, parameters)
  1743.  
  1744. Slowest Tests
  1745. Test id Runtime (s)
  1746. --------------------------------------------------------------------------------------------------------- -----------
  1747. neutron.tests.functional.agent.test_l3_agent.L3HATestFramework.test_ha_router_failover 22.563
  1748. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_keepalived_state_change_bulk_rpc 18.458
  1749. neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_namespace_exists 14.809
  1750. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_ipv6_ha_router_lifecycle 14.545
  1751. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_keepalived_state_change_notification 14.225
  1752. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_ha_router_lifecycle 14.035
  1753. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_ha_router_conf_on_restarted_agent 14.028
  1754. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_conntrack_disassociate_fip 11.458
  1755. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_periodic_sync_routers_task 10.432
  1756. neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_fips_for_multiple_ext_networks 8.995
  1757.  
  1758. ==============================
  1759. Failed 2 tests - output below:
  1760. ==============================
  1761.  
  1762. neutron.tests.functional.agent.test_l3_agent.L3AgentTestCase.test_periodic_sync_routers_task
  1763. --------------------------------------------------------------------------------------------
  1764.  
  1765. Captured traceback-1:
  1766. ~~~~~~~~~~~~~~~~~~~~~
  1767. Traceback (most recent call last):
  1768. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  1769. agent._router_removed(router_id)
  1770. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  1771. self, router=ri)
  1772. File "neutron/callbacks/registry.py", line 44, in notify
  1773. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  1774. File "neutron/callbacks/manager.py", line 123, in notify
  1775. raise exceptions.CallbackFailure(errors=errors)
  1776. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  1777. Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  1778. Exit code: 1
  1779. Stdin:
  1780. Stdout:
  1781. Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  1782. "
  1783.  
  1784.  
  1785. Captured traceback-2:
  1786. ~~~~~~~~~~~~~~~~~~~~~
  1787. Traceback (most recent call last):
  1788. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  1789. agent._router_removed(router_id)
  1790. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  1791. self, router=ri)
  1792. File "neutron/callbacks/registry.py", line 44, in notify
  1793. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  1794. File "neutron/callbacks/manager.py", line 123, in notify
  1795. raise exceptions.CallbackFailure(errors=errors)
  1796. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  1797. Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  1798. Exit code: 1
  1799. Stdin:
  1800. Stdout:
  1801. Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  1802. "
  1803.  
  1804.  
  1805. Captured traceback-3:
  1806. ~~~~~~~~~~~~~~~~~~~~~
  1807. Traceback (most recent call last):
  1808. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  1809. agent._router_removed(router_id)
  1810. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  1811. self, router=ri)
  1812. File "neutron/callbacks/registry.py", line 44, in notify
  1813. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  1814. File "neutron/callbacks/manager.py", line 123, in notify
  1815. raise exceptions.CallbackFailure(errors=errors)
  1816. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  1817. Command: ['ip', 'netns', 'exec', 'qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0', 'iptables-restore', '-c']
  1818. Exit code: 1
  1819. Stdin: # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1820. *raw
  1821. :PREROUTING ACCEPT [0:0]
  1822. :OUTPUT ACCEPT [0:0]
  1823. :run.py-OUTPUT - [0:0]
  1824. :run.py-PREROUTING - [0:0]
  1825. [0:0] -A PREROUTING -j run.py-PREROUTING
  1826. [0:0] -A OUTPUT -j run.py-OUTPUT
  1827. COMMIT
  1828. # Completed on Mon Apr 20 09:24:54 2015
  1829. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1830. *nat
  1831. :PREROUTING ACCEPT [0:0]
  1832. :INPUT ACCEPT [0:0]
  1833. :OUTPUT ACCEPT [0:0]
  1834. :POSTROUTING ACCEPT [0:0]
  1835. :neutron-postrouting-bottom - [0:0]
  1836. :run.py-OUTPUT - [0:0]
  1837. :run.py-POSTROUTING - [0:0]
  1838. :run.py-PREROUTING - [0:0]
  1839. :run.py-float-snat - [0:0]
  1840. :run.py-snat - [0:0]
  1841. [0:0] -A PREROUTING -j run.py-PREROUTING
  1842. [0:0] -A OUTPUT -j run.py-OUTPUT
  1843. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1844. [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  1845. [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  1846. [0:0] -A run.py-snat -j run.py-float-snat
  1847. [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  1848. [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  1849. [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  1850. [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1851. [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  1852. [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  1853. COMMIT
  1854. # Completed on Mon Apr 20 09:24:54 2015
  1855. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1856. *mangle
  1857. :PREROUTING ACCEPT [0:0]
  1858. :INPUT ACCEPT [0:0]
  1859. :FORWARD ACCEPT [0:0]
  1860. :OUTPUT ACCEPT [0:0]
  1861. :POSTROUTING ACCEPT [0:0]
  1862. :run.py-FORWARD - [0:0]
  1863. :run.py-INPUT - [0:0]
  1864. :run.py-OUTPUT - [0:0]
  1865. :run.py-POSTROUTING - [0:0]
  1866. :run.py-PREROUTING - [0:0]
  1867. :run.py-mark - [0:0]
  1868. [0:0] -A PREROUTING -j run.py-PREROUTING
  1869. [0:0] -A INPUT -j run.py-INPUT
  1870. [0:0] -A FORWARD -j run.py-FORWARD
  1871. [0:0] -A OUTPUT -j run.py-OUTPUT
  1872. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  1873. [0:0] -A run.py-PREROUTING -j run.py-mark
  1874. [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  1875. COMMIT
  1876. # Completed on Mon Apr 20 09:24:54 2015
  1877. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  1878. *filter
  1879. :INPUT ACCEPT [0:0]
  1880. :FORWARD ACCEPT [0:0]
  1881. :neutron-filter-top - [0:0]
  1882. :run.py-FORWARD - [0:0]
  1883. :run.py-INPUT - [0:0]
  1884. :run.py-OUTPUT - [0:0]
  1885. :run.py-local - [0:0]
  1886. [0:0] -A FORWARD -j neutron-filter-top
  1887. [0:0] -A OUTPUT -j neutron-filter-top
  1888. [0:0] -A neutron-filter-top -j run.py-local
  1889. [0:0] -A INPUT -j run.py-INPUT
  1890. [0:0] -A OUTPUT -j run.py-OUTPUT
  1891. [0:0] -A FORWARD -j run.py-FORWARD
  1892. :OUTPUT ACCEPT [0:0]
  1893. COMMIT
  1894. # Completed on Mon Apr 20 09:24:54 2015
  1895.  
  1896. Stdout:
  1897. Stderr: Cannot open network namespace "qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0": No such file or directory
  1898. "
  1899.  
  1900.  
  1901. Captured traceback-4:
  1902. ~~~~~~~~~~~~~~~~~~~~~
  1903. Traceback (most recent call last):
  1904. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  1905. agent._router_removed(router_id)
  1906. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  1907. self, router=ri)
  1908. File "neutron/callbacks/registry.py", line 44, in notify
  1909. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  1910. File "neutron/callbacks/manager.py", line 123, in notify
  1911. raise exceptions.CallbackFailure(errors=errors)
  1912. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  1913. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'iptables-save', '-c']
  1914. Exit code: 1
  1915. Stdin:
  1916. Stdout:
  1917. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1918. "
  1919.  
  1920.  
  1921. Captured traceback:
  1922. ~~~~~~~~~~~~~~~~~~~
  1923. Traceback (most recent call last):
  1924. File "neutron/tests/functional/agent/test_l3_agent.py", line 551, in test_periodic_sync_routers_task
  1925. routers_to_keep[i]['id']))
  1926. File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/unittest2/case.py", line 678, in assertTrue
  1927. raise self.failureException(msg)
  1928. AssertionError: False is not true
  1929.  
  1930.  
  1931. Captured stdout:
  1932. ~~~~~~~~~~~~~~~~
  1933. 2015-04-20 09:24:43.818 10268 INFO neutron.common.config [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] Logging enabled!
  1934. 2015-04-20 09:24:43.819 10268 INFO neutron.common.config [req-dee3f173-6dd8-4406-b574-1fdc433c097e - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  1935. 2015-04-20 09:24:43.921 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  1936. 2015-04-20 09:24:43.925 10268 INFO eventlet.wsgi.server [-] (10268) wsgi starting up on http:///:t/
  1937. 2015-04-20 09:24:43.925 10268 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  1938. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  1939. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  1940. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent self.use_call)
  1941. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  1942. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  1943. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  1944. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent retry=self.retry)
  1945. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  1946. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  1947. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  1948. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  1949. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  1950. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  1951. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  1952. 2015-04-20 09:24:43.925 10268 TRACE neutron.agent.l3.agent
  1953. 2015-04-20 09:24:45.326 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  1954. 2015-04-20 09:24:45.326 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  1955. 2015-04-20 09:24:46.395 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  1956. 2015-04-20 09:24:47.527 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  1957. 2015-04-20 09:24:47.527 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  1958. 2015-04-20 09:24:48.624 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  1959. 2015-04-20 09:24:49.760 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  1960. 2015-04-20 09:24:49.761 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  1961. 2015-04-20 09:24:50.958 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_create
  1962. 2015-04-20 09:24:52.405 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_create
  1963. 2015-04-20 09:24:52.406 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.after_router_added
  1964. 2015-04-20 09:24:53.700 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_update
  1965. 2015-04-20 09:24:53.945 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, after_update
  1966. 2015-04-20 09:24:53.956 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_update
  1967. 2015-04-20 09:24:53.970 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1968. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  1969. Exit code: 1
  1970. Stdin:
  1971. Stdout:
  1972. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1973.  
  1974. 2015-04-20 09:24:53.971 10268 ERROR neutron.agent.l3.router_info [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  1975. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  1976. Exit code: 1
  1977. Stdin:
  1978. Stdout:
  1979. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1980. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  1981. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/common/utils.py", line 330, in call
  1982. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
  1983. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 605, in process
  1984. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info self._process_internal_ports()
  1985. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 385, in _process_internal_ports
  1986. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info existing_devices = self._get_existing_devices()
  1987. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 323, in _get_existing_devices
  1988. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  1989. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 102, in get_devices
  1990. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error
  1991. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/utils.py", line 137, in execute
  1992. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info raise RuntimeError(m)
  1993. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info RuntimeError:
  1994. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  1995. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Exit code: 1
  1996. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Stdin:
  1997. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Stdout:
  1998. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  1999. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info
  2000. 2015-04-20 09:24:53.971 10268 TRACE neutron.agent.l3.router_info
  2001. 2015-04-20 09:24:53.975 10268 ERROR neutron.agent.l3.agent [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Failed to process compatible router '8ba02e38-13ad-493b-be0e-cf33d6a6f815'
  2002. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2003. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 452, in _process_router_update
  2004. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
  2005. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 406, in _process_router_if_compatible
  2006. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self._process_updated_router(router)
  2007. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 420, in _process_updated_router
  2008. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent ri.process(self)
  2009. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/common/utils.py", line 333, in call
  2010. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self.logger(e)
  2011. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2012. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
  2013. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/common/utils.py", line 330, in call
  2014. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
  2015. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/router_info.py", line 605, in process
  2016. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent self._process_internal_ports()
  2017. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/router_info.py", line 385, in _process_internal_ports
  2018. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent existing_devices = self._get_existing_devices()
  2019. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/l3/router_info.py", line 323, in _get_existing_devices
  2020. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2021. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2022. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error
  2023. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent File "neutron/agent/linux/utils.py", line 137, in execute
  2024. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent raise RuntimeError(m)
  2025. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent RuntimeError:
  2026. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2027. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Exit code: 1
  2028. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Stdin:
  2029. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Stdout:
  2030. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  2031. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent
  2032. 2015-04-20 09:24:53.975 10268 TRACE neutron.agent.l3.agent
  2033. 2015-04-20 09:24:53.977 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  2034. 2015-04-20 09:24:53.978 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2035. 2015-04-20 09:24:53.982 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  2036. Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  2037. Exit code: 1
  2038. Stdin:
  2039. Stdout:
  2040. Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  2041.  
  2042. 2015-04-20 09:24:53.982 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2043. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2044. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2045. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2046. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2047. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2048. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2049. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager self._apply()
  2050. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2051. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2052. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  2053. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  2054. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2055. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2056. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager RuntimeError:
  2057. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  2058. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Exit code: 1
  2059. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Stdin:
  2060. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Stdout:
  2061. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  2062. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager
  2063. 2015-04-20 09:24:53.982 10268 TRACE neutron.callbacks.manager
  2064. 2015-04-20 09:24:53.983 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  2065. 2015-04-20 09:24:53.983 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  2066. 2015-04-20 09:24:53.984 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2067. 2015-04-20 09:24:53.987 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  2068. Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  2069. Exit code: 1
  2070. Stdin:
  2071. Stdout:
  2072. Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  2073.  
  2074. 2015-04-20 09:24:53.988 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2075. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2076. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2077. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2078. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2079. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2080. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2081. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager self._apply()
  2082. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2083. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2084. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  2085. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  2086. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2087. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2088. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager RuntimeError:
  2089. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  2090. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Exit code: 1
  2091. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Stdin:
  2092. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Stdout:
  2093. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  2094. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager
  2095. 2015-04-20 09:24:53.988 10268 TRACE neutron.callbacks.manager
  2096. 2015-04-20 09:24:53.988 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  2097. 2015-04-20 09:24:54.000 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  2098. 2015-04-20 09:24:54.005 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2099. 2015-04-20 09:24:54.006 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-m mark --mark 0x1 -j ACCEPT' True False
  2100. 2015-04-20 09:24:54.006 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-p tcp -m tcp --dport 9697 -j DROP' True False
  2101. 2015-04-20 09:24:54.006 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff' True False
  2102. 2015-04-20 09:24:54.007 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-port 9697' True False
  2103. 2015-04-20 09:24:54.015 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  2104. Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  2105. Exit code: 1
  2106. Stdin:
  2107. Stdout:
  2108. Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  2109.  
  2110. 2015-04-20 09:24:54.016 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2111. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2112. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2113. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2114. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2115. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2116. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2117. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager self._apply()
  2118. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2119. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2120. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  2121. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  2122. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2123. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2124. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager RuntimeError:
  2125. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4', 'iptables-save', '-c']
  2126. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Exit code: 1
  2127. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Stdin:
  2128. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Stdout:
  2129. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-f4561330-c7ca-42b0-a3e1-c39afc1e03e4": No such file or directory
  2130. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager
  2131. 2015-04-20 09:24:54.016 10268 TRACE neutron.callbacks.manager
  2132. 2015-04-20 09:24:54.017 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  2133. 2015-04-20 09:24:54.018 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  2134. 2015-04-20 09:24:54.020 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2135. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-m mark --mark 0x1 -j ACCEPT' True False
  2136. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'INPUT' '-p tcp -m tcp --dport 9697 -j DROP' True False
  2137. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff' True False
  2138. 2015-04-20 09:24:54.021 10268 WARNING neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-port 9697' True False
  2139. 2015-04-20 09:24:54.034 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  2140. Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  2141. Exit code: 1
  2142. Stdin:
  2143. Stdout:
  2144. Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  2145.  
  2146. 2015-04-20 09:24:54.035 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2147. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2148. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2149. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2150. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2151. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2152. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2153. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager self._apply()
  2154. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2155. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2156. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  2157. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  2158. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2159. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2160. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager RuntimeError:
  2161. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58', 'iptables-save', '-c']
  2162. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Exit code: 1
  2163. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Stdin:
  2164. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Stdout:
  2165. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-e2c759eb-1834-4ea1-9951-9fd7b2659e58": No such file or directory
  2166. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager
  2167. 2015-04-20 09:24:54.035 10268 TRACE neutron.callbacks.manager
  2168. 2015-04-20 09:24:54.036 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  2169. 2015-04-20 09:24:54.037 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  2170. 2015-04-20 09:24:54.038 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2171. 2015-04-20 09:24:54.083 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  2172. Command: ['ip', 'netns', 'exec', 'qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0', 'iptables-restore', '-c']
  2173. Exit code: 1
  2174. Stdin: # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2175. *raw
  2176. :PREROUTING ACCEPT [0:0]
  2177. :OUTPUT ACCEPT [0:0]
  2178. :run.py-OUTPUT - [0:0]
  2179. :run.py-PREROUTING - [0:0]
  2180. [0:0] -A PREROUTING -j run.py-PREROUTING
  2181. [0:0] -A OUTPUT -j run.py-OUTPUT
  2182. COMMIT
  2183. # Completed on Mon Apr 20 09:24:54 2015
  2184. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2185. *nat
  2186. :PREROUTING ACCEPT [0:0]
  2187. :INPUT ACCEPT [0:0]
  2188. :OUTPUT ACCEPT [0:0]
  2189. :POSTROUTING ACCEPT [0:0]
  2190. :neutron-postrouting-bottom - [0:0]
  2191. :run.py-OUTPUT - [0:0]
  2192. :run.py-POSTROUTING - [0:0]
  2193. :run.py-PREROUTING - [0:0]
  2194. :run.py-float-snat - [0:0]
  2195. :run.py-snat - [0:0]
  2196. [0:0] -A PREROUTING -j run.py-PREROUTING
  2197. [0:0] -A OUTPUT -j run.py-OUTPUT
  2198. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  2199. [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  2200. [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  2201. [0:0] -A run.py-snat -j run.py-float-snat
  2202. [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  2203. [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  2204. [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  2205. [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  2206. [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  2207. [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  2208. COMMIT
  2209. # Completed on Mon Apr 20 09:24:54 2015
  2210. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2211. *mangle
  2212. :PREROUTING ACCEPT [0:0]
  2213. :INPUT ACCEPT [0:0]
  2214. :FORWARD ACCEPT [0:0]
  2215. :OUTPUT ACCEPT [0:0]
  2216. :POSTROUTING ACCEPT [0:0]
  2217. :run.py-FORWARD - [0:0]
  2218. :run.py-INPUT - [0:0]
  2219. :run.py-OUTPUT - [0:0]
  2220. :run.py-POSTROUTING - [0:0]
  2221. :run.py-PREROUTING - [0:0]
  2222. :run.py-mark - [0:0]
  2223. [0:0] -A PREROUTING -j run.py-PREROUTING
  2224. [0:0] -A INPUT -j run.py-INPUT
  2225. [0:0] -A FORWARD -j run.py-FORWARD
  2226. [0:0] -A OUTPUT -j run.py-OUTPUT
  2227. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  2228. [0:0] -A run.py-PREROUTING -j run.py-mark
  2229. [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  2230. COMMIT
  2231. # Completed on Mon Apr 20 09:24:54 2015
  2232. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2233. *filter
  2234. :INPUT ACCEPT [0:0]
  2235. :FORWARD ACCEPT [0:0]
  2236. :neutron-filter-top - [0:0]
  2237. :run.py-FORWARD - [0:0]
  2238. :run.py-INPUT - [0:0]
  2239. :run.py-OUTPUT - [0:0]
  2240. :run.py-local - [0:0]
  2241. [0:0] -A FORWARD -j neutron-filter-top
  2242. [0:0] -A OUTPUT -j neutron-filter-top
  2243. [0:0] -A neutron-filter-top -j run.py-local
  2244. [0:0] -A INPUT -j run.py-INPUT
  2245. [0:0] -A OUTPUT -j run.py-OUTPUT
  2246. [0:0] -A FORWARD -j run.py-FORWARD
  2247. :OUTPUT ACCEPT [0:0]
  2248. COMMIT
  2249. # Completed on Mon Apr 20 09:24:54 2015
  2250.  
  2251. Stdout:
  2252. Stderr: Cannot open network namespace "qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0": No such file or directory
  2253.  
  2254. 2015-04-20 09:24:54.085 10268 ERROR neutron.agent.linux.iptables_manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] IPTablesManager.apply failed to apply the following set of iptables rules:
  2255. 1. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2256. 2. *raw
  2257. 3. :PREROUTING ACCEPT [0:0]
  2258. 4. :OUTPUT ACCEPT [0:0]
  2259. 5. :run.py-OUTPUT - [0:0]
  2260. 6. :run.py-PREROUTING - [0:0]
  2261. 7. [0:0] -A PREROUTING -j run.py-PREROUTING
  2262. 8. [0:0] -A OUTPUT -j run.py-OUTPUT
  2263. 9. COMMIT
  2264. 10. # Completed on Mon Apr 20 09:24:54 2015
  2265. 11. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2266. 12. *nat
  2267. 13. :PREROUTING ACCEPT [0:0]
  2268. 14. :INPUT ACCEPT [0:0]
  2269. 15. :OUTPUT ACCEPT [0:0]
  2270. 16. :POSTROUTING ACCEPT [0:0]
  2271. 17. :neutron-postrouting-bottom - [0:0]
  2272. 18. :run.py-OUTPUT - [0:0]
  2273. 19. :run.py-POSTROUTING - [0:0]
  2274. 20. :run.py-PREROUTING - [0:0]
  2275. 21. :run.py-float-snat - [0:0]
  2276. 22. :run.py-snat - [0:0]
  2277. 23. [0:0] -A PREROUTING -j run.py-PREROUTING
  2278. 24. [0:0] -A OUTPUT -j run.py-OUTPUT
  2279. 25. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  2280. 26. [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  2281. 27. [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  2282. 28. [0:0] -A run.py-snat -j run.py-float-snat
  2283. 29. [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  2284. 30. [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  2285. 31. [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  2286. 32. [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  2287. 33. [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  2288. 34. [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  2289. 35. COMMIT
  2290. 36. # Completed on Mon Apr 20 09:24:54 2015
  2291. 37. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2292. 38. *mangle
  2293. 39. :PREROUTING ACCEPT [0:0]
  2294. 40. :INPUT ACCEPT [0:0]
  2295. 41. :FORWARD ACCEPT [0:0]
  2296. 42. :OUTPUT ACCEPT [0:0]
  2297. 43. :POSTROUTING ACCEPT [0:0]
  2298. 44. :run.py-FORWARD - [0:0]
  2299. 45. :run.py-INPUT - [0:0]
  2300. 46. :run.py-OUTPUT - [0:0]
  2301. 47. :run.py-POSTROUTING - [0:0]
  2302. 48. :run.py-PREROUTING - [0:0]
  2303. 49. :run.py-mark - [0:0]
  2304. 50. [0:0] -A PREROUTING -j run.py-PREROUTING
  2305. 51. [0:0] -A INPUT -j run.py-INPUT
  2306. 52. [0:0] -A FORWARD -j run.py-FORWARD
  2307. 53. [0:0] -A OUTPUT -j run.py-OUTPUT
  2308. 54. [0:0] -A POSTROUTING -j run.py-POSTROUTING
  2309. 55. [0:0] -A run.py-PREROUTING -j run.py-mark
  2310. 56. [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  2311. 57. COMMIT
  2312. 58. # Completed on Mon Apr 20 09:24:54 2015
  2313. 59. # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2314. 60. *filter
  2315. 61. :INPUT ACCEPT [0:0]
  2316. 62. :FORWARD ACCEPT [0:0]
  2317. 63. :neutron-filter-top - [0:0]
  2318. 64. :run.py-FORWARD - [0:0]
  2319. 65. :run.py-INPUT - [0:0]
  2320. 66. :run.py-OUTPUT - [0:0]
  2321. 67. :run.py-local - [0:0]
  2322. 68. [0:0] -A FORWARD -j neutron-filter-top
  2323. 69. [0:0] -A OUTPUT -j neutron-filter-top
  2324. 70. [0:0] -A neutron-filter-top -j run.py-local
  2325. 71. [0:0] -A INPUT -j run.py-INPUT
  2326. 72. [0:0] -A OUTPUT -j run.py-OUTPUT
  2327. 73. [0:0] -A FORWARD -j run.py-FORWARD
  2328. 74. :OUTPUT ACCEPT [0:0]
  2329. 75. COMMIT
  2330. 76. # Completed on Mon Apr 20 09:24:54 2015
  2331. 77.
  2332. 2015-04-20 09:24:54.086 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2333. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2334. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2335. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2336. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2337. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2338. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2339. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager self._apply()
  2340. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2341. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2342. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 473, in _apply_synchronized
  2343. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager '\n'.join(log_lines))
  2344. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2345. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager six.reraise(self.type_, self.value, self.tb)
  2346. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 452, in _apply_synchronized
  2347. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager run_as_root=True)
  2348. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2349. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2350. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager RuntimeError:
  2351. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0', 'iptables-restore', '-c']
  2352. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Exit code: 1
  2353. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Stdin: # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2354. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *raw
  2355. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :PREROUTING ACCEPT [0:0]
  2356. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  2357. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  2358. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-PREROUTING - [0:0]
  2359. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A PREROUTING -j run.py-PREROUTING
  2360. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  2361. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  2362. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  2363. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2364. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *nat
  2365. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :PREROUTING ACCEPT [0:0]
  2366. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :INPUT ACCEPT [0:0]
  2367. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  2368. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :POSTROUTING ACCEPT [0:0]
  2369. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :neutron-postrouting-bottom - [0:0]
  2370. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  2371. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-POSTROUTING - [0:0]
  2372. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-PREROUTING - [0:0]
  2373. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-float-snat - [0:0]
  2374. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-snat - [0:0]
  2375. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A PREROUTING -j run.py-PREROUTING
  2376. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  2377. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A POSTROUTING -j run.py-POSTROUTING
  2378. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A POSTROUTING -j neutron-postrouting-bottom
  2379. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A neutron-postrouting-bottom -j run.py-snat -m comment --comment "Perform source NAT on outgoing traffic."
  2380. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-snat -j run.py-float-snat
  2381. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-POSTROUTING ! -i qg-779deac0-de ! -o qg-779deac0-de -m conntrack ! --ctstate DNAT -j ACCEPT
  2382. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-snat -o qg-779deac0-de -j SNAT --to-source 19.4.4.4
  2383. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-snat -m mark ! --mark 0x2 -m conntrack --ctstate DNAT -j SNAT --to-source 19.4.4.4
  2384. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-PREROUTING -d 19.4.4.2 -j DNAT --to 10.0.0.1
  2385. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-OUTPUT -d 19.4.4.2 -j DNAT --to 10.0.0.1
  2386. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-float-snat -s 10.0.0.1 -j SNAT --to 19.4.4.2
  2387. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  2388. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  2389. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2390. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *mangle
  2391. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :PREROUTING ACCEPT [0:0]
  2392. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :INPUT ACCEPT [0:0]
  2393. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :FORWARD ACCEPT [0:0]
  2394. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  2395. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :POSTROUTING ACCEPT [0:0]
  2396. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-FORWARD - [0:0]
  2397. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-INPUT - [0:0]
  2398. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  2399. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-POSTROUTING - [0:0]
  2400. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-PREROUTING - [0:0]
  2401. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-mark - [0:0]
  2402. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A PREROUTING -j run.py-PREROUTING
  2403. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A INPUT -j run.py-INPUT
  2404. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A FORWARD -j run.py-FORWARD
  2405. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  2406. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A POSTROUTING -j run.py-POSTROUTING
  2407. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-PREROUTING -j run.py-mark
  2408. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A run.py-mark -i qg-779deac0-de -j MARK --set-xmark 0x2/0xffffffff
  2409. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  2410. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  2411. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Generated by iptables-save v1.4.19.1 on Mon Apr 20 09:24:54 2015
  2412. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager *filter
  2413. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :INPUT ACCEPT [0:0]
  2414. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :FORWARD ACCEPT [0:0]
  2415. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :neutron-filter-top - [0:0]
  2416. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-FORWARD - [0:0]
  2417. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-INPUT - [0:0]
  2418. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-OUTPUT - [0:0]
  2419. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :run.py-local - [0:0]
  2420. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A FORWARD -j neutron-filter-top
  2421. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j neutron-filter-top
  2422. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A neutron-filter-top -j run.py-local
  2423. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A INPUT -j run.py-INPUT
  2424. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A OUTPUT -j run.py-OUTPUT
  2425. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager [0:0] -A FORWARD -j run.py-FORWARD
  2426. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager :OUTPUT ACCEPT [0:0]
  2427. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager COMMIT
  2428. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager # Completed on Mon Apr 20 09:24:54 2015
  2429. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager
  2430. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Stdout:
  2431. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-06d92e8b-7355-490d-91ed-e0007b6951c0": No such file or directory
  2432. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager
  2433. 2015-04-20 09:24:54.086 10268 TRACE neutron.callbacks.manager
  2434. 2015-04-20 09:24:54.087 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  2435. 2015-04-20 09:24:54.088 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, before_delete
  2436. 2015-04-20 09:24:54.089 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2437. 2015-04-20 09:24:54.094 10268 ERROR neutron.agent.linux.utils [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -]
  2438. Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'iptables-save', '-c']
  2439. Exit code: 1
  2440. Stdin:
  2441. Stdout:
  2442. Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  2443.  
  2444. 2015-04-20 09:24:54.095 10268 ERROR neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2445. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2446. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2447. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2448. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2449. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2450. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2451. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager self._apply()
  2452. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2453. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2454. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  2455. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  2456. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2457. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2458. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager RuntimeError:
  2459. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815', 'iptables-save', '-c']
  2460. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Exit code: 1
  2461. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Stdin:
  2462. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Stdout:
  2463. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-8ba02e38-13ad-493b-be0e-cf33d6a6f815": No such file or directory
  2464. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager
  2465. 2015-04-20 09:24:54.095 10268 TRACE neutron.callbacks.manager
  2466. 2015-04-20 09:24:54.095 10268 INFO neutron.callbacks.manager [req-42a8e595-e638-4b18-994a-d810f11c0672 - - - - -] Notify callbacks for router, abort_delete
  2467.  
  2468.  
  2469. neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_without_snat_with_fips
  2470. ----------------------------------------------------------------------------------------------------------------------
  2471.  
  2472. Captured traceback-1:
  2473. ~~~~~~~~~~~~~~~~~~~~~
  2474. Traceback (most recent call last):
  2475. File "neutron/tests/functional/agent/test_l3_agent.py", line 144, in _delete_router
  2476. agent._router_removed(router_id)
  2477. File "neutron/agent/l3/agent.py", line 328, in _router_removed
  2478. self, router=ri)
  2479. File "neutron/callbacks/registry.py", line 44, in notify
  2480. _get_callback_manager().notify(resource, event, trigger, **kwargs)
  2481. File "neutron/callbacks/manager.py", line 123, in notify
  2482. raise exceptions.CallbackFailure(errors=errors)
  2483. neutron.callbacks.exceptions.CallbackFailure: Callback neutron.agent.metadata.driver.before_router_removed failed with "
  2484. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'iptables-save', '-c']
  2485. Exit code: 1
  2486. Stdin:
  2487. Stdout:
  2488. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2489. "
  2490.  
  2491.  
  2492. Captured traceback:
  2493. ~~~~~~~~~~~~~~~~~~~
  2494. Traceback (most recent call last):
  2495. File "neutron/tests/functional/agent/test_l3_agent.py", line 878, in test_dvr_router_lifecycle_without_ha_without_snat_with_fips
  2496. self._dvr_router_lifecycle(enable_ha=False, enable_snat=False)
  2497. File "neutron/tests/functional/agent/test_l3_agent.py", line 963, in _dvr_router_lifecycle
  2498. router = self.manage_router(self.agent, router_info)
  2499. File "neutron/tests/functional/agent/test_l3_agent.py", line 136, in manage_router
  2500. ri = self._create_router(agent, router)
  2501. File "neutron/tests/functional/agent/test_l3_agent.py", line 140, in _create_router
  2502. agent._process_added_router(router)
  2503. File "neutron/agent/l3/agent.py", line 412, in _process_added_router
  2504. ri.process(self)
  2505. File "neutron/agent/l3/dvr_router.py", line 523, in process
  2506. super(DvrRouter, self).process(agent)
  2507. File "neutron/common/utils.py", line 333, in call
  2508. self.logger(e)
  2509. File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2510. six.reraise(self.type_, self.value, self.tb)
  2511. File "neutron/common/utils.py", line 330, in call
  2512. return func(*args, **kwargs)
  2513. File "neutron/agent/l3/router_info.py", line 606, in process
  2514. self.process_external(agent)
  2515. File "neutron/agent/l3/dvr_router.py", line 492, in process_external
  2516. self.create_dvr_fip_interfaces(ex_gw_port)
  2517. File "neutron/agent/l3/dvr_router.py", line 511, in create_dvr_fip_interfaces
  2518. self.fip_ns.create_rtr_2_fip_link(self)
  2519. File "neutron/agent/l3/dvr_fip_ns.py", line 195, in create_rtr_2_fip_link
  2520. fip_ns_name)
  2521. File "neutron/agent/linux/ip_lib.py", line 130, in add_veth
  2522. self._as_root([], 'link', tuple(args))
  2523. File "neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2524. log_fail_as_error=self.log_fail_as_error)
  2525. File "neutron/agent/linux/ip_lib.py", line 78, in _execute
  2526. log_fail_as_error=log_fail_as_error)
  2527. File "neutron/agent/linux/utils.py", line 137, in execute
  2528. raise RuntimeError(m)
  2529. RuntimeError:
  2530. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  2531. Exit code: 1
  2532. Stdin:
  2533. Stdout:
  2534. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2535.  
  2536.  
  2537.  
  2538. Captured stdout:
  2539. ~~~~~~~~~~~~~~~~
  2540. 2015-04-20 09:24:52.139 10266 INFO neutron.common.config [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] Logging enabled!
  2541. 2015-04-20 09:24:52.139 10266 INFO neutron.common.config [req-0aecc6b9-7d1f-4475-9a6b-8c213182b3fa - - - - -] /opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/subunit/run.py version 2015.2.dev112
  2542. 2015-04-20 09:24:52.268 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Notify callbacks for router, before_create
  2543. 2015-04-20 09:24:52.273 10266 INFO eventlet.wsgi.server [-] (10266) wsgi starting up on http:///:t/
  2544. 2015-04-20 09:24:52.273 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  2545. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2546. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  2547. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent self.use_call)
  2548. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  2549. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  2550. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  2551. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  2552. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2553. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  2554. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  2555. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  2556. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  2557. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  2558. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  2559. 2015-04-20 09:24:52.273 10266 TRACE neutron.agent.l3.agent
  2560. 2015-04-20 09:24:52.797 10266 ERROR neutron.agent.l3.agent [-] Failed reporting state!
  2561. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2562. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "neutron/agent/l3/agent.py", line 594, in _report_state
  2563. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent self.use_call)
  2564. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "neutron/agent/rpc.py", line 80, in report_state
  2565. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
  2566. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  2567. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent retry=self.retry)
  2568. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2569. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
  2570. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 211, in send
  2571. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout)
  2572. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent File "/opt/openstack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 203, in _send
  2573. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent 'No reply on topic %s' % target.topic)
  2574. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent MessagingTimeout: No reply on topic q-plugin
  2575. 2015-04-20 09:24:52.797 10266 TRACE neutron.agent.l3.agent
  2576. 2015-04-20 09:24:54.232 10266 ERROR neutron.agent.linux.utils [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -]
  2577. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  2578. Exit code: 1
  2579. Stdin:
  2580. Stdout:
  2581. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2582.  
  2583. 2015-04-20 09:24:54.233 10266 ERROR neutron.agent.l3.router_info [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -]
  2584. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  2585. Exit code: 1
  2586. Stdin:
  2587. Stdout:
  2588. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2589. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  2590. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/common/utils.py", line 330, in call
  2591. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
  2592. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 606, in process
  2593. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self.process_external(agent)
  2594. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/dvr_router.py", line 492, in process_external
  2595. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self.create_dvr_fip_interfaces(ex_gw_port)
  2596. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/dvr_router.py", line 511, in create_dvr_fip_interfaces
  2597. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self.fip_ns.create_rtr_2_fip_link(self)
  2598. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/l3/dvr_fip_ns.py", line 195, in create_rtr_2_fip_link
  2599. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info fip_ns_name)
  2600. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 130, in add_veth
  2601. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info self._as_root([], 'link', tuple(args))
  2602. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2603. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
  2604. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 78, in _execute
  2605. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
  2606. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info File "neutron/agent/linux/utils.py", line 137, in execute
  2607. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info raise RuntimeError(m)
  2608. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info RuntimeError:
  2609. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'ip', 'link', 'add', 'rfp-2abddaeb-0', 'type', 'veth', 'peer', 'name', 'fpr-2abddaeb-0', 'netns', 'fip-ecd0c472-3db7-4839-a9fc-78568b39ee42']
  2610. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Exit code: 1
  2611. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Stdin:
  2612. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Stdout:
  2613. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2614. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info
  2615. 2015-04-20 09:24:54.233 10266 TRACE neutron.agent.l3.router_info
  2616. 2015-04-20 09:24:54.246 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Notify callbacks for router, before_delete
  2617. 2015-04-20 09:24:54.246 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Calling callback neutron.agent.metadata.driver.before_router_removed
  2618. 2015-04-20 09:24:54.246 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'INPUT' '-m mark --mark 0x1 -j ACCEPT' True False
  2619. 2015-04-20 09:24:54.246 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'INPUT' '-p tcp -m tcp --dport 9697 -j DROP' True False
  2620. 2015-04-20 09:24:54.247 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffffffff' True False
  2621. 2015-04-20 09:24:54.247 10266 WARNING neutron.agent.linux.iptables_manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Tried to remove rule that was not there: 'PREROUTING' '-d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-port 9697' True False
  2622. 2015-04-20 09:24:54.252 10266 ERROR neutron.agent.linux.utils [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -]
  2623. Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'iptables-save', '-c']
  2624. Exit code: 1
  2625. Stdin:
  2626. Stdout:
  2627. Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2628.  
  2629. 2015-04-20 09:24:54.252 10266 ERROR neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Error during notification for neutron.agent.metadata.driver.before_router_removed router, before_delete
  2630. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Traceback (most recent call last):
  2631. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/callbacks/manager.py", line 143, in _notify_loop
  2632. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager callback(resource, event, trigger, **kwargs)
  2633. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/metadata/driver.py", line 171, in before_router_removed
  2634. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager router.iptables_manager.apply()
  2635. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 408, in apply
  2636. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager self._apply()
  2637. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 418, in _apply
  2638. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager return self._apply_synchronized()
  2639. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/iptables_manager.py", line 438, in _apply_synchronized
  2640. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager all_tables = self.execute(args, run_as_root=True)
  2641. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager File "neutron/agent/linux/utils.py", line 137, in execute
  2642. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager raise RuntimeError(m)
  2643. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager RuntimeError:
  2644. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Command: ['ip', 'netns', 'exec', 'qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9', 'iptables-save', '-c']
  2645. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Exit code: 1
  2646. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Stdin:
  2647. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Stdout:
  2648. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager Stderr: Cannot open network namespace "qrouter-2abddaeb-07b3-4972-9c11-12447c57ecf9": No such file or directory
  2649. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager
  2650. 2015-04-20 09:24:54.252 10266 TRACE neutron.callbacks.manager
  2651. 2015-04-20 09:24:54.255 10266 INFO neutron.callbacks.manager [req-ebe4e479-59a2-4a3d-9d76-244d7e4433d4 - - - - -] Notify callbacks for router, abort_delete
  2652.  
  2653.  
  2654.  
  2655. ======
  2656. Totals
  2657. ======
  2658. Ran: 22 tests in 54.0000 sec.
  2659. - Passed: 20
  2660. - Skipped: 0
  2661. - Expected Fail: 0
  2662. - Unexpected Success: 0
  2663. - Failed: 2
  2664. Sum of execute time for each test: 192.7783 sec.
  2665.  
  2666. ==============
  2667. Worker Balance
  2668. ==============
  2669. - Worker 0 (5 tests) => 0:00:47.804172s
  2670. - Worker 1 (5 tests) => 0:00:46.174154s
  2671. - Worker 2 (6 tests) => 0:00:50.865232s
  2672. - Worker 3 (6 tests) => 0:00:48.041275s
  2673. ERROR: InvocationError: '/usr/bin/sh tools/pretty_tox.sh neutron.tests.functional.agent.test_l3_agent neutron.tests.fullstack'
  2674. ___________________________________ summary ____________________________________
  2675. ERROR: dsvm-functional: commands failed
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!