2021-09-18 16:17:09.463 5547 DEBUG nova.osapi_compute.wsgi.server [req-33cdee0f-a50c-4e2c-9a40-7d74726e49d0 - - - - -] (5547) accepted ('192.168.10.198', 36564) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
2021-09-18 16:17:09.677 5547 DEBUG nova.api.openstack.wsgi [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method ServiceController.delete of <nova.api.openstack.compute.services.ServiceController object at 0x7f15f7976ac8>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:09.681 5547 DEBUG oslo_concurrency.lockutils [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:09.681 5547 DEBUG oslo_concurrency.lockutils [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:09.685 5547 DEBUG oslo_concurrency.lockutils [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:09.686 5547 DEBUG oslo_concurrency.lockutils [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:09.689 5547 DEBUG oslo_concurrency.lockutils [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:09.689 5547 DEBUG oslo_concurrency.lockutils [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Unexpected exception in API method: ValueError: No such provider 9188f859-18f8-40b3-8897-0ad7ab572b43
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/services.py", line 288, in delete
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi     context, compute_node, cascade=True)
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 2173, in delete_resource_provider
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi     rp_uuid)
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/compute/provider_tree.py", line 290, in get_provider_uuids_in_tree
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi     name_or_uuid, return_root=True).get_provider_uuids()
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/compute/provider_tree.py", line 440, in _find_with_lock
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi     raise ValueError(_("No such provider %s") % name_or_uuid)
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi ValueError: No such provider 9188f859-18f8-40b3-8897-0ad7ab572b43
2021-09-18 16:17:09.732 5547 ERROR nova.api.openstack.wsgi
2021-09-18 16:17:09.732 5547 INFO nova.api.openstack.wsgi [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ValueError'>
2021-09-18 16:17:09.733 5547 DEBUG nova.api.openstack.wsgi [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ValueError'> __call__ /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:927
2021-09-18 16:17:09.734 5547 INFO nova.osapi_compute.wsgi.server [req-a41525ba-cfa1-4717-9aca-60dba8249aa2 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "DELETE /v2.1/os-services/23 HTTP/1.1" status: 500 len: 617 time: 0.2699182
2021-09-18 16:17:10.358 5551 DEBUG nova.api.openstack.wsgi [req-0434872a-18b6-4102-9f55-406c1799cb85 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method VersionsV2.index of <nova.api.openstack.compute.versions.VersionsV2 object at 0x7f15fd0da9e8>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:10.359 5551 INFO nova.osapi_compute.wsgi.server [req-0434872a-18b6-4102-9f55-406c1799cb85 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/ HTTP/1.1" status: 200 len: 761 time: 0.0018117
2021-09-18 16:17:11.384 5551 DEBUG nova.api.openstack.wsgi [req-be2675ec-bdd1-435d-b6bb-b2588d7878ff 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method FlavorsController.detail of <nova.api.openstack.compute.flavors.FlavorsController object at 0x7f15f79c3588>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.400 5551 INFO nova.osapi_compute.wsgi.server [req-be2675ec-bdd1-435d-b6bb-b2588d7878ff 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/flavors/detail HTTP/1.1" status: 200 len: 4922 time: 0.2463624
2021-09-18 16:17:11.408 5551 DEBUG nova.api.openstack.wsgi [req-39fb1e1a-382d-4a67-a4be-cfe2ee2c94e4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<function Controller.__getattribute__.<locals>.version_select at 0x7f15f70366a8>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.416 5551 DEBUG oslo_concurrency.lockutils [req-39fb1e1a-382d-4a67-a4be-cfe2ee2c94e4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.416 5551 DEBUG oslo_concurrency.lockutils [req-39fb1e1a-382d-4a67-a4be-cfe2ee2c94e4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.427 5551 DEBUG oslo_concurrency.lockutils [req-39fb1e1a-382d-4a67-a4be-cfe2ee2c94e4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.428 5551 DEBUG oslo_concurrency.lockutils [req-39fb1e1a-382d-4a67-a4be-cfe2ee2c94e4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.437 5551 INFO nova.osapi_compute.wsgi.server [req-39fb1e1a-382d-4a67-a4be-cfe2ee2c94e4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/os-simple-tenant-usage?detailed=1 HTTP/1.1" status: 200 len: 983 time: 0.0333683
2021-09-18 16:17:11.442 5551 DEBUG nova.api.openstack.wsgi [req-bc85f01b-f395-4618-b5ac-c272a5d0a17e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method AvailabilityZoneController.index of <nova.api.openstack.compute.availability_zone.AvailabilityZoneController object at 0x7f15f79fe6d8>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.445 5551 DEBUG oslo_concurrency.lockutils [req-bc85f01b-f395-4618-b5ac-c272a5d0a17e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.446 5551 DEBUG oslo_concurrency.lockutils [req-bc85f01b-f395-4618-b5ac-c272a5d0a17e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.446 5551 DEBUG oslo_concurrency.lockutils [req-bc85f01b-f395-4618-b5ac-c272a5d0a17e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.446 5551 DEBUG oslo_concurrency.lockutils [req-bc85f01b-f395-4618-b5ac-c272a5d0a17e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.466 5551 INFO nova.osapi_compute.wsgi.server [req-bc85f01b-f395-4618-b5ac-c272a5d0a17e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/os-availability-zone HTTP/1.1" status: 200 len: 473 time: 0.0281320
2021-09-18 16:17:11.471 5551 DEBUG nova.api.openstack.wsgi [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method ServiceController.index of <nova.api.openstack.compute.services.ServiceController object at 0x7f15f79762b0>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.475 5551 DEBUG oslo_concurrency.lockutils [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.475 5551 DEBUG oslo_concurrency.lockutils [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.476 5551 DEBUG oslo_concurrency.lockutils [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.476 5551 DEBUG oslo_concurrency.lockutils [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.492 5551 DEBUG nova.servicegroup.drivers.db [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Seems service nova-compute on host kvm-gpu-node1 is down. Last heartbeat was 2021-09-18 10:57:56. Elapsed time is 8355.492101 is_up /usr/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py:81
2021-09-18 16:17:11.493 5551 INFO nova.osapi_compute.wsgi.server [req-be6d469f-d799-4d96-a06f-520a3ed56d1e 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/os-services HTTP/1.1" status: 200 len: 1856 time: 0.0263097
2021-09-18 16:17:11.503 5551 DEBUG nova.api.openstack.wsgi [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<function Controller.__getattribute__.<locals>.version_select at 0x7f15f6e6f840>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.506 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.507 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.520 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.520 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.532 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.532 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.539 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.540 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.543 5551 DEBUG nova.servicegroup.drivers.db [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Seems service nova-compute on host kvm-gpu-node1 is down. Last heartbeat was 2021-09-18 10:57:56. Elapsed time is 8355.543765 is_up /usr/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py:81
2021-09-18 16:17:11.547 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.548 5551 DEBUG oslo_concurrency.lockutils [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.552 5551 INFO nova.osapi_compute.wsgi.server [req-0b24d894-c0ec-4c03-91b2-d4022f15e37a 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/os-hypervisors/detail HTTP/1.1" status: 200 len: 6085 time: 0.0583858
2021-09-18 16:17:11.558 5551 DEBUG nova.api.openstack.wsgi [req-abefd64a-12a7-4f2d-b0a8-8605162029bc 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method AggregateController.index of <nova.api.openstack.compute.aggregates.AggregateController object at 0x7f15f79dcbe0>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.567 5551 INFO nova.osapi_compute.wsgi.server [req-abefd64a-12a7-4f2d-b0a8-8605162029bc 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/os-aggregates HTTP/1.1" status: 200 len: 394 time: 0.0138795
2021-09-18 16:17:11.572 5551 DEBUG nova.api.openstack.wsgi [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x7f15f781cda0>>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.581 5551 DEBUG nova.compute.api [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Searching by: {'deleted': False} get_all /usr/lib/python3.6/site-packages/nova/compute/api.py:2852
2021-09-18 16:17:11.585 5551 DEBUG oslo_concurrency.lockutils [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.585 5551 DEBUG oslo_concurrency.lockutils [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.586 5551 DEBUG oslo_concurrency.lockutils [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.586 5551 DEBUG oslo_concurrency.lockutils [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.649 5551 DEBUG nova.compute.multi_cell_list [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Listed batch of 2 results from cell out of 1000 limit. Returned 2 total so far. do_query /usr/lib/python3.6/site-packages/nova/compute/multi_cell_list.py:376
2021-09-18 16:17:11.718 5551 DEBUG oslo_concurrency.lockutils [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.718 5551 DEBUG oslo_concurrency.lockutils [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.810 5551 INFO nova.osapi_compute.wsgi.server [req-47a819a3-e4e0-44d1-a43d-d618b6bf68c6 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/servers/detail?all_tenants=true HTTP/1.1" status: 200 len: 4618 time: 0.2424777
2021-09-18 16:17:11.820 5551 DEBUG nova.api.openstack.wsgi [req-6b53ad76-85d1-4329-a5fb-c7a18b3230db 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<function Controller.__getattribute__.<locals>.version_select at 0x7f15f66b38c8>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.820 5551 DEBUG nova.api.openstack.wsgi [req-6b53ad76-85d1-4329-a5fb-c7a18b3230db 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Returning 404 to user: The resource could not be found. __call__ /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:927
2021-09-18 16:17:11.821 5551 INFO nova.osapi_compute.wsgi.server [req-6b53ad76-85d1-4329-a5fb-c7a18b3230db 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/os-security-groups HTTP/1.1" status: 404 len: 476 time: 0.0090494
2021-09-18 16:17:11.973 5551 DEBUG nova.api.openstack.wsgi [req-b5af4440-b3bd-4526-8efd-659556e7a4d4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<function Controller.__getattribute__.<locals>.version_select at 0x7f15f6dc28c8>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:11.984 5551 DEBUG oslo_concurrency.lockutils [req-b5af4440-b3bd-4526-8efd-659556e7a4d4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.985 5551 DEBUG oslo_concurrency.lockutils [req-b5af4440-b3bd-4526-8efd-659556e7a4d4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.985 5551 DEBUG oslo_concurrency.lockutils [req-b5af4440-b3bd-4526-8efd-659556e7a4d4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:11.985 5551 DEBUG oslo_concurrency.lockutils [req-b5af4440-b3bd-4526-8efd-659556e7a4d4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:11.998 5551 INFO nova.osapi_compute.wsgi.server [req-b5af4440-b3bd-4526-8efd-659556e7a4d4 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/limits?tenant_id=94fe3887c422488690960f3d870be860 HTTP/1.1" status: 200 len: 679 time: 0.0306115
2021-09-18 16:17:12.003 5551 DEBUG nova.api.openstack.wsgi [req-94e6e580-bc02-43fc-8d1c-cab4c8d4fa0c 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Calling method '<function Controller.__getattribute__.<locals>.version_select at 0x7f15f6dc3048>' _process_stack /usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py:514
2021-09-18 16:17:12.013 5551 DEBUG oslo_concurrency.lockutils [req-94e6e580-bc02-43fc-8d1c-cab4c8d4fa0c 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:12.013 5551 DEBUG oslo_concurrency.lockutils [req-94e6e580-bc02-43fc-8d1c-cab4c8d4fa0c 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:12.014 5551 DEBUG oslo_concurrency.lockutils [req-94e6e580-bc02-43fc-8d1c-cab4c8d4fa0c 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2021-09-18 16:17:12.014 5551 DEBUG oslo_concurrency.lockutils [req-94e6e580-bc02-43fc-8d1c-cab4c8d4fa0c 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] Lock "6d2f70db-9982-4c51-8d4b-2c0c4b280ef8" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
2021-09-18 16:17:12.026 5551 INFO nova.osapi_compute.wsgi.server [req-94e6e580-bc02-43fc-8d1c-cab4c8d4fa0c 61d267ae52a044c4beb9ab7d9084e167 bbd7a134c6e7413b8afc532c9f223cfc - default default] 192.168.10.198 "GET /v2.1/limits?tenant_id=bbd7a134c6e7413b8afc532c9f223cfc HTTP/1.1" status: 200 len: 684 time: 0.0268764