2019-03-25 13:54:09.137 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:09.210 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Auditing locally available compute resources for hv01ve.worker.devgr1a.openstack.test.de (node: hv01ve.worker.devgr1a.openstack.test.de) update_available_resource /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:696
2019-03-25 13:54:09.561 1 DEBUG nova.virt.libvirt.driver [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] skipping disk for instance-0000007b as it does not have a path _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7389
2019-03-25 13:54:09.563 1 DEBUG nova.virt.libvirt.driver [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] skipping disk for instance-00000084 as it does not have a path _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7389
2019-03-25 13:54:09.565 1 DEBUG nova.virt.libvirt.driver [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] skipping disk for instance-0000006f as it does not have a path _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7389
2019-03-25 13:54:09.656 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Hypervisor/Node resource view: name=hv01ve.worker.devgr1a.openstack.test.de free_ram=7207MB free_disk=468GB free_vcpus=5 pci_devices=[{"dev_id": "pci_0000_00_08_0", "product_id": "1002", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1002", "address": "0000:00:08.0"}, {"dev_id": "pci_0000_00_07_0", "product_id": "1003", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1003", "address": "0000:00:07.0"}, {"dev_id": "pci_0000_00_09_0", "product_id": "1000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1000", "address": "0000:00:09.0"}, {"dev_id": "pci_0000_00_06_0", "product_id": "1004", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1004", "address": "0000:00:06.0"}, {"dev_id": "pci_0000_00_01_0", "product_id": "7000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7000", "address": "0000:00:01.0"}, {"dev_id": "pci_0000_00_05_2", "product_id": "2936", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2936", "address": "0000:00:05.2"}, {"dev_id": "pci_0000_00_05_0", "product_id": "2934", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2934", "address": "0000:00:05.0"}, {"dev_id": "pci_0000_00_01_1", "product_id": "7010", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7010", "address": "0000:00:01.1"}, {"dev_id": "pci_0000_00_05_1", "product_id": "2935", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2935", "address": "0000:00:05.1"}, {"dev_id": "pci_0000_00_03_0", "product_id": "1000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1000", "address": "0000:00:03.0"}, {"dev_id": "pci_0000_00_02_0", "product_id": "0100", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1b36", "label": "label_1b36_0100", "address": "0000:00:02.0"}, {"dev_id": "pci_0000_00_04_0", "product_id": "2668", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2668", "address": "0000:00:04.0"}, {"dev_id": "pci_0000_00_01_3", "product_id": "7113", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7113", "address": "0000:00:01.3"}, {"dev_id": "pci_0000_00_05_7", "product_id": "293a", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_293a", "address": "0000:00:05.7"}, {"dev_id": "pci_0000_00_00_0", "product_id": "1237", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_1237", "address": "0000:00:00.0"}, {"dev_id": "pci_0000_00_0a_0", "product_id": "1005", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1005", "address": "0000:00:0a.0"}] _report_hypervisor_resource_view /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:832
2019-03-25 13:54:09.657 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker._update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2019-03-25 13:54:09.739 1 DEBUG nova.scheduler.client.report [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Refreshing aggregate associations for resource provider 17f4e1c1-979c-4690-b839-c1715772afa0 _ensure_resource_provider /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:509
2019-03-25 13:54:09.948 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Compute driver doesn't require allocation refresh and we're on a compute host in a deployment that only has compute hosts with Nova versions >=16 (Pike). Skipping auto-correction of allocations.  _update_usage_from_instances /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1209
2019-03-25 13:54:09.995 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Instance d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 1024, u'DISK_GB': 10}}. _remove_deleted_instances_allocations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1230
2019-03-25 13:54:09.995 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Instance 91673651-94c0-4e85-9670-c671bec623e9 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 1024, u'DISK_GB': 10}}. _remove_deleted_instances_allocations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1230
2019-03-25 13:54:09.996 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Instance 64d3dc5a-2113-4cb7-9675-937285f0d23d actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 1024, u'DISK_GB': 10}}. _remove_deleted_instances_allocations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1230
2019-03-25 13:54:10.029 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Total usable vcpus: 8, total allocated vcpus: 3 _report_final_resource_view /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:848
2019-03-25 13:54:10.030 1 INFO nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Final resource view: name=hv01ve.worker.devgr1a.openstack.test.de phys_ram=12007MB used_ram=4273MB phys_disk=499GB used_disk=30GB total_vcpus=8 used_vcpus=3 pci_stats=[]
2019-03-25 13:54:10.174 1 DEBUG nova.scheduler.client.report [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Refreshing aggregate associations for resource provider 17f4e1c1-979c-4690-b839-c1715772afa0 _ensure_resource_provider /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:509
2019-03-25 13:54:10.261 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Compute_service record updated for hv01ve.worker.devgr1a.openstack.test.de:hv01ve.worker.devgr1a.openstack.test.de _update_available_resource /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:788
2019-03-25 13:54:10.262 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.605s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2019-03-25 13:54:11.137 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._run_pending_deletes run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:11.137 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Cleaning up deleted instances _run_pending_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py:7131
2019-03-25 13:54:11.175 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] There are 0 instances to clean _run_pending_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py:7140
2019-03-25 13:54:13.172 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._sync_scheduler_instance_info run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:14.132 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:16.137 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:19.669 1 DEBUG nova.compute.manager [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1186
2019-03-25 13:54:19.946 1 INFO nova.compute.manager [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] [instance: 91673651-94c0-4e85-9670-c671bec623e9] instance snapshotting
2019-03-25 13:54:20.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:20.142 1 DEBUG nova.objects.instance [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] Lazy-loading 'pci_devices' on Instance uuid 91673651-94c0-4e85-9670-c671bec623e9 obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:1065
2019-03-25 13:54:20.172 1 DEBUG nova.virt.driver [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] Emitting event <LifecycleEvent: 1553522060.17, 91673651-94c0-4e85-9670-c671bec623e9 => Paused> emit_event /usr/lib/python2.7/dist-packages/nova/virt/driver.py:1465
2019-03-25 13:54:20.172 1 INFO nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] VM Paused (Lifecycle Event)
2019-03-25 13:54:20.260 1 DEBUG nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1186
2019-03-25 13:54:20.527 1 DEBUG nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Synchronizing instance power state after lifecycle event "Paused"; current vm_state: active, current task_state: image_snapshot, current DB power_state: 1, VM power_state: 3 handle_lifecycle_event /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1090
2019-03-25 13:54:20.646 1 INFO nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] During sync_power_state the instance has a pending task (image_snapshot). Skip.
2019-03-25 13:54:26.989 1 INFO nova.virt.libvirt.driver [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Beginning cold snapshot process
2019-03-25 13:54:27.677 1 DEBUG nova.virt.libvirt.storage.rbd_utils [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] creating snapshot(f23b259d000d4695aafbadeb13ad6db9) on rbd image(91673651-94c0-4e85-9670-c671bec623e9_disk) create_snap /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:384
2019-03-25 13:54:27.917 1 DEBUG nova.virt.libvirt.storage.rbd_utils [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] cloning cinder/91673651-94c0-4e85-9670-c671bec623e9_disk@f23b259d000d4695aafbadeb13ad6db9 to glance/dadecbfe-2262-40ff-bf8f-b062f76c357d clone /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:234
2019-03-25 13:54:28.230 1 DEBUG nova.virt.libvirt.storage.rbd_utils [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] flattening glance/dadecbfe-2262-40ff-bf8f-b062f76c357d flatten /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:284
2019-03-25 13:54:32.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:33.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:33.137 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6726
2019-03-25 13:54:41.997 1 DEBUG nova.virt.driver [-] Emitting event <LifecycleEvent: 1553522066.99, 91673651-94c0-4e85-9670-c671bec623e9 => Stopped> emit_event /usr/lib/python2.7/dist-packages/nova/virt/driver.py:1465
2019-03-25 13:54:41.997 1 INFO nova.compute.manager [-] [instance: 91673651-94c0-4e85-9670-c671bec623e9] VM Stopped (Lifecycle Event)
2019-03-25 13:54:42.091 1 DEBUG nova.compute.manager [req-3a531eef-d32c-4f68-b7a6-283a4d3b4aa0 - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1186
2019-03-25 13:54:42.097 1 DEBUG nova.compute.manager [req-3a531eef-d32c-4f68-b7a6-283a4d3b4aa0 - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Synchronizing instance power state after lifecycle event "Stopped"; current vm_state: active, current task_state: image_uploading, current DB power_state: 1, VM power_state: 4 handle_lifecycle_event /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1090
2019-03-25 13:54:42.178 1 INFO nova.compute.manager [req-3a531eef-d32c-4f68-b7a6-283a4d3b4aa0 - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] During sync_power_state the instance has a pending task (image_uploading). Skip.
2019-03-25 13:54:44.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:44.137 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6061
2019-03-25 13:54:44.292 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Acquired semaphore "refresh_cache-91673651-94c0-4e85-9670-c671bec623e9" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:215
2019-03-25 13:54:44.292 1 DEBUG nova.network.neutronv2.api [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] _get_instance_nw_info() _get_instance_nw_info /usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py:1350
2019-03-25 13:54:45.924 1 DEBUG nova.network.base_api [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "170d1b2f-c937-4f76-8dad-ed2624cf1cb9", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "192.168.100.4"}], "version": 4, "meta": {"dhcp_server": "192.168.100.3"}, "dns": [], "routes": [], "cidr": "192.168.100.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "192.168.100.1"}}], "meta": {"injected": false, "tenant_id": "a4c89f7757df42e3acf10877d3e57366", "mtu": 1450}, "id": "8a4d0c31-a2e1-4109-89d6-6ac7c29eae27", "label": "test-az-1"}, "devname": "tap170d1b2f-c9", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true}, "address": "fa:16:3e:35:0a:b3", "active": true, "type": "ovs", "id": "170d1b2f-c937-4f76-8dad-ed2624cf1cb9", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/dist-packages/nova/network/base_api.py:48
2019-03-25 13:54:46.112 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Releasing semaphore "refresh_cache-91673651-94c0-4e85-9670-c671bec623e9" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2019-03-25 13:54:46.113 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Updated the network info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6123
2019-03-25 13:54:48.115 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:54:53.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:06.570 1 DEBUG nova.virt.libvirt.storage.rbd_utils [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] removing snapshot(f23b259d000d4695aafbadeb13ad6db9) on rbd image(91673651-94c0-4e85-9670-c671bec623e9_disk) remove_snap /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:411
2019-03-25 13:55:07.003 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._sync_power_states run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:07.056 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Triggering sync for uuid d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1 _sync_power_states /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6534
2019-03-25 13:55:07.056 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Triggering sync for uuid 64d3dc5a-2113-4cb7-9675-937285f0d23d _sync_power_states /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6534
2019-03-25 13:55:07.057 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Triggering sync for uuid 91673651-94c0-4e85-9670-c671bec623e9 _sync_power_states /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6534
2019-03-25 13:55:07.058 1 DEBUG oslo_concurrency.lockutils [-] Lock "d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1" acquired by "nova.compute.manager.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2019-03-25 13:55:07.059 1 DEBUG oslo_concurrency.lockutils [-] Lock "64d3dc5a-2113-4cb7-9675-937285f0d23d" acquired by "nova.compute.manager.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2019-03-25 13:55:07.060 1 DEBUG oslo_concurrency.lockutils [-] Lock "91673651-94c0-4e85-9670-c671bec623e9" acquired by "nova.compute.manager.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2019-03-25 13:55:07.060 1 INFO nova.compute.manager [-] [instance: 91673651-94c0-4e85-9670-c671bec623e9] During sync_power_state the instance has a pending task (image_uploading). Skip.
2019-03-25 13:55:07.061 1 DEBUG oslo_concurrency.lockutils [-] Lock "91673651-94c0-4e85-9670-c671bec623e9" released by "nova.compute.manager.query_driver_power_state_and_sync" :: held 0.001s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2019-03-25 13:55:07.136 1 DEBUG oslo_concurrency.lockutils [-] Lock "d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1" released by "nova.compute.manager.query_driver_power_state_and_sync" :: held 0.078s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2019-03-25 13:55:07.147 1 DEBUG oslo_concurrency.lockutils [-] Lock "64d3dc5a-2113-4cb7-9675-937285f0d23d" released by "nova.compute.manager.query_driver_power_state_and_sync" :: held 0.088s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2019-03-25 13:55:07.274 1 DEBUG nova.virt.libvirt.storage.rbd_utils [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] creating snapshot(snap) on rbd image(dadecbfe-2262-40ff-bf8f-b062f76c357d) create_snap /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:384
2019-03-25 13:55:09.137 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:09.179 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Auditing locally available compute resources for hv01ve.worker.devgr1a.openstack.test.de (node: hv01ve.worker.devgr1a.openstack.test.de) update_available_resource /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:696
2019-03-25 13:55:09.975 1 DEBUG nova.virt.libvirt.driver [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] skipping disk for instance-0000007b as it does not have a path _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7389
2019-03-25 13:55:09.979 1 DEBUG nova.virt.libvirt.driver [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] skipping disk for instance-00000084 as it does not have a path _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7389
2019-03-25 13:55:09.982 1 DEBUG nova.virt.libvirt.driver [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] skipping disk for instance-0000006f as it does not have a path _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7389
2019-03-25 13:55:10.096 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Hypervisor/Node resource view: name=hv01ve.worker.devgr1a.openstack.test.de free_ram=7621MB free_disk=467GB free_vcpus=5 pci_devices=[{"dev_id": "pci_0000_00_08_0", "product_id": "1002", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1002", "address": "0000:00:08.0"}, {"dev_id": "pci_0000_00_07_0", "product_id": "1003", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1003", "address": "0000:00:07.0"}, {"dev_id": "pci_0000_00_09_0", "product_id": "1000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1000", "address": "0000:00:09.0"}, {"dev_id": "pci_0000_00_06_0", "product_id": "1004", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1004", "address": "0000:00:06.0"}, {"dev_id": "pci_0000_00_01_0", "product_id": "7000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7000", "address": "0000:00:01.0"}, {"dev_id": "pci_0000_00_05_2", "product_id": "2936", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2936", "address": "0000:00:05.2"}, {"dev_id": "pci_0000_00_05_0", "product_id": "2934", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2934", "address": "0000:00:05.0"}, {"dev_id": "pci_0000_00_01_1", "product_id": "7010", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7010", "address": "0000:00:01.1"}, {"dev_id": "pci_0000_00_05_1", "product_id": "2935", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2935", "address": "0000:00:05.1"}, {"dev_id": "pci_0000_00_03_0", "product_id": "1000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1000", "address": "0000:00:03.0"}, {"dev_id": "pci_0000_00_02_0", "product_id": "0100", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1b36", "label": "label_1b36_0100", "address": "0000:00:02.0"}, {"dev_id": "pci_0000_00_04_0", "product_id": "2668", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_2668", "address": "0000:00:04.0"}, {"dev_id": "pci_0000_00_01_3", "product_id": "7113", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7113", "address": "0000:00:01.3"}, {"dev_id": "pci_0000_00_05_7", "product_id": "293a", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_293a", "address": "0000:00:05.7"}, {"dev_id": "pci_0000_00_00_0", "product_id": "1237", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_1237", "address": "0000:00:00.0"}, {"dev_id": "pci_0000_00_0a_0", "product_id": "1005", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "1af4", "label": "label_1af4_1005", "address": "0000:00:0a.0"}] _report_hypervisor_resource_view /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:832
2019-03-25 13:55:10.097 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker._update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2019-03-25 13:55:10.229 1 DEBUG nova.scheduler.client.report [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Refreshing aggregate associations for resource provider 17f4e1c1-979c-4690-b839-c1715772afa0 _ensure_resource_provider /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:509
2019-03-25 13:55:10.436 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Compute driver doesn't require allocation refresh and we're on a compute host in a deployment that only has compute hosts with Nova versions >=16 (Pike). Skipping auto-correction of allocations.  _update_usage_from_instances /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1209
2019-03-25 13:55:10.505 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Instance d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 1024, u'DISK_GB': 10}}. _remove_deleted_instances_allocations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1230
2019-03-25 13:55:10.506 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Instance 91673651-94c0-4e85-9670-c671bec623e9 actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 1024, u'DISK_GB': 10}}. _remove_deleted_instances_allocations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1230
2019-03-25 13:55:10.506 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Instance 64d3dc5a-2113-4cb7-9675-937285f0d23d actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 1024, u'DISK_GB': 10}}. _remove_deleted_instances_allocations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:1230
2019-03-25 13:55:10.641 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Total usable vcpus: 8, total allocated vcpus: 3 _report_final_resource_view /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:848
2019-03-25 13:55:10.642 1 INFO nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Final resource view: name=hv01ve.worker.devgr1a.openstack.test.de phys_ram=12007MB used_ram=4273MB phys_disk=499GB used_disk=30GB total_vcpus=8 used_vcpus=3 pci_stats=[]
2019-03-25 13:55:10.808 1 DEBUG nova.scheduler.client.report [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Refreshing aggregate associations for resource provider 17f4e1c1-979c-4690-b839-c1715772afa0 _ensure_resource_provider /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:509
2019-03-25 13:55:10.899 1 DEBUG nova.compute.resource_tracker [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Compute_service record updated for hv01ve.worker.devgr1a.openstack.test.de:hv01ve.worker.devgr1a.openstack.test.de _update_available_resource /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:788
2019-03-25 13:55:10.900 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.802s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2019-03-25 13:55:11.339 1 DEBUG nova.virt.driver [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] Emitting event <LifecycleEvent: 1553522111.34, 91673651-94c0-4e85-9670-c671bec623e9 => Started> emit_event /usr/lib/python2.7/dist-packages/nova/virt/driver.py:1465
2019-03-25 13:55:11.340 1 INFO nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] VM Started (Lifecycle Event)
2019-03-25 13:55:11.469 1 DEBUG nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1186
2019-03-25 13:55:13.137 1 INFO nova.virt.libvirt.driver [req-2fcbcbc9-e6b7-4dc4-8a48-76f9dc3a7c58 cde23909bc4d44fea20ad3c67c4a92d4 a4c89f7757df42e3acf10877d3e57366 - default default] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Snapshot image upload complete
2019-03-25 13:55:13.145 1 DEBUG nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Synchronizing instance power state after lifecycle event "Started"; current vm_state: active, current task_state: image_uploading, current DB power_state: 1, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1090
2019-03-25 13:55:13.217 1 INFO nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] During sync_power_state the instance has a pending task (image_uploading). Skip.
2019-03-25 13:55:13.218 1 DEBUG nova.virt.driver [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] Emitting event <LifecycleEvent: 1553522111.34, 91673651-94c0-4e85-9670-c671bec623e9 => Resumed> emit_event /usr/lib/python2.7/dist-packages/nova/virt/driver.py:1465
2019-03-25 13:55:13.219 1 INFO nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] VM Resumed (Lifecycle Event)
2019-03-25 13:55:13.304 1 DEBUG nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1186
2019-03-25 13:55:13.312 1 DEBUG nova.compute.manager [req-22be35a7-8241-4dfa-b91f-c38dbb7d37af - - - - -] [instance: 91673651-94c0-4e85-9670-c671bec623e9] Synchronizing instance power state after lifecycle event "Resumed"; current vm_state: active, current task_state: image_uploading, current DB power_state: 1, VM power_state: 1 handle_lifecycle_event /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1090
2019-03-25 13:55:16.131 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:18.138 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:21.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:34.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:35.136 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:35.137 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6726
2019-03-25 13:55:44.137 1 DEBUG oslo_service.periodic_task [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215
2019-03-25 13:55:44.138 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6061
2019-03-25 13:55:44.138 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6065
2019-03-25 13:55:44.195 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Acquired semaphore "refresh_cache-d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:215
2019-03-25 13:55:44.195 1 DEBUG nova.network.neutronv2.api [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] [instance: d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1] _get_instance_nw_info() _get_instance_nw_info /usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py:1350
2019-03-25 13:55:44.196 1 DEBUG nova.objects.instance [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Lazy-loading 'info_cache' on Instance uuid d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1 obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:1065
2019-03-25 13:55:45.028 1 DEBUG nova.network.base_api [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] [instance: d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "7d7de42c-a6ad-44c6-8b58-94ebd788217b", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "192.168.1.106"}], "version": 4, "meta": {"dhcp_server": "192.168.1.101"}, "dns": [], "routes": [], "cidr": "192.168.1.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "192.168.1.1"}}], "meta": {"injected": false, "tenant_id": "a4c89f7757df42e3acf10877d3e57366", "mtu": 1450}, "id": "e803cb27-3748-4d17-a42e-7b41f17ff30f", "label": "Internal"}, "devname": "tap7d7de42c-a6", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true}, "address": "fa:16:3e:39:dd:26", "active": true, "type": "ovs", "id": "7d7de42c-a6ad-44c6-8b58-94ebd788217b", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/dist-packages/nova/network/base_api.py:48
2019-03-25 13:55:45.068 1 DEBUG oslo_concurrency.lockutils [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] Releasing semaphore "refresh_cache-d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2019-03-25 13:55:45.069 1 DEBUG nova.compute.manager [req-cd4cf92a-33ca-434d-aff5-56f24197ce84 - - - - -] [instance: d7f3b957-7a0e-45d4-82b7-cdbd3efee0d1] Updated the network info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:6123