2021-09-24 05:00:16,488 INFO  [c.c.h.v.m.VirtualMachineMO] (DirectAgent-445:ctx-7958445e lnd-uk-002.test.host, job-35363/job-35364, cmd: StopCommand) (logid:82107a6a) Try gracefully shut down VM s-2053-VM
2021-09-24 05:00:19,021 INFO  [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-8415aa82) (logid:cf6768ba) No running secondary storage vms found in datacenter id=7, starting one
2021-09-24 05:00:19,023 INFO  [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-8415aa82) (logid:cf6768ba) Found SSVMs that are currently at stopping state, wait until they are settled
2021-09-24 05:00:19,751 INFO  [c.c.a.ApiServer] (qtp769798433-20:ctx-7de2d508 ctx-69b14dfa ctx-241a4578) (logid:beafbb2a) Unable to execute API command listvirtualmachines due to invalid value. Invalid parameter domainid value=76ea1e8a-e8ee-4461-b4d1-e652ddce06e1 due to incorrect long value format, or entity does not exist or due to incorrect parameter annotation for the field in api cmd class.
2021-09-24 05:00:26,664 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:ctx-622c2882 job-35363/job-35364) (logid:82107a6a) Remove job-35364 from job monitoring
2021-09-24 05:00:26,720 INFO  [c.c.s.r.VmwareStorageProcessor] (DirectAgent-449:ctx-8fda5006 lnd-uk-002.test.host, job-35363, cmd: DeleteCommand) (logid:82107a6a) Executing resource DeleteCommand: {"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a19d0461-b049-479a-8852-52711dc709b2","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fb7b551a-584e-4b51-92b8-f371af56b303","id":21,"poolType":"VMFS","host":"VMFS datastore: datastore-6694","path":"datastore-6694","port":0,"url":"VMFS://VMFS datastore: datastore-6694/datastore-6694/?ROLE=Primary&STOREUUID=fb7b551a-584e-4b51-92b8-f371af56b303","isManaged":false}},"name":"ROOT-2053","size":0,"path":"s-2053-VM","volumeId":3068,"vmName":"s-2053-VM","accountId":1,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[lnd-uk-002] s-2053-VM/s-2053-VM.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":3068,"deviceId":0,"hypervisorType":"VMware","directDownload":false,"deployAsIs":true}},"wait":0,"bypassHostMaintenance":true}
2021-09-24 05:00:26,761 INFO  [c.c.s.r.VmwareStorageProcessor] (DirectAgent-449:ctx-8fda5006 lnd-uk-002.test.host, job-35363, cmd: DeleteCommand) (logid:82107a6a) Destroying root volume s-2053-VM of deploy-as-is VM s-2053-VM
2021-09-24 05:00:28,134 INFO  [o.a.c.s.v.VolumeServiceImpl] (API-Job-Executor-3:ctx-49c686d4 job-35363 ctx-c5b8883d) (logid:82107a6a) Volume 3068 is not referred anywhere, remove it from volumes table
2021-09-24 05:00:28,151 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-3:ctx-49c686d4 job-35363) (logid:82107a6a) Remove job-35363 from job monitoring
2021-09-24 05:00:31,270 WARN  [c.c.a.d.ParamGenericValidationWorker] (qtp769798433-22:ctx-0754e40c ctx-11123d72) (logid:09a8cb74) Received unknown parameters for command listSystemVms. Unknown parameters : listall
2021-09-24 05:00:48,723 INFO  [c.c.h.v.m.HostMO] (DirectAgent-225:ctx-fc24d14d ns320278.ip-5-135-143.eu, cmd: GetVmStatsCommand) (logid:056bde2f) VM i-1231-1561-VM not found in host cache
2021-09-24 05:00:49,032 INFO  [c.c.h.v.m.HostMO] (DirectAgent-450:ctx-a46cb2a8 ns349017.ip-176-31-224.eu, cmd: GetVmStatsCommand) (logid:056bde2f) VM i-2-1931-VM not found in host cache
2021-09-24 05:00:49,033 INFO  [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-61f9c60b) (logid:0d48c9f7) No running secondary storage vms found in datacenter id=7, starting one
2021-09-24 05:00:49,036 INFO  [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-61f9c60b) (logid:0d48c9f7) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
2021-09-24 05:00:49,065 INFO  [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-61f9c60b) (logid:0d48c9f7) allocating virtual machine from template:83d8f1c0-c2f6-4ca8-80a7-72bffc8d6341 with hostname:s-2056-VM and 3 networks
2021-09-24 05:00:49,164 INFO  [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-61f9c60b) (logid:0d48c9f7) adding disk object ROOT-2056 to s-2056-VM
2021-09-24 05:00:49,522 INFO  [c.c.h.v.m.HostMO] (DirectAgent-113:ctx-cda205dc ns63144175.ip-51-83-1.eu, cmd: GetVmStatsCommand) (logid:056bde2f) VM i-1099-1266-VM not found in host cache
2021-09-24 05:00:50,411 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365) (logid:6d078906) Add job-35365 into job monitoring
2021-09-24 05:00:50,431 INFO  [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb FirstFitRoutingAllocator) (logid:d442432d)  Guest VM is requested with Custom[UEFI] Boot Type false
2021-09-24 05:00:50,448 INFO  [c.c.s.StorageManagerImpl] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb) (logid:d442432d) Storage pool null (21) does not supply IOPS capacity, assuming enough capacity
2021-09-24 05:00:50,463 INFO  [c.c.s.StorageManagerImpl] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb) (logid:d442432d) Storage pool null (22) does not supply IOPS capacity, assuming enough capacity
2021-09-24 05:00:50,630 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-226:ctx-75047a08 lnd-uk-002.test.host, job-25099/job-35365, cmd: CopyCommand) (logid:d442432d) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
2021-09-24 05:00:50,682 INFO  [c.c.h.v.m.HostMO] (DirectAgent-451:ctx-66170119 fra-de-001.test.host, cmd: GetVmStatsCommand) (logid:056bde2f) VM i-2-2008-VM not found in host cache
2021-09-24 05:00:50,706 INFO  [c.c.s.r.VmwareStorageProcessor] (DirectAgent-226:ctx-75047a08 lnd-uk-002.test.host, job-25099/job-35365, cmd: CopyCommand) (logid:d442432d) ROOT Volume from deploy-as-is template, cloning template
2021-09-24 05:00:50,751 INFO  [c.c.s.r.VmwareStorageProcessor] (DirectAgent-226:ctx-75047a08 lnd-uk-002.test.host, job-25099/job-35365, cmd: CopyCommand) (logid:d442432d) Cloning VM s-2056-VM from template 450d7fc3-5026-4cf6-9d30-27974f8e9f11 into datastore fb7b551a-584e-4b51-92b8-f371af56b303
2021-09-24 05:00:50,751 INFO  [c.c.s.r.VmwareStorageProcessor] (DirectAgent-226:ctx-75047a08 lnd-uk-002.test.host, job-25099/job-35365, cmd: CopyCommand) (logid:d442432d) creating full clone from template
2021-09-24 05:00:55,496 ERROR [o.a.c.s.i.d.DeployAsIsHelperImpl] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb) (logid:d442432d) Different number of networks provided vs networks defined in deploy-as-is template
2021-09-24 05:00:55,514 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Executing resource StartCommand: {"vm":{"id":"2056","name":"s-2056-VM","bootloader":"HVM","state":"Starting","type":"SecondaryStorageVm","cpus":"1","minSpeed":"633","maxSpeed":"1900","minRam":"(1.00 GB) 1073741824","maxRam":"(2.00 GB) 2147483648","hostName":"s-2056-VM","arch":"x86_64","os":"Other Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":" template=domP type=secstorage host=172.11.11.2,10.12.0.254 port=8250 name=s-2056-VM zone=7 pod=7 guid=s-2056-VM workers=5 resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 eth2ip=51.xxx.xxx.201 eth2mask=255.255.255.248 gateway=51.xxx.xxx.206 public.network.device=eth2 eth0mask=0.0.0.0 eth0ip=0.0.0.0 eth1ip=10.15.0.170 eth1mask=255.255.255.0 localgw=10.15.0.4 private.network.device=eth1 internaldns1=10.15.0.4 dns1=8.8.8.8 nfsVersion=null","enableHA":"false","limitCpuUse":"false","enableDynamicallyScaleVm":"false","vncPassword":"0BLqbNIBiTKktxRQ6vxCUA","params":{"cpuOvercommitRatio":"3.0","vmware.reserve.mem":"false","vmware.reserve.cpu":"false","memoryOvercommitRatio":"2.0","nicAdapter":"Vmxnet3"},"uuid":"e2852583-701f-4587-a79e-901b527433db","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"76bfd94f-516a-4aa8-bc80-3c7134433bf5","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fb7b551a-584e-4b51-92b8-f371af56b303","id":"21","poolType":"VMFS","host":"VMFS datastore: datastore-6694","path":"datastore-6694","port":"0","url":"VMFS://VMFS datastore: datastore-6694/datastore-6694/?ROLE=Primary&STOREUUID=fb7b551a-584e-4b51-92b8-f371af56b303","isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes) 0","volumeId":"3071","vmName":"s-2056-VM","accountId":"1","format":"OVA","provisioningType":"THIN","id":"3071","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"VMware","directDownload":"false","deployAsIs":"true"}},"diskSeq":"0","type":"ROOT","_details":{"storageHost":"VMFS datastore: datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 bytes) 0"}}],"nics":[{"deviceId":"2","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"true","nicUuid":"08c09d90-23e9-46c9-9ad5-5ea202b9133d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","uuid":"604dd92d-9c0f-487c-bd4c-f0004326d059","ip":"51.xxx.xxx.201","netmask":"255.255.255.248","gateway":"51.xxx.xxx.206","mac":"1e:00:f7:00:00:52","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":"false","name":"vSwitch1"},{"deviceId":"0","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"766d34e1-b0e3-46fd-a123-855d29aba11f","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","uuid":"1ff09951-d6ae-4dbf-8123-2a7c6439751c","mac":"02:00:00:01:00:25","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":"false","name":"vSwitch1"},{"deviceId":"1","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"a7770b23-9191-45f2-8e12-4e0c94cbb1bf","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","uuid":"f26abc86-6cc9-4636-b1f4-397a18b239e9","ip":"10.15.0.170","netmask":"255.255.255.0","gateway":"10.15.0.4","mac":"1e:00:f0:00:00:15","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":"false","name":"vSwitch1"}],"guestOsDetails":{},"extraConfig":{},"deployAsIsInfo":{"nicAdapterMap":{}}},"hostIp":"10.15.0.31","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}
2021-09-24 05:00:55,561 INFO  [c.c.h.v.m.HostMO] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) VM s-2056-VM not found in host cache
2021-09-24 05:00:55,568 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) VM s-2056-VM already exists, tear down devices for reconfiguration
2021-09-24 05:00:55,590 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapping VM disks to spec disks and tearing down datadisks (if any)
2021-09-24 05:00:55,594 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapped disk size is not the same as the cloned VM disk size: 0 - 2621440000
2021-09-24 05:00:55,594 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303
2021-09-24 05:00:56,680 INFO  [c.c.h.v.m.VirtualMachineMO] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Look for disk device info for volume : s-2056-VM.vmdk with base name: s-2056-VM
2021-09-24 05:00:56,680 INFO  [c.c.h.v.m.VirtualMachineMO] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Test against disk device, controller key: 1000, unit number: 0
2021-09-24 05:00:56,680 INFO  [c.c.h.v.m.VirtualMachineMO] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Test against disk backing : [lnd-uk-002] s-2056-VM/s-2056-VM.vmdk
2021-09-24 05:00:56,680 INFO  [c.c.h.v.m.VirtualMachineMO] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Disk backing : [lnd-uk-002] s-2056-VM/s-2056-VM.vmdk matches ==> scsi0:0
2021-09-24 05:00:56,680 INFO  [c.c.u.n.n.p.NiciraNvpApiVersion] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) NSX API VERSION:  NOT PRESENT
2021-09-24 05:00:56,680 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Prepare NIC device based on NicTO: {"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"766d34e1-b0e3-46fd-a123-855d29aba11f","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"1ff09951-d6ae-4dbf-8123-2a7c6439751c","mac":"02:00:00:01:00:25","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch1"}
2021-09-24 05:00:56,684 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Prepare network on vmwaresvs vSwitch1 with name prefix: cloud.private
2021-09-24 05:00:56,685 WARN  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Unrecognized broadcast type in VmwareResource, type: LinkLocal. Use vlan info from labeling: untagged
2021-09-24 05:00:57,840 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Network cloud.private.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
2021-09-24 05:00:57,840 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Network cloud.private.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
2021-09-24 05:00:57,840 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Preparing NIC device on network cloud.private.untagged.0.1-vSwitch1
2021-09-24 05:00:57,841 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Prepare NIC device based on NicTO: {"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"a7770b23-9191-45f2-8e12-4e0c94cbb1bf","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f26abc86-6cc9-4636-b1f4-397a18b239e9","ip":"10.15.0.170","netmask":"255.255.255.0","gateway":"10.15.0.4","mac":"1e:00:f0:00:00:15","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch1"}
2021-09-24 05:00:57,844 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Prepare network on vmwaresvs vSwitch1 with name prefix: cloud.private
2021-09-24 05:00:59,055 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Network cloud.private.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
2021-09-24 05:00:59,055 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Network cloud.private.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
2021-09-24 05:00:59,055 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Preparing NIC device on network cloud.private.untagged.0.1-vSwitch1
2021-09-24 05:00:59,055 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Prepare NIC device based on NicTO: {"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"08c09d90-23e9-46c9-9ad5-5ea202b9133d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"604dd92d-9c0f-487c-bd4c-f0004326d059","ip":"51.xxx.xxx.201","netmask":"255.255.255.248","gateway":"51.xxx.xxx.206","mac":"1e:00:f7:00:00:52","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch1"}
2021-09-24 05:00:59,059 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Prepare network on vmwaresvs vSwitch1 with name prefix: cloud.public
2021-09-24 05:01:00,091 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Network cloud.public.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
2021-09-24 05:01:00,092 INFO  [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Network cloud.public.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
2021-09-24 05:01:00,092 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Preparing NIC device on network cloud.public.untagged.0.1-vSwitch1
2021-09-24 05:01:00,105 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Configure VNC port for VM s-2056-VM, port: 5927, host: lnd-uk-002.test.host
2021-09-24 05:01:00,677 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Found existing disk info from volume path: s-2056-VM
2021-09-24 05:01:03,697 INFO  [c.c.h.v.m.VirtualMachineMO] (VM-Question-Monitor-3:null) (logid:d442432d) VM Question monitor started...
2021-09-24 05:01:04,558 INFO  [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: CheckSshCommand) (logid:d442432d) Trying to connect to 10.15.0.170
2021-09-24 05:01:04,715 INFO  [c.c.h.v.m.VirtualMachineMO] (VM-Question-Monitor-3:null) (logid:d442432d) VM Question monitor stopped
2021-09-24 05:01:05,473 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-159:ctx-e93cc817) (logid:f3623b4c) Scan hung worker VM to recycle
2021-09-24 05:01:05,642 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-319:ctx-f27c481a) (logid:5c333729) Scan hung worker VM to recycle
2021-09-24 05:01:06,221 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-160:ctx-555d833d) (logid:a02ab47f) Scan hung worker VM to recycle
2021-09-24 05:01:06,437 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-320:ctx-5c417d19) (logid:d5357f56) Scan hung worker VM to recycle
2021-09-24 05:01:06,487 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-41:ctx-7f8836d0) (logid:325e379f) Scan hung worker VM to recycle
2021-09-24 05:01:07,665 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-321:ctx-8084ac49) (logid:f88092bf) Scan hung worker VM to recycle
2021-09-24 05:01:07,706 INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-161:ctx-82ab8ee0) (logid:747dccbe) Scan hung worker VM to recycle
2021-09-24 05:01:07,749 INFO  [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-161:ctx-82ab8ee0) (logid:747dccbe) There is pending job or HA tasks working on the VM. vm id: 2056, postpone power-change report by resetting power-change counters
2021-09-24 05:01:19,672 INFO  [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb) (logid:d442432d) Update volume disk chain info. vol: 3071, s-2056-VM -> s-2056-VM, null -> {"diskDeviceBusName":"scsi0:0","diskChain":["[lnd-uk-002] s-2056-VM/s-2056-VM.vmdk"]}
2021-09-24 05:01:20,261 ERROR [c.c.u.s.SshHelper] (DirectAgent-457:ctx-a91a049f lnd-uk-002.test.host, job-25099/job-35365, cmd: SetupKeyStoreCommand) (logid:d442432d) Failed to authentication SSH user root on host 10.15.0.170
2021-09-24 05:01:20,261 INFO  [c.c.h.v.u.VmwareHelper] (DirectAgent-457:ctx-a91a049f lnd-uk-002.test.host, job-25099/job-35365, cmd: SetupKeyStoreCommand) (logid:d442432d) [ignored]failed to get message for exception: Failed to authentication SSH user root on host 10.15.0.170
2021-09-24 05:01:20,261 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-457:ctx-a91a049f lnd-uk-002.test.host, job-25099/job-35365, cmd: SetupKeyStoreCommand) (logid:d442432d) Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.170

2021-09-24 05:01:20,272 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb) (logid:d442432d) Retrying after catching exception while trying to secure agent for systemvm id=2056
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.170

        at org.apache.cloudstack.ca.provider.RootCAProvider.generateCertificateUsingCsr(RootCAProvider.java:170)
        at org.apache.cloudstack.ca.provider.RootCAProvider.issueCertificate(RootCAProvider.java:230)
        at org.apache.cloudstack.ca.CAManagerImpl.issueCertificate(CAManagerImpl.java:164)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy292.issueCertificate(Unknown Source)
        at com.cloud.vm.VirtualMachineManagerImpl.setupAgentSecurity(VirtualMachineManagerImpl.java:970)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1241)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-09-24 05:01:20,814 ERROR [c.c.u.s.SshHelper] (DirectAgent-229:ctx-f7caef28 lnd-uk-002.test.host, job-25099/job-35365, cmd: SetupKeyStoreCommand) (logid:d442432d) Failed to authentication SSH user root on host 10.15.0.170
2021-09-24 05:01:20,814 INFO  [c.c.h.v.u.VmwareHelper] (DirectAgent-229:ctx-f7caef28 lnd-uk-002.test.host, job-25099/job-35365, cmd: SetupKeyStoreCommand) (logid:d442432d) [ignored]failed to get message for exception: Failed to authentication SSH user root on host 10.15.0.170
2021-09-24 05:01:20,814 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-229:ctx-f7caef28 lnd-uk-002.test.host, job-25099/job-35365, cmd: SetupKeyStoreCommand) (logid:d442432d) Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.170

2021-09-24 05:01:20,826 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-6:ctx-28949761 job-25099/job-35365 ctx-9a134bfb) (logid:d442432d) Retrying after catching exception while trying to secure agent for systemvm id=2056
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.170

        at org.apache.cloudstack.ca.provider.RootCAProvider.generateCertificateUsingCsr(RootCAProvider.java:170)
        at org.apache.cloudstack.ca.provider.RootCAProvider.issueCertificate(RootCAProvider.java:230)
        at org.apache.cloudstack.ca.CAManagerImpl.issueCertificate(CAManagerImpl.java:164)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy292.issueCertificate(Unknown Source)
        at com.cloud.vm.VirtualMachineManagerImpl.setupAgentSecurity(VirtualMachineManagerImpl.java:970)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1241)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)