API tools faq
paste
Guest User

Untitled

a guest
Jun 3rd, 2019
650
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 79.38 KB | None | 0 0
raw download clone embed print report
  1. 2019-06-03 18:37:50,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-805f62c5) Begin cleanup expired async-jobs
  2. 2019-06-03 18:37:50,690 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-805f62c5) End cleanup expired async-jobs
  3. 2019-06-03 18:37:56,581 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-747114e8) No stopped console proxy is available, need to allocate a new console proxy
  4. 2019-06-03 18:37:56,600 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-1d853c88) No running secondary storage vms found in datacenter id=1, starting one
  5. 2019-06-03 18:37:56,606 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-1d853c88) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
  6. 2019-06-03 18:37:58,699 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29) Add job-29 into job monitoring
  7. 2019-06-03 18:37:58,772 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29 ctx-e45e2ee1) Storage pool null (1) does not supply IOPS capacity, assuming enough capacity
  8. 2019-06-03 18:37:59,011 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
  9. 2019-06-03 18:37:59,085 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) Template routing-8 is not setup yet. Set up template from secondary storage with uuid name: 533b6fcf3fa6301aadcc2b168f3f999a
  10. 2019-06-03 18:37:59,113 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://netapp1.example.org/np_sec1, templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
  11. 2019-06-03 18:37:59,367 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) Secondary storage mount point: /var/cloudstack/mnt/VM/345052023320.322b059c
  12. 2019-06-03 18:38:00,482 INFO [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) Resolving host name in url through vCenter, url: https://esx-0001-a-001.example.org/nfc/52e4ba95-2e7c-b06f-8365-10179d91b718/disk-0.vmdk
  13. 2019-06-03 18:38:00,549 INFO [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) url https://esx-0001-a-001.example.org/nfc/52e4ba95-2e7c-b06f-8365-10179d91b718/disk-0.vmdk is resolved to https://10.0.209.8/nfc/52e4ba95-2e7c-b06f-8365-10179d91b718/disk-0.vmdk through vCenter
  14. 2019-06-03 18:38:00,568 INFO [c.c.h.v.u.VmwareContext] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) Connected, conn: sun.net.www.protocol.https.DelegateHttpsURLConnection:https://10.0.209.8/nfc/52e4ba95-2e7c-b06f-8365-10179d91b718/disk-0.vmdk, retry: 0
  15. 2019-06-03 18:38:00,687 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-1c01efe2) Begin cleanup expired async-jobs
  16. 2019-06-03 18:38:00,691 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-1c01efe2) End cleanup expired async-jobs
  17. 2019-06-03 18:38:02,693 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30) Add job-30 into job monitoring
  18. 2019-06-03 18:38:02,748 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30 ctx-ac9ab7a6) Storage pool null (1) does not supply IOPS capacity, assuming enough capacity
  19. 2019-06-03 18:38:10,687 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-16918ddd) Begin cleanup expired async-jobs
  20. 2019-06-03 18:38:10,691 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-16918ddd) End cleanup expired async-jobs
  21. 2019-06-03 18:38:15,412 INFO [c.c.h.v.m.HttpNfcLeaseMO] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) close ProgressReporter, interrupt reporter runner to let it quit
  22. 2019-06-03 18:38:15,412 INFO [c.c.h.v.m.HttpNfcLeaseMO] (Thread-123:null) ProgressReporter is interrupted, quiting
  23. 2019-06-03 18:38:15,414 INFO [c.c.h.v.m.HttpNfcLeaseMO] (Thread-123:null) ProgressReporter stopped
  24. 2019-06-03 18:38:15,764 INFO [c.c.h.v.m.HostMO] (DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) VM 533b6fcf3fa6301aadcc2b168f3f999a not found in host cache
  25. 2019-06-03 18:38:17,017 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
  26. 2019-06-03 18:38:17,128 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) creating full clone from template
  27. 2019-06-03 18:38:17,657 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) [ignored]failed toi get message for exception: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  28. 2019-06-03 18:38:17,658 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: CopyCommand) clone volume from base image failed due to Exception: java.lang.RuntimeException
  29. Message: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  30.  
  31. java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  32. at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:358)
  33. at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createFullClone(VirtualMachineMO.java:720)
  34. at com.cloud.storage.resource.VmwareStorageProcessor.createVMFullClone(VmwareStorageProcessor.java:749)
  35. at com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:828)
  36. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:92)
  37. at com.cloud.storage.resource.VmwareStorageSubsystemCommandHandler.execute(VmwareStorageSubsystemCommandHandler.java:197)
  38. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:56)
  39. at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
  40. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
  41. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  42. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  43. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  44. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  45. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  46. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  47. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  48. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  49. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  50. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  51. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  52. at java.lang.Thread.run(Thread.java:748)
  53. 2019-06-03 18:38:17,679 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29 ctx-e45e2ee1) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@52f7141), no need to delete from object in store ref table
  54. 2019-06-03 18:38:17,684 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29 ctx-e45e2ee1) Unable to contact resource.
  55. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[1|vm=1|ROOT]:java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  56. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1301)
  57. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1376)
  58. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1113)
  59. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  60. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  61. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  62. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  63. at java.lang.reflect.Method.invoke(Method.java:498)
  64. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  65. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  66. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  67. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  68. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  69. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  70. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  71. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  72. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  73. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  74. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  75. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  76. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  77. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  78. at java.lang.Thread.run(Thread.java:748)
  79. 2019-06-03 18:38:17,696 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-8:ctx-851fa025 esx-0001-a-001.example.org, job-3/job-29, cmd: StopCommand) Executing resource StopCommand:
  80. 2019-06-03 18:38:17,710 INFO [c.c.h.v.m.HostMO] (DirectAgent-8:ctx-851fa025 esx-0001-a-001.example.org, job-3/job-29, cmd: StopCommand) VM v-1-VM not found in host cache
  81. 2019-06-03 18:38:17,732 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-8:ctx-851fa025 esx-0001-a-001.example.org, job-3/job-29, cmd: StopCommand) VM v-1-VM is no longer in vSphere
  82. 2019-06-03 18:38:17,795 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29 ctx-e45e2ee1) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  83. 2019-06-03 18:38:17,795 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29 ctx-e45e2ee1) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  84. 2019-06-03 18:38:17,795 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29) Unable to complete AsyncJobVO {id:29, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052023320, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Jun 03 18:37:56 CDT 2019}, job origin:3
  85. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  86. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  87. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  88. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  89. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  90. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  91. at java.lang.reflect.Method.invoke(Method.java:498)
  92. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  93. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  94. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  95. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  96. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  97. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  98. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  99. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  100. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  101. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  102. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  103. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  104. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  105. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  106. at java.lang.Thread.run(Thread.java:748)
  107. 2019-06-03 18:38:17,825 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-4edcb273 job-3/job-29) Remove job-29 from job monitoring
  108. 2019-06-03 18:38:17,832 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-747114e8) Exception while trying to start console proxy
  109. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  110. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  111. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  112. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  113. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  114. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  115. at java.lang.reflect.Method.invoke(Method.java:498)
  116. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  117. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  118. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  119. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  120. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  121. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  122. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  123. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  124. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  125. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  126. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  127. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  128. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  129. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  130. at java.lang.Thread.run(Thread.java:748)
  131. 2019-06-03 18:38:17,833 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-747114e8) Unable to start console proxy vm for standby capacity, vm id : 1, will recycle it and start a new one
  132. 2019-06-03 18:38:17,840 WARN [o.a.c.alerts] (consoleproxy-1:ctx-747114e8) AlertType:: 10 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Console proxy creation failure. zone: nonprod, error details: null
  133. 2019-06-03 18:38:17,905 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30 ctx-ac9ab7a6) Template routing-8 is already copied to primary storage, skip copying
  134. 2019-06-03 18:38:17,949 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-5:ctx-b2fbcbdb esx-0001-a-001.example.org, job-4/job-30, cmd: CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
  135. 2019-06-03 18:38:18,049 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-5:ctx-b2fbcbdb esx-0001-a-001.example.org, job-4/job-30, cmd: CopyCommand) creating full clone from template
  136. 2019-06-03 18:38:18,520 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-5:ctx-b2fbcbdb esx-0001-a-001.example.org, job-4/job-30, cmd: CopyCommand) [ignored]failed toi get message for exception: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  137. 2019-06-03 18:38:18,521 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-5:ctx-b2fbcbdb esx-0001-a-001.example.org, job-4/job-30, cmd: CopyCommand) clone volume from base image failed due to Exception: java.lang.RuntimeException
  138. Message: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  139.  
  140. java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  141. at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:358)
  142. at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createFullClone(VirtualMachineMO.java:720)
  143. at com.cloud.storage.resource.VmwareStorageProcessor.createVMFullClone(VmwareStorageProcessor.java:749)
  144. at com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:828)
  145. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:92)
  146. at com.cloud.storage.resource.VmwareStorageSubsystemCommandHandler.execute(VmwareStorageSubsystemCommandHandler.java:197)
  147. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:56)
  148. at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
  149. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
  150. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  151. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  152. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  153. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  154. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  155. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  156. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  157. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  158. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  159. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  160. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  161. at java.lang.Thread.run(Thread.java:748)
  162. 2019-06-03 18:38:18,530 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30 ctx-ac9ab7a6) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@341fb0c), no need to delete from object in store ref table
  163. 2019-06-03 18:38:18,533 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30 ctx-ac9ab7a6) Unable to contact resource.
  164. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  165. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1301)
  166. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1376)
  167. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1113)
  168. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  169. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  170. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  171. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  172. at java.lang.reflect.Method.invoke(Method.java:498)
  173. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  174. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  175. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  176. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  177. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  178. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  179. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  180. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  181. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  182. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  183. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  184. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  185. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  186. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  187. at java.lang.Thread.run(Thread.java:748)
  188. 2019-06-03 18:38:18,540 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-9:ctx-02444d05 esx-0001-a-001.example.org, job-4/job-30, cmd: StopCommand) Executing resource StopCommand:
  189. 2019-06-03 18:38:18,550 INFO [c.c.h.v.m.HostMO] (DirectAgent-9:ctx-02444d05 esx-0001-a-001.example.org, job-4/job-30, cmd: StopCommand) VM s-2-VM not found in host cache
  190. 2019-06-03 18:38:18,564 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-9:ctx-02444d05 esx-0001-a-001.example.org, job-4/job-30, cmd: StopCommand) VM s-2-VM is no longer in vSphere
  191. 2019-06-03 18:38:18,620 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30 ctx-ac9ab7a6) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  192. 2019-06-03 18:38:18,621 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30 ctx-ac9ab7a6) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  193. 2019-06-03 18:38:18,621 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30) Unable to complete AsyncJobVO {id:30, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052023320, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Jun 03 18:38:01 CDT 2019}, job origin:4
  194. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  195. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  196. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  197. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  198. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  199. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  200. at java.lang.reflect.Method.invoke(Method.java:498)
  201. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  202. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  203. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  204. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  205. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  206. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  207. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  208. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  209. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  210. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  211. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  212. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  213. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  214. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  215. at java.lang.Thread.run(Thread.java:748)
  216. 2019-06-03 18:38:18,633 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-2:ctx-ce3fc563 job-4/job-30) Remove job-30 from job monitoring
  217. 2019-06-03 18:38:18,640 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-1d853c88) Exception while trying to start secondary storage vm
  218. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  219. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  220. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  221. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  222. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  223. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  224. at java.lang.reflect.Method.invoke(Method.java:498)
  225. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  226. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  227. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  228. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  229. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  230. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  231. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  232. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  233. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  234. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  235. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  236. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  237. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  238. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  239. at java.lang.Thread.run(Thread.java:748)
  240. 2019-06-03 18:38:18,641 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-1d853c88) Unable to start secondary storage vm for standby capacity, vm id : 2, will recycle it and start a new one
  241. 2019-06-03 18:38:18,645 WARN [o.a.c.alerts] (secstorage-1:ctx-1d853c88) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: nonprod, error details: null
  242. 2019-06-03 18:38:18,647 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-1d853c88) Primary secondary storage is not even started, wait until next turn
  243. 2019-06-03 18:38:20,687 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d4473148) Begin cleanup expired async-jobs
  244. 2019-06-03 18:38:20,693 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d4473148) End cleanup expired async-jobs
  245. 2019-06-03 18:38:26,579 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-5751ca93) Found a stopped console proxy, starting it. Vm id : 1
  246. 2019-06-03 18:38:26,591 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-57d497bf) No running secondary storage vms found in datacenter id=1, starting one
  247. 2019-06-03 18:38:26,595 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-57d497bf) Found a stopped secondary storage vm, starting it. Vm id : 2
  248. 2019-06-03 18:38:26,693 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31) Add job-31 into job monitoring
  249. 2019-06-03 18:38:26,695 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32) Add job-32 into job monitoring
  250. 2019-06-03 18:38:26,746 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31 ctx-29f14560) Storage pool null (1) does not supply IOPS capacity, assuming enough capacity
  251. 2019-06-03 18:38:26,751 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32 ctx-40efb72e) Storage pool null (1) does not supply IOPS capacity, assuming enough capacity
  252. 2019-06-03 18:38:26,895 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-3:ctx-c36b7ae5 esx-0001-a-001.example.org, job-3/job-31, cmd: CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
  253. 2019-06-03 18:38:26,977 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-3:ctx-c36b7ae5 esx-0001-a-001.example.org, job-3/job-31, cmd: CopyCommand) creating full clone from template
  254. 2019-06-03 18:38:27,407 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-3:ctx-c36b7ae5 esx-0001-a-001.example.org, job-3/job-31, cmd: CopyCommand) [ignored]failed toi get message for exception: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  255. 2019-06-03 18:38:27,407 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-3:ctx-c36b7ae5 esx-0001-a-001.example.org, job-3/job-31, cmd: CopyCommand) clone volume from base image failed due to Exception: java.lang.RuntimeException
  256. Message: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  257.  
  258. java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  259. at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:358)
  260. at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createFullClone(VirtualMachineMO.java:720)
  261. at com.cloud.storage.resource.VmwareStorageProcessor.createVMFullClone(VmwareStorageProcessor.java:749)
  262. at com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:828)
  263. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:92)
  264. at com.cloud.storage.resource.VmwareStorageSubsystemCommandHandler.execute(VmwareStorageSubsystemCommandHandler.java:197)
  265. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:56)
  266. at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
  267. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
  268. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  269. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  270. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  271. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  272. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  273. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  274. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  275. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  276. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  277. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  278. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  279. at java.lang.Thread.run(Thread.java:748)
  280. 2019-06-03 18:38:27,412 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-11:ctx-428f4b0e esx-0001-a-001.example.org, job-4/job-32, cmd: CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
  281. 2019-06-03 18:38:27,419 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31 ctx-29f14560) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@601740e4), no need to delete from object in store ref table
  282. 2019-06-03 18:38:27,420 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31 ctx-29f14560) Unable to contact resource.
  283. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[1|vm=1|ROOT]:java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  284. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1301)
  285. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1376)
  286. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1113)
  287. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  288. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  289. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  290. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  291. at java.lang.reflect.Method.invoke(Method.java:498)
  292. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  293. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  294. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  295. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  296. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  297. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  298. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  299. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  300. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  301. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  302. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  303. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  304. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  305. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  306. at java.lang.Thread.run(Thread.java:748)
  307. 2019-06-03 18:38:27,519 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-11:ctx-428f4b0e esx-0001-a-001.example.org, job-4/job-32, cmd: CopyCommand) creating full clone from template
  308. 2019-06-03 18:38:27,934 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-11:ctx-428f4b0e esx-0001-a-001.example.org, job-4/job-32, cmd: CopyCommand) [ignored]failed toi get message for exception: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  309. 2019-06-03 18:38:27,934 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-11:ctx-428f4b0e esx-0001-a-001.example.org, job-4/job-32, cmd: CopyCommand) clone volume from base image failed due to Exception: java.lang.RuntimeException
  310. Message: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  311.  
  312. java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  313. at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:358)
  314. at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createFullClone(VirtualMachineMO.java:720)
  315. at com.cloud.storage.resource.VmwareStorageProcessor.createVMFullClone(VmwareStorageProcessor.java:749)
  316. at com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:828)
  317. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:92)
  318. at com.cloud.storage.resource.VmwareStorageSubsystemCommandHandler.execute(VmwareStorageSubsystemCommandHandler.java:197)
  319. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:56)
  320. at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
  321. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
  322. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  323. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  324. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  325. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  326. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  327. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  328. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  329. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  330. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  331. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  332. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  333. at java.lang.Thread.run(Thread.java:748)
  334. 2019-06-03 18:38:27,936 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-6:ctx-220f2646 esx-0001-a-001.example.org, job-3/job-31, cmd: StopCommand) Executing resource StopCommand:
  335. 2019-06-03 18:38:27,944 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32 ctx-40efb72e) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@77f3141), no need to delete from object in store ref table
  336. 2019-06-03 18:38:27,945 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32 ctx-40efb72e) Unable to contact resource.
  337. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  338. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1301)
  339. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1376)
  340. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1113)
  341. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  342. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  343. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  344. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  345. at java.lang.reflect.Method.invoke(Method.java:498)
  346. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  347. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  348. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  349. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  350. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  351. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  352. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  353. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  354. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  355. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  356. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  357. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  358. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  359. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  360. at java.lang.Thread.run(Thread.java:748)
  361. 2019-06-03 18:38:27,945 INFO [c.c.h.v.m.HostMO] (DirectAgent-6:ctx-220f2646 esx-0001-a-001.example.org, job-3/job-31, cmd: StopCommand) VM v-1-VM not found in host cache
  362. 2019-06-03 18:38:27,969 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-6:ctx-220f2646 esx-0001-a-001.example.org, job-3/job-31, cmd: StopCommand) VM v-1-VM is no longer in vSphere
  363. 2019-06-03 18:38:27,972 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-12:ctx-51153734 esx-0001-a-001.example.org, job-4/job-32, cmd: StopCommand) Executing resource StopCommand:
  364. 2019-06-03 18:38:27,988 INFO [c.c.h.v.m.HostMO] (DirectAgent-12:ctx-51153734 esx-0001-a-001.example.org, job-4/job-32, cmd: StopCommand) VM s-2-VM not found in host cache
  365. 2019-06-03 18:38:28,018 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-12:ctx-51153734 esx-0001-a-001.example.org, job-4/job-32, cmd: StopCommand) VM s-2-VM is no longer in vSphere
  366. 2019-06-03 18:38:28,033 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31 ctx-29f14560) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  367. 2019-06-03 18:38:28,034 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31 ctx-29f14560) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  368. 2019-06-03 18:38:28,034 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31) Unable to complete AsyncJobVO {id:31, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052023320, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Jun 03 18:38:26 CDT 2019}, job origin:3
  369. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  370. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  371. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  372. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  373. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  374. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  375. at java.lang.reflect.Method.invoke(Method.java:498)
  376. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  377. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  378. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  379. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  380. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  381. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  382. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  383. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  384. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  385. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  386. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  387. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  388. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  389. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  390. at java.lang.Thread.run(Thread.java:748)
  391. 2019-06-03 18:38:28,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-3:ctx-13a37e55 job-3/job-31) Remove job-31 from job monitoring
  392. 2019-06-03 18:38:28,066 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-5751ca93) Exception while trying to start console proxy
  393. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  394. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  395. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  396. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  397. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  398. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  399. at java.lang.reflect.Method.invoke(Method.java:498)
  400. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  401. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  402. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  403. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  404. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  405. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  406. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  407. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  408. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  409. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  410. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  411. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  412. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  413. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  414. at java.lang.Thread.run(Thread.java:748)
  415. 2019-06-03 18:38:28,067 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-5751ca93) Unable to start console proxy vm for standby capacity, vm id : 1, will recycle it and start a new one
  416. 2019-06-03 18:38:28,113 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32 ctx-40efb72e) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  417. 2019-06-03 18:38:28,114 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32 ctx-40efb72e) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  418. 2019-06-03 18:38:28,114 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32) Unable to complete AsyncJobVO {id:32, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052023320, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Jun 03 18:38:26 CDT 2019}, job origin:4
  419. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  420. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  421. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  422. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  423. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  424. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  425. at java.lang.reflect.Method.invoke(Method.java:498)
  426. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  427. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  428. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  429. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  430. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  431. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  432. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  433. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  434. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  435. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  436. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  437. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  438. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  439. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  440. at java.lang.Thread.run(Thread.java:748)
  441. 2019-06-03 18:38:28,126 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-4:ctx-f434fed5 job-4/job-32) Remove job-32 from job monitoring
  442. 2019-06-03 18:38:28,136 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-57d497bf) Exception while trying to start secondary storage vm
  443. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  444. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  445. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  446. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  447. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  448. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  449. at java.lang.reflect.Method.invoke(Method.java:498)
  450. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  451. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  452. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  453. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  454. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  455. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  456. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  457. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  458. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  459. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  460. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  461. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  462. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  463. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  464. at java.lang.Thread.run(Thread.java:748)
  465. 2019-06-03 18:38:28,138 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-57d497bf) Unable to start secondary storage vm for standby capacity, vm id : 2, will recycle it and start a new one
  466. 2019-06-03 18:38:28,702 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:ctx-348380f8 job-3/job-33) Add job-33 into job monitoring
  467. 2019-06-03 18:38:28,706 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-10c708c1 job-4/job-34) Add job-34 into job monitoring
  468. 2019-06-03 18:38:28,718 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:ctx-348380f8 job-3/job-33) Remove job-33 from job monitoring
  469. 2019-06-03 18:38:28,725 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-10c708c1 job-4/job-34) Remove job-34 from job monitoring
  470. 2019-06-03 18:38:28,878 WARN [o.a.c.alerts] (consoleproxy-1:ctx-5751ca93) AlertType:: 10 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Console proxy creation failure. zone: nonprod, error details: null
  471. 2019-06-03 18:38:28,906 WARN [o.a.c.alerts] (secstorage-1:ctx-57d497bf) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: nonprod, error details: null
  472. 2019-06-03 18:38:28,909 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-57d497bf) Primary secondary storage is not even started, wait until next turn
  473. 2019-06-03 18:38:30,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4e106092) Begin cleanup expired async-jobs
  474. 2019-06-03 18:38:30,690 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4e106092) End cleanup expired async-jobs
  475. 2019-06-03 18:38:39,602 INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-1:ctx-fb0bba9f) Scan hung worker VM to recycle
  476. 2019-06-03 18:38:40,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c8ea696e) Begin cleanup expired async-jobs
  477. 2019-06-03 18:38:40,691 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c8ea696e) End cleanup expired async-jobs
  478. 2019-06-03 18:38:50,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-1389a09e) Begin cleanup expired async-jobs
  479. 2019-06-03 18:38:50,690 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-1389a09e) End cleanup expired async-jobs
  480. 2019-06-03 18:38:56,579 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a36a39a6) No stopped console proxy is available, need to allocate a new console proxy
  481. 2019-06-03 18:38:56,593 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-24c3e5f4) No running secondary storage vms found in datacenter id=1, starting one
  482. 2019-06-03 18:38:56,597 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-24c3e5f4) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
  483. 2019-06-03 18:38:56,643 ERROR [c.c.n.IpAddressManagerImpl] (secstorage-1:ctx-24c3e5f4) Failed to fetch any free public IP address
  484. 2019-06-03 18:38:56,651 WARN [o.a.c.alerts] (secstorage-1:ctx-24c3e5f4) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: nonprod, error details: Failed to fetch any free public IP address
  485. 2019-06-03 18:38:56,656 WARN [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-24c3e5f4) Unexpected exception Failed to fetch any free public IP address
  486. com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any free public IP address
  487. at com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:855)
  488. at com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:701)
  489. at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
  490. at com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:701)
  491. at com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:688)
  492. at com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:122)
  493. at com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:167)
  494. at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:846)
  495. at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:772)
  496. at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
  497. at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
  498. at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
  499. at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:732)
  500. at com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:431)
  501. at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
  502. at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
  503. at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
  504. at com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:422)
  505. at com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:478)
  506. at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:660)
  507. at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:506)
  508. at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:746)
  509. at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1388)
  510. at org.apache.cloudstack.secondarystorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:126)
  511. at org.apache.cloudstack.secondarystorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  512. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:114)
  513. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  514. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
  515. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
  516. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  517. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  518. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  519. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  520. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  521. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  522. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  523. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  524. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  525. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  526. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  527. at java.lang.Thread.run(Thread.java:748)
  528. 2019-06-03 18:38:56,692 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35) Add job-35 into job monitoring
  529. 2019-06-03 18:38:56,736 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35 ctx-4291c48a) Storage pool null (1) does not supply IOPS capacity, assuming enough capacity
  530. 2019-06-03 18:38:56,841 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-2:ctx-e645aa2a esx-0001-a-001.example.org, job-3/job-35, cmd: CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler successfully reconfigured
  531. 2019-06-03 18:38:56,941 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-2:ctx-e645aa2a esx-0001-a-001.example.org, job-3/job-35, cmd: CopyCommand) creating full clone from template
  532. 2019-06-03 18:38:57,417 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-2:ctx-e645aa2a esx-0001-a-001.example.org, job-3/job-35, cmd: CopyCommand) [ignored]failed toi get message for exception: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  533. 2019-06-03 18:38:57,417 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-2:ctx-e645aa2a esx-0001-a-001.example.org, job-3/job-35, cmd: CopyCommand) clone volume from base image failed due to Exception: java.lang.RuntimeException
  534. Message: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  535.  
  536. java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  537. at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:358)
  538. at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createFullClone(VirtualMachineMO.java:720)
  539. at com.cloud.storage.resource.VmwareStorageProcessor.createVMFullClone(VmwareStorageProcessor.java:749)
  540. at com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:828)
  541. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:92)
  542. at com.cloud.storage.resource.VmwareStorageSubsystemCommandHandler.execute(VmwareStorageSubsystemCommandHandler.java:197)
  543. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:56)
  544. at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
  545. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
  546. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  547. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  548. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  549. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  550. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  551. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  552. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  553. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  554. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  555. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  556. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  557. at java.lang.Thread.run(Thread.java:748)
  558. 2019-06-03 18:38:57,426 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35 ctx-4291c48a) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@3f622083), no need to delete from object in store ref table
  559. 2019-06-03 18:38:57,427 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35 ctx-4291c48a) Unable to contact resource.
  560. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[3|vm=3|ROOT]:java.lang.RuntimeException: Error caused by file /vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
  561. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1301)
  562. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1376)
  563. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1113)
  564. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  565. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  566. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  567. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  568. at java.lang.reflect.Method.invoke(Method.java:498)
  569. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  570. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  571. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  572. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  573. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  574. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  575. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  576. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  577. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  578. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  579. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  580. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  581. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  582. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  583. at java.lang.Thread.run(Thread.java:748)
  584. 2019-06-03 18:38:57,434 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-13:ctx-e7f74178 esx-0001-a-001.example.org, job-3/job-35, cmd: StopCommand) Executing resource StopCommand:
  585. 2019-06-03 18:38:57,450 INFO [c.c.h.v.m.HostMO] (DirectAgent-13:ctx-e7f74178 esx-0001-a-001.example.org, job-3/job-35, cmd: StopCommand) VM v-3-VM not found in host cache
  586. 2019-06-03 18:38:57,467 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-13:ctx-e7f74178 esx-0001-a-001.example.org, job-3/job-35, cmd: StopCommand) VM v-3-VM is no longer in vSphere
  587. 2019-06-03 18:38:57,540 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35 ctx-4291c48a) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-3-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  588. 2019-06-03 18:38:57,540 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35 ctx-4291c48a) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-3-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  589. 2019-06-03 18:38:57,540 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35) Unable to complete AsyncJobVO {id:35, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052023320, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Jun 03 18:38:56 CDT 2019}, job origin:3
  590. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-3-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  591. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  592. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  593. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  594. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  595. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  596. at java.lang.reflect.Method.invoke(Method.java:498)
  597. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  598. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  599. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  600. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  601. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  602. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  603. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  604. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  605. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  606. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  607. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  608. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  609. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  610. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  611. at java.lang.Thread.run(Thread.java:748)
  612. 2019-06-03 18:38:57,549 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-7:ctx-b0c75c53 job-3/job-35) Remove job-35 from job monitoring
  613. 2019-06-03 18:38:57,557 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a36a39a6) Exception while trying to start console proxy
  614. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-3-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  615. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1075)
  616. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937)
  617. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  618. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  619. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  620. at java.lang.reflect.Method.invoke(Method.java:498)
  621. at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
  622. at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100)
  623. at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
  624. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
  625. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  626. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  627. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  628. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  629. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  630. at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
  631. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  632. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  633. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  634. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  635. at java.lang.Thread.run(Thread.java:748)
  636. 2019-06-03 18:38:57,558 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a36a39a6) Unable to start console proxy vm for standby capacity, vm id : 3, will recycle it and start a new one
  637. 2019-06-03 18:38:57,562 WARN [o.a.c.alerts] (consoleproxy-1:ctx-a36a39a6) AlertType:: 10 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Console proxy creation failure. zone: nonprod, error details: null
  638. 2019-06-03 18:38:59,544 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp174573182-20:ctx-793b025a ctx-ab8dc47e) Received unknown parameters for command listSystemVms. Unknown parameters : listall
  639. 2019-06-03 18:39:00,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9a3f38bb) Begin cleanup expired async-jobs
  640. 2019-06-03 18:39:00,690 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9a3f38bb) End cleanup expired async-jobs
  641. 2019-06-03 18:39:10,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a470c795) Begin cleanup expired async-jobs
  642. 2019-06-03 18:39:10,690 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a470c795) End cleanup expired async-jobs
  643. 2019-06-03 18:39:20,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-3063c494) Begin cleanup expired async-jobs
  644. 2019-06-03 18:39:20,690 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-3063c494) End cleanup expired async-jobs
  645. 2019-06-03 18:39:30,689 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-94a4f244) Begin cleanup expired async-jobs
  646. 2019-06-03 18:39:30,696 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-94a4f244) End cleanup expired async-jobs
  647. 2019-06-03 18:39:39,602 INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-5:ctx-124474d8) Scan hung worker VM to recycle
  648. 2019-06-03 18:39:40,688 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-8a99705a) Begin cleanup expired async-jobs
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!