API tools faq
paste
juesor

Cloudstack SSVM starting....

juesor
Sep 26th, 2014
376
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 61.43 KB | None | 0 0
raw download clone embed print report
  1. 2014-09-26 12:13:07,900 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-dad6c8fa) HostStatsCollector is running...
  2. 2014-09-26 12:13:08,506 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-dad6c8fa) Seq 3-102367929: Received: { Ans: , MgmtId: 345050433729, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  3. 2014-09-26 12:13:09,091 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-dad6c8fa) Seq 4-64946871: Received: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  4. 2014-09-26 12:13:11,474 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-3453b1f2) Found 0 routers to update status.
  5. 2014-09-26 12:13:11,477 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-3453b1f2) Found 0 networks to update RvR status.
  6. 2014-09-26 12:13:12,132 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 3
  7. 2014-09-26 12:13:16,814 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-4:null) Ping from 4
  8. 2014-09-26 12:13:27,299 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-9597c141) VmStatsCollector is running...
  9. 2014-09-26 12:13:37,202 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-24fe054a) StorageCollector is running...
  10. 2014-09-26 12:13:37,208 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-24fe054a) There is no secondary storage VM for secondary storage host Secondary
  11. 2014-09-26 12:13:41,474 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-cd1bbfdc) Found 0 routers to update status.
  12. 2014-09-26 12:13:41,476 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-cd1bbfdc) Found 0 networks to update RvR status.
  13. 2014-09-26 12:13:46,532 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-9aa6855e) Resetting hosts suitable for reconnect
  14. 2014-09-26 12:13:46,534 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-9aa6855e) Completed resetting hosts suitable for reconnect
  15. 2014-09-26 12:13:46,534 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-9aa6855e) Acquiring hosts for clusters already owned by this management server
  16. 2014-09-26 12:13:46,535 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-9aa6855e) Completed acquiring hosts for clusters already owned by this management server
  17. 2014-09-26 12:13:46,535 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-9aa6855e) Acquiring hosts for clusters not owned by any management server
  18. 2014-09-26 12:13:46,535 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-9aa6855e) Completed acquiring hosts for clusters not owned by any management server
  19. 2014-09-26 12:14:09,092 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-d2abf04b) HostStatsCollector is running...
  20. 2014-09-26 12:14:09,735 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-d2abf04b) Seq 3-102367930: Received: { Ans: , MgmtId: 345050433729, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  21. 2014-09-26 12:14:10,348 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-d2abf04b) Seq 4-64946873: Received: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  22. 2014-09-26 12:14:11,475 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-ac2684e3) Found 0 routers to update status.
  23. 2014-09-26 12:14:11,476 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-ac2684e3) Found 0 networks to update RvR status.
  24. 2014-09-26 12:14:12,106 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 3
  25. 2014-09-26 12:14:16,814 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) Ping from 4
  26. 2014-09-26 12:14:25,457 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:null) Seq 4-64946180: Processing: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"","wait":0}}] }
  27. 2014-09-26 12:14:27,306 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-6cdfa056) VmStatsCollector is running...
  28. 2014-09-26 12:14:41,475 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8719702e) Found 0 routers to update status.
  29. 2014-09-26 12:14:41,477 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8719702e) Found 0 networks to update RvR status.
  30. 2014-09-26 12:14:42,356 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:null) Seq 4-64946867: Processing: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error Child process (/bin/mount Flex-csms:/mnt/secondary/template/tmpl/1/3 /mnt/9908184c-9739-3ba7-9501-09167c6905d7) unexpected exit status 32: mount.nfs: Connection timed out\n","wait":0}}] }
  31. 2014-09-26 12:14:42,356 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-d1b762c2) Seq 4-64946867: Received: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  32. 2014-09-26 12:14:42,358 ERROR [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-d1b762c2) Failed to start instance VM[SecondaryStorageVm|s-153-VM]
  33. java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  34. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  35. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  36. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  37. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  38. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  39. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  40. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  41. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  42. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  43. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  44. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  45. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  46. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  47. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  48. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  49. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  50. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  51. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  52. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  53. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  54. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  55. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  56. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  57. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  58. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  59. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  60. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  61. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  62. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  63. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  64. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  65. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  66. at java.lang.Thread.run(Thread.java:745)
  67. Caused by: java.lang.reflect.InvocationTargetException
  68. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  69. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  70. at java.lang.reflect.Method.invoke(Method.java:606)
  71. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  72. ... 32 more
  73. Caused by: java.lang.NullPointerException
  74. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  75. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  76. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  77. ... 35 more
  78. 2014-09-26 12:14:42,365 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-d1b762c2) Cleaning up resources for the vm VM[SecondaryStorageVm|s-153-VM] in Starting state
  79. 2014-09-26 12:14:42,369 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-d1b762c2) Seq 4-64946874: Sending { Cmd , MgmtId: 345050433729, via: 4(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-153-VM","wait":0}}] }
  80. 2014-09-26 12:14:42,521 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-24fe054a) Seq 4-64946872: Received: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  81. 2014-09-26 12:14:42,542 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 4-64946874: Processing: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
  82. 2014-09-26 12:14:42,542 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-d1b762c2) Seq 4-64946874: Received: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, { StopAnswer } }
  83. 2014-09-26 12:14:42,560 DEBUG [c.c.n.g.ControlNetworkGuru] (secstorage-1:ctx-d1b762c2) Released nic: NicProfile[609-153-null-null-null
  84. 2014-09-26 12:14:42,577 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-d1b762c2) Releasing ip address for reservationId=c398b15b-90fe-4046-91a0-768f690bbabc, instance=610
  85. 2014-09-26 12:14:42,578 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-d1b762c2) Released nic: NicProfile[610-153-null-null-null
  86. 2014-09-26 12:14:42,599 DEBUG [c.c.n.g.StorageNetworkGuru] (secstorage-1:ctx-d1b762c2) Release an storage ip 10.83.0.35
  87. 2014-09-26 12:14:42,607 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-d1b762c2) Successfully released network resources for the vm VM[SecondaryStorageVm|s-153-VM]
  88. 2014-09-26 12:14:42,607 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-d1b762c2) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-153-VM] in Starting state
  89. 2014-09-26 12:14:42,645 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-d1b762c2) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 4
  90. 2014-09-26 12:14:42,654 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-d1b762c2) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  91. 2014-09-26 12:14:42,655 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-d1b762c2) Hosts's actual total RAM: 67521052672 and RAM after applying overprovisioning: 67521052672
  92. 2014-09-26 12:14:42,655 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-d1b762c2) release cpu from host: 4, old used: 1000,reserved: 0, actual total: 48000, total with overprovisioning: 48000; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
  93. 2014-09-26 12:14:42,655 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-d1b762c2) release mem from host: 4, old used: 1610612736,reserved: 0, total: 67521052672; new used: 1073741824,reserved:0; movedfromreserved: false,moveToReserveredfalse
  94. 2014-09-26 12:14:42,674 WARN [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-d1b762c2) Exception while trying to start secondary storage vm
  95. com.cloud.exception.AgentUnavailableException: Resource [Host:4] is unreachable: Host 4: Unable to start instance due to InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  96. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1073)
  97. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  98. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  99. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  100. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  101. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  102. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  103. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  104. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  105. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  106. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  107. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  108. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  109. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  110. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  111. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  112. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  113. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  114. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  115. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  116. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  117. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  118. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  119. at java.lang.Thread.run(Thread.java:745)
  120. Caused by: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  121. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  122. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  123. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  124. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  125. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  126. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  127. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  128. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  129. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  130. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  131. ... 23 more
  132. Caused by: java.lang.reflect.InvocationTargetException
  133. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  134. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  135. at java.lang.reflect.Method.invoke(Method.java:606)
  136. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  137. ... 32 more
  138. Caused by: java.lang.NullPointerException
  139. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  140. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  141. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  142. ... 35 more
  143. 2014-09-26 12:14:42,675 INFO [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-d1b762c2) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 153, will recycle it and start a new one
  144. 2014-09-26 12:14:42,675 INFO [c.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-d1b762c2) Primary secondary storage is not even started, wait until next turn
  145. 2014-09-26 12:14:42,700 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-5411ecdf) Zone 1 is ready to launch secondary storage VM
  146. 2014-09-26 12:14:42,708 INFO [c.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-5411ecdf) No running secondary storage vms found in datacenter id=1, starting one
  147. 2014-09-26 12:14:42,717 INFO [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-5411ecdf) Found a stopped secondary storage vm, bring it up to running pool. secStorageVm vm id : 153
  148. 2014-09-26 12:14:42,734 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
  149. 2014-09-26 12:14:42,734 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-5411ecdf) Successfully transitioned to start state for VM[SecondaryStorageVm|s-153-VM] reservation id = e95f0b37-a9bc-48af-be5a-8685a71bfd16
  150. 2014-09-26 12:14:42,743 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-5411ecdf) Trying to deploy VM, vm has dcId: 1 and podId: 1
  151. 2014-09-26 12:14:42,743 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-5411ecdf) Deploy avoids pods: null, clusters: null, hosts: null
  152. 2014-09-26 12:14:42,749 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Deploy avoids pods: null, clusters: null, hosts: null
  153. 2014-09-26 12:14:42,751 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@362e0807
  154. 2014-09-26 12:14:42,751 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 536870912
  155. 2014-09-26 12:14:42,751 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Is ROOT volume READY (pool already allocated)?: No
  156. 2014-09-26 12:14:42,751 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-5411ecdf) Searching resources only under specified Pod: 1
  157. 2014-09-26 12:14:42,751 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-5411ecdf) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  158. 2014-09-26 12:14:42,763 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Checking resources in Cluster: 1 under Pod: 1
  159. 2014-09-26 12:14:42,763 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  160. 2014-09-26 12:14:42,767 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) FirstFitAllocator has 2 hosts to check for allocation: [Host[-3-Routing], Host[-4-Routing]]
  161. 2014-09-26 12:14:42,774 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Found 2 hosts for allocation after prioritization: [Host[-3-Routing], Host[-4-Routing]]
  162. 2014-09-26 12:14:42,774 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
  163. 2014-09-26 12:14:42,784 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Host: 3 has cpu capability (cpu:24, speed:2001) to support requested CPU: 1 and requested speed: 500
  164. 2014-09-26 12:14:42,785 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Checking if host: 3 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  165. 2014-09-26 12:14:42,788 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  166. 2014-09-26 12:14:42,788 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Free CPU: 48024 , Requested CPU: 500
  167. 2014-09-26 12:14:42,788 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Free RAM: 67521073152 , Requested RAM: 536870912
  168. 2014-09-26 12:14:42,788 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Host has enough CPU and RAM available
  169. 2014-09-26 12:14:42,788 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 3, used: 0, reserved: 0, actual total: 48024, total with overprovisioning: 48024; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  170. 2014-09-26 12:14:42,788 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 3, used: 0, reserved: 0, total: 67521073152; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  171. 2014-09-26 12:14:42,788 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Found a suitable host, adding to list: 3
  172. 2014-09-26 12:14:42,799 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Host: 4 has cpu capability (cpu:24, speed:2000) to support requested CPU: 1 and requested speed: 500
  173. 2014-09-26 12:14:42,799 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Checking if host: 4 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  174. 2014-09-26 12:14:42,802 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  175. 2014-09-26 12:14:42,802 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Free CPU: 47500 , Requested CPU: 500
  176. 2014-09-26 12:14:42,802 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Free RAM: 66447310848 , Requested RAM: 536870912
  177. 2014-09-26 12:14:42,802 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Host has enough CPU and RAM available
  178. 2014-09-26 12:14:42,803 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 4, used: 500, reserved: 0, actual total: 48000, total with overprovisioning: 48000; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  179. 2014-09-26 12:14:42,803 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 4, used: 1073741824, reserved: 0, total: 67521052672; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  180. 2014-09-26 12:14:42,803 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Found a suitable host, adding to list: 4
  181. 2014-09-26 12:14:42,803 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-5411ecdf FirstFitRoutingAllocator) Host Allocator returning 2 suitable hosts
  182. 2014-09-26 12:14:42,805 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Checking suitable pools for volume (Id, Type): (153,ROOT)
  183. 2014-09-26 12:14:42,805 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) We need to allocate new storagepool for this volume
  184. 2014-09-26 12:14:42,805 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Calling StoragePoolAllocators to find suitable pools
  185. 2014-09-26 12:14:42,808 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  186. 2014-09-26 12:14:42,808 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) ClusterScopeStoragePoolAllocator looking for storage pool
  187. 2014-09-26 12:14:42,808 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) Looking for pools in dc: 1 pod:1 cluster:1
  188. 2014-09-26 12:14:42,810 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) Found pools matching tags: [Pool[2|CLVM], Pool[3|CLVM]]
  189. 2014-09-26 12:14:42,812 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) Removing pool Pool[2|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  190. 2014-09-26 12:14:42,812 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) Removing pool Pool[3|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  191. 2014-09-26 12:14:42,814 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) Checking if storage pool is suitable, name: null ,poolId: 2
  192. 2014-09-26 12:14:42,818 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-5411ecdf) Checking pool 2 for storage, totalSize: 2997945892864, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  193. 2014-09-26 12:14:42,823 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-5411ecdf) Checking pool: 2 for volume allocation [Vol[153|vm=153|ROOT]], maxSize : 2997945892864, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  194. 2014-09-26 12:14:42,824 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) Checking if storage pool is suitable, name: null ,poolId: 3
  195. 2014-09-26 12:14:42,825 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-5411ecdf) Checking pool 3 for storage, totalSize: 2997950087168, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  196. 2014-09-26 12:14:42,830 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-5411ecdf) Checking pool: 3 for volume allocation [Vol[153|vm=153|ROOT]], maxSize : 2997950087168, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  197. 2014-09-26 12:14:42,830 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-5411ecdf) ClusterScopeStoragePoolAllocator returning 2 suitable storage pools
  198. 2014-09-26 12:14:42,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  199. 2014-09-26 12:14:42,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Checking if host: 3 can access any suitable storage pool for volume: ROOT
  200. 2014-09-26 12:14:42,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Host: 3 can access pool: 3
  201. 2014-09-26 12:14:42,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Found a potential host id: 3 name: Flex-H2.flexhost.local and associated storage pools for this VM
  202. 2014-09-26 12:14:42,832 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-5411ecdf) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(3)-Storage(Volume(153|ROOT-->Pool(3))]
  203. 2014-09-26 12:14:42,832 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-5411ecdf) Deployment found - P0=VM[SecondaryStorageVm|s-153-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(3)-Storage(Volume(153|ROOT-->Pool(3))]
  204. 2014-09-26 12:14:42,850 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 3 host id before state transition: null
  205. 2014-09-26 12:14:42,859 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  206. 2014-09-26 12:14:42,859 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) We are allocating VM, increasing the used capacity of this host:3
  207. 2014-09-26 12:14:42,859 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) Current Used CPU: 0 , Free CPU:48024 ,Requested CPU: 500
  208. 2014-09-26 12:14:42,859 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) Current Used RAM: 0 , Free RAM:67521073152 ,Requested RAM: 536870912
  209. 2014-09-26 12:14:42,859 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) CPU STATS after allocation: for host: 3, old used: 0, old reserved: 0, actual total: 48024, total with overprovisioning: 48024; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
  210. 2014-09-26 12:14:42,859 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-5411ecdf) RAM STATS after allocation: for host: 3, old used: 0, old reserved: 0, total: 67521073152; new used: 536870912, reserved: 0; requested mem: 536870912,alloc_from_last:false
  211. 2014-09-26 12:14:42,865 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-5411ecdf) VM is being created in podId: 1
  212. 2014-09-26 12:14:42,877 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-5411ecdf) Network id=204 is already implemented
  213. 2014-09-26 12:14:42,918 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-5411ecdf) Asking VirtualRouter to prepare for Nic[608-153-null-205.185.158.242]
  214. 2014-09-26 12:14:42,921 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-5411ecdf) Asking SecurityGroupProvider to prepare for Nic[608-153-null-205.185.158.242]
  215. 2014-09-26 12:14:42,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-5411ecdf) Network id=201 is already implemented
  216. 2014-09-26 12:14:42,979 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-5411ecdf) Network id=200 is already implemented
  217. 2014-09-26 12:14:43,010 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-5411ecdf) Allocated a nic NicProfile[610-153-e95f0b37-a9bc-48af-be5a-8685a71bfd16-192.168.30.11-null for VM[SecondaryStorageVm|s-153-VM]
  218. 2014-09-26 12:14:43,024 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-5411ecdf) Network id=202 is already implemented
  219. 2014-09-26 12:14:43,039 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) Seq 3-102367928: Processing: { Ans: , MgmtId: 345050433729, via: 3, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error Child process (/bin/mount Flex-csms:/mnt/secondary/template/tmpl/1/3 /mnt/9908184c-9739-3ba7-9501-09167c6905d7) unexpected exit status 32: mount.nfs: Connection timed out\n","wait":0}}] }
  220. 2014-09-26 12:14:43,039 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-17e1aa5e) Seq 3-102367928: Received: { Ans: , MgmtId: 345050433729, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  221. 2014-09-26 12:14:43,040 ERROR [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Failed to start instance VM[ConsoleProxy|v-1-VM]
  222. java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  223. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  224. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  225. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  226. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  227. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  228. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  229. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  230. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  231. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  232. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  233. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  234. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  235. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  236. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  237. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  238. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  239. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  240. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  241. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  242. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  243. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  244. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  245. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  246. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  247. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  248. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  249. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  250. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  251. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  252. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  253. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  254. at java.lang.Thread.run(Thread.java:745)
  255. Caused by: java.lang.reflect.InvocationTargetException
  256. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  257. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  258. at java.lang.reflect.Method.invoke(Method.java:606)
  259. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  260. ... 31 more
  261. Caused by: java.lang.NullPointerException
  262. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  263. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  264. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  265. ... 34 more
  266. 2014-09-26 12:14:43,048 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Cleaning up resources for the vm VM[ConsoleProxy|v-1-VM] in Starting state
  267. 2014-09-26 12:14:43,052 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-17e1aa5e) Seq 4-64946875: Sending { Cmd , MgmtId: 345050433729, via: 4(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-1-VM","wait":0}}] }
  268. 2014-09-26 12:14:43,060 DEBUG [c.c.n.g.StorageNetworkGuru] (secstorage-1:ctx-5411ecdf) Allocated a storage nic NicProfile[611-153-e95f0b37-a9bc-48af-be5a-8685a71bfd16-10.83.0.186-storage://830 for VM[SecondaryStorageVm|s-153-VM]
  269. 2014-09-26 12:14:43,071 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-5411ecdf) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-153-VM]
  270. 2014-09-26 12:14:43,087 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-5411ecdf) template 3 is already in store:6, type:Image
  271. 2014-09-26 12:14:43,092 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-5411ecdf) template 3 is already in store:3, type:Primary
  272. 2014-09-26 12:14:43,134 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-5411ecdf) template 3 is already in store:6, type:Image
  273. 2014-09-26 12:14:43,145 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-5411ecdf) Seq 4-64946876: Sending { Cmd , MgmtId: 345050433729, via: 4(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2","uuid":"7e218d6a-403d-11e4-b04c-0050569b56d4","id":3,"format":"QCOW2","accountId":1,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://Flex-csms/mnt/secondary","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"13076277-20c9-4a95-b350-3d13de712f0d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b5852e3c-936d-42a1-aec6-067d64069a02","id":3,"poolType":"CLVM","host":"localhost","path":"/vg_s2c0","port":0,"url":"CLVM://localhost//vg_s2c0/?ROLE=Primary&STOREUUID=b5852e3c-936d-42a1-aec6-067d64069a02"}},"name":"ROOT-153","size":0,"volumeId":153,"vmName":"s-153-VM","accountId":1,"format":"QCOW2","id":153,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  274. 2014-09-26 12:14:43,180 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-24fe054a) Seq 3-102367931: Received: { Ans: , MgmtId: 345050433729, via: 3, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  275. 2014-09-26 12:14:43,209 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 4-64946875: Processing: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
  276. 2014-09-26 12:14:43,209 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-17e1aa5e) Seq 4-64946875: Received: { Ans: , MgmtId: 345050433729, via: 4, Ver: v1, Flags: 10, { StopAnswer } }
  277. 2014-09-26 12:14:43,225 DEBUG [c.c.n.g.ControlNetworkGuru] (consoleproxy-1:ctx-17e1aa5e) Released nic: NicProfile[2-1-null-null-null
  278. 2014-09-26 12:14:43,241 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (consoleproxy-1:ctx-17e1aa5e) Releasing ip address for reservationId=cb0a9b0e-9cde-4807-a336-e5ea31f48015, instance=3
  279. 2014-09-26 12:14:43,241 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-17e1aa5e) Released nic: NicProfile[3-1-null-null-null
  280. 2014-09-26 12:14:43,251 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Successfully released network resources for the vm VM[ConsoleProxy|v-1-VM]
  281. 2014-09-26 12:14:43,251 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Successfully cleanued up resources for the vm VM[ConsoleProxy|v-1-VM] in Starting state
  282. 2014-09-26 12:14:43,267 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-17e1aa5e) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 4
  283. 2014-09-26 12:14:43,274 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  284. 2014-09-26 12:14:43,275 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Hosts's actual total RAM: 67521052672 and RAM after applying overprovisioning: 67521052672
  285. 2014-09-26 12:14:43,275 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-17e1aa5e) release cpu from host: 4, old used: 500,reserved: 0, actual total: 48000, total with overprovisioning: 48000; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  286. 2014-09-26 12:14:43,275 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-17e1aa5e) release mem from host: 4, old used: 1073741824,reserved: 0, total: 67521052672; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  287. 2014-09-26 12:14:43,283 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-17e1aa5e) Exception while trying to start console proxy
  288. com.cloud.exception.AgentUnavailableException: Resource [Host:4] is unreachable: Host 4: Unable to start instance due to InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  289. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1073)
  290. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  291. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  292. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  293. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  294. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  295. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  296. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  297. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  298. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  299. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  300. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  301. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  302. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  303. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  304. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  305. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  306. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  307. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  308. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  309. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  310. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  311. at java.lang.Thread.run(Thread.java:745)
  312. Caused by: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  313. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  314. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  315. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  316. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  317. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  318. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  319. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  320. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  321. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  322. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  323. ... 22 more
  324. Caused by: java.lang.reflect.InvocationTargetException
  325. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  326. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  327. at java.lang.reflect.Method.invoke(Method.java:606)
  328. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  329. ... 31 more
  330. Caused by: java.lang.NullPointerException
  331. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  332. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  333. at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
  334. ... 34 more
  335. 2014-09-26 12:14:43,310 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Zone 1 is ready to launch console proxy
  336. 2014-09-26 12:14:43,310 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Expand console proxy standby capacity for zone Rushford
  337. 2014-09-26 12:14:43,314 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Found a stopped console proxy, bring it up to running pool. proxy vm id : 1
  338. 2014-09-26 12:14:43,329 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
  339. 2014-09-26 12:14:43,329 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Successfully transitioned to start state for VM[ConsoleProxy|v-1-VM] reservation id = 44dde5f9-0eef-4159-ab4a-09b4e97a85aa
  340. 2014-09-26 12:14:43,337 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Trying to deploy VM, vm has dcId: 1 and podId: 1
  341. 2014-09-26 12:14:43,337 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Deploy avoids pods: null, clusters: null, hosts: null
  342. 2014-09-26 12:14:43,343 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Deploy avoids pods: null, clusters: null, hosts: null
  343. 2014-09-26 12:14:43,344 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@362e0807
  344. 2014-09-26 12:14:43,344 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 1073741824
  345. 2014-09-26 12:14:43,344 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Is ROOT volume READY (pool already allocated)?: No
  346. 2014-09-26 12:14:43,344 DEBUG [c.c.d.FirstFitPlanner] (consoleproxy-1:ctx-4ca29ebb) Searching resources only under specified Pod: 1
  347. 2014-09-26 12:14:43,345 DEBUG [c.c.d.FirstFitPlanner] (consoleproxy-1:ctx-4ca29ebb) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  348. 2014-09-26 12:14:43,355 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking resources in Cluster: 1 under Pod: 1
  349. 2014-09-26 12:14:43,355 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  350. 2014-09-26 12:14:43,359 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) FirstFitAllocator has 2 hosts to check for allocation: [Host[-3-Routing], Host[-4-Routing]]
  351. 2014-09-26 12:14:43,364 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Found 2 hosts for allocation after prioritization: [Host[-3-Routing], Host[-4-Routing]]
  352. 2014-09-26 12:14:43,364 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=1024
  353. 2014-09-26 12:14:43,374 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Host: 3 has cpu capability (cpu:24, speed:2001) to support requested CPU: 1 and requested speed: 500
  354. 2014-09-26 12:14:43,374 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Checking if host: 3 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  355. 2014-09-26 12:14:43,376 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  356. 2014-09-26 12:14:43,376 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Free CPU: 47524 , Requested CPU: 500
  357. 2014-09-26 12:14:43,376 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Free RAM: 66984202240 , Requested RAM: 1073741824
  358. 2014-09-26 12:14:43,376 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Host has enough CPU and RAM available
  359. 2014-09-26 12:14:43,377 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 3, used: 500, reserved: 0, actual total: 48024, total with overprovisioning: 48024; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  360. 2014-09-26 12:14:43,377 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 3, used: 536870912, reserved: 0, total: 67521073152; requested mem: 1073741824,alloc_from_last_host?:false ,considerReservedCapacity?: true
  361. 2014-09-26 12:14:43,377 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Found a suitable host, adding to list: 3
  362. 2014-09-26 12:14:43,386 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Host: 4 has cpu capability (cpu:24, speed:2000) to support requested CPU: 1 and requested speed: 500
  363. 2014-09-26 12:14:43,386 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Checking if host: 4 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  364. 2014-09-26 12:14:43,389 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  365. 2014-09-26 12:14:43,389 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Free CPU: 48000 , Requested CPU: 500
  366. 2014-09-26 12:14:43,389 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Free RAM: 67521052672 , Requested RAM: 1073741824
  367. 2014-09-26 12:14:43,389 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Host has enough CPU and RAM available
  368. 2014-09-26 12:14:43,389 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 4, used: 0, reserved: 0, actual total: 48000, total with overprovisioning: 48000; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  369. 2014-09-26 12:14:43,389 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 4, used: 0, reserved: 0, total: 67521052672; requested mem: 1073741824,alloc_from_last_host?:false ,considerReservedCapacity?: true
  370. 2014-09-26 12:14:43,389 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Found a suitable host, adding to list: 4
  371. 2014-09-26 12:14:43,389 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-4ca29ebb FirstFitRoutingAllocator) Host Allocator returning 2 suitable hosts
  372. 2014-09-26 12:14:43,391 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking suitable pools for volume (Id, Type): (1,ROOT)
  373. 2014-09-26 12:14:43,391 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) We need to allocate new storagepool for this volume
  374. 2014-09-26 12:14:43,391 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Calling StoragePoolAllocators to find suitable pools
  375. 2014-09-26 12:14:43,392 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  376. 2014-09-26 12:14:43,392 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) ClusterScopeStoragePoolAllocator looking for storage pool
  377. 2014-09-26 12:14:43,392 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) Looking for pools in dc: 1 pod:1 cluster:1
  378. 2014-09-26 12:14:43,394 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) Found pools matching tags: [Pool[2|CLVM], Pool[3|CLVM]]
  379. 2014-09-26 12:14:43,396 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) Removing pool Pool[2|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  380. 2014-09-26 12:14:43,396 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) Removing pool Pool[3|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  381. 2014-09-26 12:14:43,397 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) Checking if storage pool is suitable, name: null ,poolId: 2
  382. 2014-09-26 12:14:43,403 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking pool 2 for storage, totalSize: 2997945892864, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  383. 2014-09-26 12:14:43,407 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking pool: 2 for volume allocation [Vol[1|vm=1|ROOT]], maxSize : 2997945892864, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  384. 2014-09-26 12:14:43,408 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) Checking if storage pool is suitable, name: null ,poolId: 3
  385. 2014-09-26 12:14:43,410 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking pool 3 for storage, totalSize: 2997950087168, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  386. 2014-09-26 12:14:43,414 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking pool: 3 for volume allocation [Vol[1|vm=1|ROOT]], maxSize : 2997950087168, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  387. 2014-09-26 12:14:43,414 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-4ca29ebb) ClusterScopeStoragePoolAllocator returning 2 suitable storage pools
  388. 2014-09-26 12:14:43,414 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  389. 2014-09-26 12:14:43,414 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Checking if host: 3 can access any suitable storage pool for volume: ROOT
  390. 2014-09-26 12:14:43,415 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Host: 3 can access pool: 2
  391. 2014-09-26 12:14:43,416 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Found a potential host id: 3 name: Flex-H2.flexhost.local and associated storage pools for this VM
  392. 2014-09-26 12:14:43,416 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(3)-Storage(Volume(1|ROOT-->Pool(2))]
  393. 2014-09-26 12:14:43,417 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Deployment found - P0=VM[ConsoleProxy|v-1-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(3)-Storage(Volume(1|ROOT-->Pool(2))]
  394. 2014-09-26 12:14:43,435 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 3 host id before state transition: null
  395. 2014-09-26 12:14:43,445 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  396. 2014-09-26 12:14:43,445 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) We are allocating VM, increasing the used capacity of this host:3
  397. 2014-09-26 12:14:43,445 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Current Used CPU: 500 , Free CPU:47524 ,Requested CPU: 500
  398. 2014-09-26 12:14:43,445 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) Current Used RAM: 536870912 , Free RAM:66984202240 ,Requested RAM: 1073741824
  399. 2014-09-26 12:14:43,445 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) CPU STATS after allocation: for host: 3, old used: 500, old reserved: 0, actual total: 48024, total with overprovisioning: 48024; new used:1000, reserved:0; requested cpu:500,alloc_from_last:false
  400. 2014-09-26 12:14:43,445 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4ca29ebb) RAM STATS after allocation: for host: 3, old used: 536870912, old reserved: 0, total: 67521073152; new used: 1610612736, reserved: 0; requested mem: 1073741824,alloc_from_last:false
  401. 2014-09-26 12:14:43,453 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4ca29ebb) VM is being created in podId: 1
  402. 2014-09-26 12:14:43,465 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-4ca29ebb) Network id=204 is already implemented
  403. 2014-09-26 12:14:43,512 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-4ca29ebb) Asking VirtualRouter to prepare for Nic[1-1-null-205.185.158.240]
  404. 2014-09-26 12:14:43,515 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-4ca29ebb) Asking SecurityGroupProvider to prepare for Nic[1-1-null-205.185.158.240]
  405. 2014-09-26 12:14:43,523 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-4ca29ebb) Network id=201 is already implemented
  406. 2014-09-26 12:14:43,563 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-4ca29ebb) Network id=200 is already implemented
  407. 2014-09-26 12:14:43,624 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-4ca29ebb) Allocated a nic NicProfile[3-1-44dde5f9-0eef-4159-ab4a-09b4e97a85aa-192.168.30.10-null for VM[ConsoleProxy|v-1-VM]
  408. 2014-09-26 12:14:43,635 DEBUG [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-4ca29ebb) Checking if we need to prepare 1 volumes for VM[ConsoleProxy|v-1-VM]
  409. 2014-09-26 12:14:43,651 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-4ca29ebb) template 3 is already in store:6, type:Image
  410. 2014-09-26 12:14:43,656 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-4ca29ebb) template 3 is already in store:2, type:Primary
  411. 2014-09-26 12:14:43,697 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-4ca29ebb) template 3 is already in store:6, type:Image
  412. 2014-09-26 12:14:43,708 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-4ca29ebb) Seq 3-102367932: Sending { Cmd , MgmtId: 345050433729, via: 3(Flex-H2.flexhost.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2","uuid":"7e218d6a-403d-11e4-b04c-0050569b56d4","id":3,"format":"QCOW2","accountId":1,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://Flex-csms/mnt/secondary","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"f7308867-a99b-4e93-8843-81288febf624","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"86cdf4ee-fd52-4ed9-8beb-0979f2548904","id":2,"poolType":"CLVM","host":"localhost","path":"/vg_s1c0","port":0,"url":"CLVM://localhost//vg_s1c0/?ROLE=Primary&STOREUUID=86cdf4ee-fd52-4ed9-8beb-0979f2548904"}},"name":"ROOT-1","size":0,"volumeId":1,"vmName":"v-1-VM","accountId":1,"format":"QCOW2","id":1,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}] }
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!