API tools faq
paste
juesor

System VM's - Starting

juesor
Sep 30th, 2014
263
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 113.09 KB | None | 0 0
raw download clone embed print report
  1. 2014-09-30 08:44:55,607 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-9c705eb1) VmStatsCollector is running...
  2. 2014-09-30 08:45:01,833 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) Ping from 2
  3. 2014-09-30 08:45:02,010 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-80079e5e) Found 0 routers to update status.
  4. 2014-09-30 08:45:02,012 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-80079e5e) Found 0 networks to update RvR status.
  5. 2014-09-30 08:45:05,071 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-f7fc2c3b) HostStatsCollector is running...
  6. 2014-09-30 08:45:05,699 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-f7fc2c3b) Seq 1-792595058: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  7. 2014-09-30 08:45:06,456 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-f7fc2c3b) Seq 2-2053047764: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  8. 2014-09-30 08:45:16,128 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-8c7c6802) StorageCollector is running...
  9. 2014-09-30 08:45:16,133 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-8c7c6802) There is no secondary storage VM for secondary storage host Secondary1
  10. 2014-09-30 08:45:16,435 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8c7c6802) Seq 2-2053047765: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  11. 2014-09-30 08:45:32,010 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-912f9435) Found 0 routers to update status.
  12. 2014-09-30 08:45:32,012 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-912f9435) Found 0 networks to update RvR status.
  13. 2014-09-30 08:45:37,028 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-1b32fe6f) Resetting hosts suitable for reconnect
  14. 2014-09-30 08:45:37,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-1b32fe6f) Completed resetting hosts suitable for reconnect
  15. 2014-09-30 08:45:37,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-1b32fe6f) Acquiring hosts for clusters already owned by this management server
  16. 2014-09-30 08:45:37,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-1b32fe6f) Completed acquiring hosts for clusters already owned by this management server
  17. 2014-09-30 08:45:37,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-1b32fe6f) Acquiring hosts for clusters not owned by any management server
  18. 2014-09-30 08:45:37,031 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-1b32fe6f) Completed acquiring hosts for clusters not owned by any management server
  19. 2014-09-30 08:45:39,975 INFO [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-d9c43899) Found the following agents behind on ping: [1]
  20. 2014-09-30 08:45:39,977 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-d9c43899) Ping timeout for host 1, do invstigation
  21. 2014-09-30 08:45:39,979 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-ef428246) Investigating why host 1 has disconnected with event PingTimeout
  22. 2014-09-30 08:45:39,980 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-ef428246) checking if agent (1) is alive
  23. 2014-09-30 08:45:39,984 DEBUG [c.c.a.t.Request] (AgentTaskPool-7:ctx-ef428246) Seq 1-792595059: Sending { Cmd , MgmtId: 345050433729, via: 1(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckHealthCommand":{"wait":50}}] }
  24. 2014-09-30 08:45:39,987 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:null) Seq 1-792595059: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckHealthAnswer":{"result":true,"details":"resource is alive","wait":0}}] }
  25. 2014-09-30 08:45:39,987 DEBUG [c.c.a.t.Request] (AgentTaskPool-7:ctx-ef428246) Seq 1-792595059: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  26. 2014-09-30 08:45:39,987 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-ef428246) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is alive
  27. 2014-09-30 08:45:39,987 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-ef428246) agent (1) responded to checkHeathCommand, reporting that agent is Up
  28. 2014-09-30 08:45:39,987 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-ef428246) The state determined is Up
  29. 2014-09-30 08:45:39,988 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-ef428246) Agent is determined to be up and running
  30. 2014-09-30 08:45:39,988 DEBUG [c.c.h.Status] (AgentTaskPool-7:ctx-ef428246) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 1, name = Flex-H1.flexhost.local]
  31. 2014-09-30 08:45:39,998 DEBUG [c.c.h.Status] (AgentTaskPool-7:ctx-ef428246) Agent status update: [id = 1; name = Flex-H1.flexhost.local; old status = Up; event = Ping; new status = Up; old update count = 999; new update count = 1000]
  32. 2014-09-30 08:45:55,614 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-b1bc032e) VmStatsCollector is running...
  33. 2014-09-30 08:46:01,822 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) Ping from 2
  34. 2014-09-30 08:46:02,009 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-7a0d8611) Found 0 routers to update status.
  35. 2014-09-30 08:46:02,011 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-7a0d8611) Found 0 networks to update RvR status.
  36. 2014-09-30 08:46:06,457 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-fc3ede97) HostStatsCollector is running...
  37. 2014-09-30 08:46:07,076 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-fc3ede97) Seq 1-792595060: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  38. 2014-09-30 08:46:07,710 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-fc3ede97) Seq 2-2053047766: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  39. 2014-09-30 08:46:16,435 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-354b6b7d) StorageCollector is running...
  40. 2014-09-30 08:46:16,441 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-354b6b7d) There is no secondary storage VM for secondary storage host Secondary1
  41. 2014-09-30 08:46:16,572 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-354b6b7d) Seq 2-2053047767: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  42. 2014-09-30 08:46:21,289 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) Seq 1-792595055: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error Child process (/bin/mount Flex-CSMS.flexhost.local:/mnt/secondary/template/tmpl/1/3 /mnt/7a4f0f30-05b1-3f2f-9acf-b5827be546c0) unexpected exit status 32: mount.nfs: Connection timed out\n","wait":0}}] }
  43. 2014-09-30 08:46:21,289 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-4bf88d17) Seq 1-792595055: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  44. 2014-09-30 08:46:21,290 ERROR [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4bf88d17) Failed to start instance VM[ConsoleProxy|v-33-VM]
  45. java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  46. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  47. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  48. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  49. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  50. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  51. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  52. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  53. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  54. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  55. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  56. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  57. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  58. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  59. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  60. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  61. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  62. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  63. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  64. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  65. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  66. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  67. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  68. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  69. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  70. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  71. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  72. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  73. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  74. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  75. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  76. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  77. at java.lang.Thread.run(Thread.java:745)
  78. Caused by: java.lang.reflect.InvocationTargetException
  79. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  80. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  81. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  82. at java.lang.reflect.Method.invoke(Method.java:606)
  83. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  84. ... 31 more
  85. Caused by: java.lang.NullPointerException
  86. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  87. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  88. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  89. ... 35 more
  90. 2014-09-30 08:46:21,541 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4bf88d17) Cleaning up resources for the vm VM[ConsoleProxy|v-33-VM] in Starting state
  91. 2014-09-30 08:46:21,545 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-4bf88d17) Seq 2-2053047768: Sending { Cmd , MgmtId: 345050433729, via: 2(Flex-H2.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-33-VM","wait":0}}] }
  92. 2014-09-30 08:46:21,760 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) Seq 2-2053047768: Processing: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
  93. 2014-09-30 08:46:21,760 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-4bf88d17) Seq 2-2053047768: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { StopAnswer } }
  94. 2014-09-30 08:46:21,956 DEBUG [c.c.n.g.ControlNetworkGuru] (consoleproxy-1:ctx-4bf88d17) Released nic: NicProfile[2-33-null-null-null
  95. 2014-09-30 08:46:22,108 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (consoleproxy-1:ctx-4bf88d17) Releasing ip address for reservationId=58fe2fb1-5b36-4377-a58a-0eb9223338cc, instance=3
  96. 2014-09-30 08:46:22,109 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-4bf88d17) Released nic: NicProfile[3-33-null-null-null
  97. 2014-09-30 08:46:22,236 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4bf88d17) Successfully released network resources for the vm VM[ConsoleProxy|v-33-VM]
  98. 2014-09-30 08:46:22,236 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-4bf88d17) Successfully cleanued up resources for the vm VM[ConsoleProxy|v-33-VM] in Starting state
  99. 2014-09-30 08:46:22,289 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4bf88d17) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2
  100. 2014-09-30 08:46:22,298 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4bf88d17) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  101. 2014-09-30 08:46:22,298 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4bf88d17) Hosts's actual total RAM: 67521073152 and RAM after applying overprovisioning: 67521073152
  102. 2014-09-30 08:46:22,298 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4bf88d17) release cpu from host: 2, old used: 500,reserved: 0, actual total: 48024, total with overprovisioning: 48024; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  103. 2014-09-30 08:46:22,298 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-4bf88d17) release mem from host: 2, old used: 1073741824,reserved: 0, total: 67521073152; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  104. 2014-09-30 08:46:22,324 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4bf88d17) Exception while trying to start console proxy
  105. com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Unable to start instance due to InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  106. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1073)
  107. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  108. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  109. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  110. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  111. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  112. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  113. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  114. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  115. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  116. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  117. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  118. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  119. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  120. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  121. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  122. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  123. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  124. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  125. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  126. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  127. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  128. at java.lang.Thread.run(Thread.java:745)
  129. Caused by: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  130. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  131. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  132. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  133. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  134. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  135. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  136. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  137. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  138. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  139. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  140. ... 22 more
  141. Caused by: java.lang.reflect.InvocationTargetException
  142. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  143. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  144. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  145. at java.lang.reflect.Method.invoke(Method.java:606)
  146. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  147. ... 31 more
  148. Caused by: java.lang.NullPointerException
  149. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  150. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  151. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  152. ... 35 more
  153. 2014-09-30 08:46:22,353 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Zone 1 is ready to launch console proxy
  154. 2014-09-30 08:46:22,353 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Expand console proxy standby capacity for zone Rushford HEND
  155. 2014-09-30 08:46:22,357 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Found a stopped console proxy, bring it up to running pool. proxy vm id : 33
  156. 2014-09-30 08:46:22,369 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
  157. 2014-09-30 08:46:22,369 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Successfully transitioned to start state for VM[ConsoleProxy|v-33-VM] reservation id = 607571f8-f8be-4bbd-b6cd-6cb7abd9cef8
  158. 2014-09-30 08:46:22,377 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Trying to deploy VM, vm has dcId: 1 and podId: 1
  159. 2014-09-30 08:46:22,377 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Deploy avoids pods: null, clusters: null, hosts: null
  160. 2014-09-30 08:46:22,383 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Deploy avoids pods: null, clusters: null, hosts: null
  161. 2014-09-30 08:46:22,384 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@2a0b20fb
  162. 2014-09-30 08:46:22,384 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 1073741824
  163. 2014-09-30 08:46:22,384 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Is ROOT volume READY (pool already allocated)?: No
  164. 2014-09-30 08:46:22,385 DEBUG [c.c.d.FirstFitPlanner] (consoleproxy-1:ctx-ca3aa2f5) Searching resources only under specified Pod: 1
  165. 2014-09-30 08:46:22,385 DEBUG [c.c.d.FirstFitPlanner] (consoleproxy-1:ctx-ca3aa2f5) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  166. 2014-09-30 08:46:22,396 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Checking resources in Cluster: 1 under Pod: 1
  167. 2014-09-30 08:46:22,396 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  168. 2014-09-30 08:46:22,401 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
  169. 2014-09-30 08:46:22,406 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
  170. 2014-09-30 08:46:22,406 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=1024
  171. 2014-09-30 08:46:22,416 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Host: 2 has cpu capability (cpu:24, speed:2001) to support requested CPU: 1 and requested speed: 500
  172. 2014-09-30 08:46:22,416 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  173. 2014-09-30 08:46:22,420 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  174. 2014-09-30 08:46:22,420 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Free CPU: 48024 , Requested CPU: 500
  175. 2014-09-30 08:46:22,420 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Free RAM: 67521073152 , Requested RAM: 1073741824
  176. 2014-09-30 08:46:22,420 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Host has enough CPU and RAM available
  177. 2014-09-30 08:46:22,420 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 48024, total with overprovisioning: 48024; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  178. 2014-09-30 08:46:22,420 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 2, used: 0, reserved: 0, total: 67521073152; requested mem: 1073741824,alloc_from_last_host?:false ,considerReservedCapacity?: true
  179. 2014-09-30 08:46:22,420 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Found a suitable host, adding to list: 2
  180. 2014-09-30 08:46:22,430 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Host: 1 has cpu capability (cpu:24, speed:2000) to support requested CPU: 1 and requested speed: 500
  181. 2014-09-30 08:46:22,430 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  182. 2014-09-30 08:46:22,433 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  183. 2014-09-30 08:46:22,433 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Free CPU: 47500 , Requested CPU: 500
  184. 2014-09-30 08:46:22,433 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Free RAM: 66984181760 , Requested RAM: 1073741824
  185. 2014-09-30 08:46:22,433 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Host has enough CPU and RAM available
  186. 2014-09-30 08:46:22,433 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 1, used: 500, reserved: 0, actual total: 48000, total with overprovisioning: 48000; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  187. 2014-09-30 08:46:22,433 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 1, used: 536870912, reserved: 0, total: 67521052672; requested mem: 1073741824,alloc_from_last_host?:false ,considerReservedCapacity?: true
  188. 2014-09-30 08:46:22,434 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Found a suitable host, adding to list: 1
  189. 2014-09-30 08:46:22,434 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-ca3aa2f5 FirstFitRoutingAllocator) Host Allocator returning 2 suitable hosts
  190. 2014-09-30 08:46:22,436 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Checking suitable pools for volume (Id, Type): (1,ROOT)
  191. 2014-09-30 08:46:22,436 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) We need to allocate new storagepool for this volume
  192. 2014-09-30 08:46:22,436 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Calling StoragePoolAllocators to find suitable pools
  193. 2014-09-30 08:46:22,439 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  194. 2014-09-30 08:46:22,439 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) ClusterScopeStoragePoolAllocator looking for storage pool
  195. 2014-09-30 08:46:22,439 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) Looking for pools in dc: 1 pod:1 cluster:1
  196. 2014-09-30 08:46:22,441 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) Found pools matching tags: [Pool[1|CLVM]]
  197. 2014-09-30 08:46:22,443 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) Removing pool Pool[1|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  198. 2014-09-30 08:46:22,445 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) Checking if storage pool is suitable, name: null ,poolId: 1
  199. 2014-09-30 08:46:22,450 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Checking pool 1 for storage, totalSize: 2997945892864, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  200. 2014-09-30 08:46:22,455 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Checking pool: 1 for volume allocation [Vol[1|vm=33|ROOT]], maxSize : 2997945892864, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  201. 2014-09-30 08:46:22,455 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-ca3aa2f5) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  202. 2014-09-30 08:46:22,455 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  203. 2014-09-30 08:46:22,455 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Checking if host: 2 can access any suitable storage pool for volume: ROOT
  204. 2014-09-30 08:46:22,456 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Host: 2 can access pool: 1
  205. 2014-09-30 08:46:22,458 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Found a potential host id: 2 name: Flex-H2.flexhost.local and associated storage pools for this VM
  206. 2014-09-30 08:46:22,459 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1|ROOT-->Pool(1))]
  207. 2014-09-30 08:46:22,459 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Deployment found - P0=VM[ConsoleProxy|v-33-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1|ROOT-->Pool(1))]
  208. 2014-09-30 08:46:22,479 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 2 host id before state transition: null
  209. 2014-09-30 08:46:22,489 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  210. 2014-09-30 08:46:22,489 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) We are allocating VM, increasing the used capacity of this host:2
  211. 2014-09-30 08:46:22,489 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Current Used CPU: 0 , Free CPU:48024 ,Requested CPU: 500
  212. 2014-09-30 08:46:22,489 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Current Used RAM: 0 , Free RAM:67521073152 ,Requested RAM: 1073741824
  213. 2014-09-30 08:46:22,489 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) CPU STATS after allocation: for host: 2, old used: 0, old reserved: 0, actual total: 48024, total with overprovisioning: 48024; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
  214. 2014-09-30 08:46:22,489 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) RAM STATS after allocation: for host: 2, old used: 0, old reserved: 0, total: 67521073152; new used: 1073741824, reserved: 0; requested mem: 1073741824,alloc_from_last:false
  215. 2014-09-30 08:46:22,499 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) VM is being created in podId: 1
  216. 2014-09-30 08:46:22,512 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-ca3aa2f5) Network id=204 is already implemented
  217. 2014-09-30 08:46:22,547 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-ca3aa2f5) Asking SecurityGroupProvider to prepare for Nic[1-33-null-205.185.158.240]
  218. 2014-09-30 08:46:22,551 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-ca3aa2f5) Asking VirtualRouter to prepare for Nic[1-33-null-205.185.158.240]
  219. 2014-09-30 08:46:22,560 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-ca3aa2f5) Network id=201 is already implemented
  220. 2014-09-30 08:46:22,602 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-ca3aa2f5) Network id=200 is already implemented
  221. 2014-09-30 08:46:22,634 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-ca3aa2f5) Allocated a nic NicProfile[3-33-607571f8-f8be-4bbd-b6cd-6cb7abd9cef8-192.168.30.7-null for VM[ConsoleProxy|v-33-VM]
  222. 2014-09-30 08:46:22,643 DEBUG [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-ca3aa2f5) Checking if we need to prepare 1 volumes for VM[ConsoleProxy|v-33-VM]
  223. 2014-09-30 08:46:22,659 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-ca3aa2f5) template 3 is already in store:2, type:Image
  224. 2014-09-30 08:46:22,664 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-ca3aa2f5) template 3 is already in store:1, type:Primary
  225. 2014-09-30 08:46:22,703 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-ca3aa2f5) template 3 is already in store:2, type:Image
  226. 2014-09-30 08:46:22,714 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-ca3aa2f5) Seq 2-2053047769: Sending { Cmd , MgmtId: 345050433729, via: 2(Flex-H2.flexhost.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2","uuid":"a3383c10-47eb-11e4-8415-0050569b56d4","id":3,"format":"QCOW2","accountId":1,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://Flex-CSMS.flexhost.local/mnt/secondary","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"3b6a23be-02d2-448a-b070-de030c8a5317","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"83aa613b-a469-4714-b24d-66a728752471","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_s1c0","port":0,"url":"CLVM://localhost//vg_s1c0/?ROLE=Primary&STOREUUID=83aa613b-a469-4714-b24d-66a728752471"}},"name":"ROOT-33","size":0,"volumeId":1,"vmName":"v-33-VM","accountId":1,"format":"QCOW2","id":1,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  227. 2014-09-30 08:46:32,009 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-18a2edad) Found 0 routers to update status.
  228. 2014-09-30 08:46:32,011 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-18a2edad) Found 0 networks to update RvR status.
  229. 2014-09-30 08:46:39,980 INFO [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-77fd1610) Found the following agents behind on ping: [1]
  230. 2014-09-30 08:46:39,983 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-77fd1610) Ping timeout for host 1, do invstigation
  231. 2014-09-30 08:46:39,985 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-7b9b2e3a) Investigating why host 1 has disconnected with event PingTimeout
  232. 2014-09-30 08:46:39,985 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-7b9b2e3a) checking if agent (1) is alive
  233. 2014-09-30 08:46:39,989 DEBUG [c.c.a.t.Request] (AgentTaskPool-8:ctx-7b9b2e3a) Seq 1-792595061: Sending { Cmd , MgmtId: 345050433729, via: 1(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckHealthCommand":{"wait":50}}] }
  234. 2014-09-30 08:46:39,992 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 1-792595061: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckHealthAnswer":{"result":true,"details":"resource is alive","wait":0}}] }
  235. 2014-09-30 08:46:39,992 DEBUG [c.c.a.t.Request] (AgentTaskPool-8:ctx-7b9b2e3a) Seq 1-792595061: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  236. 2014-09-30 08:46:39,992 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-7b9b2e3a) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is alive
  237. 2014-09-30 08:46:39,993 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-7b9b2e3a) agent (1) responded to checkHeathCommand, reporting that agent is Up
  238. 2014-09-30 08:46:39,993 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-7b9b2e3a) The state determined is Up
  239. 2014-09-30 08:46:39,993 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-7b9b2e3a) Agent is determined to be up and running
  240. 2014-09-30 08:46:39,993 DEBUG [c.c.h.Status] (AgentTaskPool-8:ctx-7b9b2e3a) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 1, name = Flex-H1.flexhost.local]
  241. 2014-09-30 08:46:40,001 DEBUG [c.c.h.Status] (AgentTaskPool-8:ctx-7b9b2e3a) Agent status update: [id = 1; name = Flex-H1.flexhost.local; old status = Up; event = Ping; new status = Up; old update count = 1000; new update count = 1001]
  242. 2014-09-30 08:46:55,621 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-b20765b3) VmStatsCollector is running...
  243. 2014-09-30 08:47:01,835 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 2
  244. 2014-09-30 08:47:02,010 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-973147d1) Found 0 routers to update status.
  245. 2014-09-30 08:47:02,012 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-973147d1) Found 0 networks to update RvR status.
  246. 2014-09-30 08:47:07,028 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5cacc53a) Resetting hosts suitable for reconnect
  247. 2014-09-30 08:47:07,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5cacc53a) Completed resetting hosts suitable for reconnect
  248. 2014-09-30 08:47:07,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5cacc53a) Acquiring hosts for clusters already owned by this management server
  249. 2014-09-30 08:47:07,031 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5cacc53a) Completed acquiring hosts for clusters already owned by this management server
  250. 2014-09-30 08:47:07,031 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5cacc53a) Acquiring hosts for clusters not owned by any management server
  251. 2014-09-30 08:47:07,031 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5cacc53a) Completed acquiring hosts for clusters not owned by any management server
  252. 2014-09-30 08:47:07,711 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-5f562b5f) HostStatsCollector is running...
  253. 2014-09-30 08:47:08,332 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-5f562b5f) Seq 1-792595062: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  254. 2014-09-30 08:47:08,953 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-5f562b5f) Seq 2-2053047770: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  255. 2014-09-30 08:47:16,573 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-33257f92) StorageCollector is running...
  256. 2014-09-30 08:47:16,579 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-33257f92) There is no secondary storage VM for secondary storage host Secondary1
  257. 2014-09-30 08:47:32,009 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4fafa8a6) Found 0 routers to update status.
  258. 2014-09-30 08:47:32,011 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4fafa8a6) Found 0 networks to update RvR status.
  259. 2014-09-30 08:47:39,986 INFO [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-09821f86) Found the following agents behind on ping: [1]
  260. 2014-09-30 08:47:39,989 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-09821f86) Ping timeout for host 1, do invstigation
  261. 2014-09-30 08:47:39,991 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-9:ctx-f63aeee4) Investigating why host 1 has disconnected with event PingTimeout
  262. 2014-09-30 08:47:39,992 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-9:ctx-f63aeee4) checking if agent (1) is alive
  263. 2014-09-30 08:47:39,995 DEBUG [c.c.a.t.Request] (AgentTaskPool-9:ctx-f63aeee4) Seq 1-792595063: Sending { Cmd , MgmtId: 345050433729, via: 1(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckHealthCommand":{"wait":50}}] }
  264. 2014-09-30 08:47:39,999 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:null) Seq 1-792595063: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckHealthAnswer":{"result":true,"details":"resource is alive","wait":0}}] }
  265. 2014-09-30 08:47:39,999 DEBUG [c.c.a.t.Request] (AgentTaskPool-9:ctx-f63aeee4) Seq 1-792595063: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  266. 2014-09-30 08:47:39,999 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-9:ctx-f63aeee4) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is alive
  267. 2014-09-30 08:47:39,999 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-9:ctx-f63aeee4) agent (1) responded to checkHeathCommand, reporting that agent is Up
  268. 2014-09-30 08:47:39,999 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-9:ctx-f63aeee4) The state determined is Up
  269. 2014-09-30 08:47:39,999 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-9:ctx-f63aeee4) Agent is determined to be up and running
  270. 2014-09-30 08:47:39,999 DEBUG [c.c.h.Status] (AgentTaskPool-9:ctx-f63aeee4) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 1, name = Flex-H1.flexhost.local]
  271. 2014-09-30 08:47:40,009 DEBUG [c.c.h.Status] (AgentTaskPool-9:ctx-f63aeee4) Agent status update: [id = 1; name = Flex-H1.flexhost.local; old status = Up; event = Ping; new status = Up; old update count = 1001; new update count = 1002]
  272. 2014-09-30 08:47:55,628 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-e0165910) VmStatsCollector is running...
  273. 2014-09-30 08:48:01,853 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-8:null) Ping from 2
  274. 2014-09-30 08:48:02,009 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6f75f70c) Found 0 routers to update status.
  275. 2014-09-30 08:48:02,011 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6f75f70c) Found 0 networks to update RvR status.
  276. 2014-09-30 08:48:08,954 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-34ed0e9d) HostStatsCollector is running...
  277. 2014-09-30 08:48:09,581 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-34ed0e9d) Seq 1-792595064: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  278. 2014-09-30 08:48:10,223 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-34ed0e9d) Seq 2-2053047772: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  279. 2014-09-30 08:48:26,436 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:null) Seq 1-792595056: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error Child process (/bin/mount Flex-CSMS.flexhost.local:/mnt/secondary/template/tmpl/1/3 /mnt/7a4f0f30-05b1-3f2f-9acf-b5827be546c0) unexpected exit status 32: mount.nfs: Connection timed out\n","wait":0}}] }
  280. 2014-09-30 08:48:26,437 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-9fb0b338) Seq 1-792595056: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  281. 2014-09-30 08:48:26,438 ERROR [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Failed to start instance VM[SecondaryStorageVm|s-35-VM]
  282. java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  283. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  284. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  285. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  286. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  287. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  288. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  289. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  290. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  291. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  292. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  293. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  294. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  295. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  296. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  297. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  298. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  299. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  300. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  301. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  302. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  303. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  304. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  305. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  306. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  307. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  308. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  309. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  310. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  311. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  312. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  313. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  314. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  315. at java.lang.Thread.run(Thread.java:745)
  316. Caused by: java.lang.reflect.InvocationTargetException
  317. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  318. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  319. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  320. at java.lang.reflect.Method.invoke(Method.java:606)
  321. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  322. ... 32 more
  323. Caused by: java.lang.NullPointerException
  324. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  325. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  326. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  327. ... 36 more
  328. 2014-09-30 08:48:26,446 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Cleaning up resources for the vm VM[SecondaryStorageVm|s-35-VM] in Starting state
  329. 2014-09-30 08:48:26,450 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-9fb0b338) Seq 1-792595065: Sending { Cmd , MgmtId: 345050433729, via: 1(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-35-VM","wait":0}}] }
  330. 2014-09-30 08:48:26,663 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) Seq 1-792595065: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
  331. 2014-09-30 08:48:26,663 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-9fb0b338) Seq 1-792595065: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { StopAnswer } }
  332. 2014-09-30 08:48:26,680 DEBUG [c.c.n.g.ControlNetworkGuru] (secstorage-1:ctx-9fb0b338) Released nic: NicProfile[9-35-null-null-null
  333. 2014-09-30 08:48:26,695 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-9fb0b338) Releasing ip address for reservationId=e8e72dd8-b296-46db-853a-ec397eb11d17, instance=10
  334. 2014-09-30 08:48:26,695 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-9fb0b338) Released nic: NicProfile[10-35-null-null-null
  335. 2014-09-30 08:48:26,712 DEBUG [c.c.n.g.StorageNetworkGuru] (secstorage-1:ctx-9fb0b338) Release an storage ip 10.83.0.210
  336. 2014-09-30 08:48:26,719 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Successfully released network resources for the vm VM[SecondaryStorageVm|s-35-VM]
  337. 2014-09-30 08:48:26,719 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-35-VM] in Starting state
  338. 2014-09-30 08:48:26,736 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9fb0b338) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 1
  339. 2014-09-30 08:48:26,744 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9fb0b338) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  340. 2014-09-30 08:48:26,745 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9fb0b338) Hosts's actual total RAM: 67521052672 and RAM after applying overprovisioning: 67521052672
  341. 2014-09-30 08:48:26,745 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9fb0b338) release cpu from host: 1, old used: 500,reserved: 0, actual total: 48000, total with overprovisioning: 48000; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  342. 2014-09-30 08:48:26,745 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9fb0b338) release mem from host: 1, old used: 536870912,reserved: 0, total: 67521052672; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  343. 2014-09-30 08:48:26,752 WARN [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-9fb0b338) Exception while trying to start secondary storage vm
  344. com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  345. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1073)
  346. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  347. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  348. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  349. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  350. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  351. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  352. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  353. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  354. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  355. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  356. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  357. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  358. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  359. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  360. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  361. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  362. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  363. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  364. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  365. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  366. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  367. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  368. at java.lang.Thread.run(Thread.java:745)
  369. Caused by: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  370. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  371. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  372. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  373. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  374. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  375. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  376. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  377. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  378. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  379. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  380. ... 23 more
  381. Caused by: java.lang.reflect.InvocationTargetException
  382. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  383. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  384. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  385. at java.lang.reflect.Method.invoke(Method.java:606)
  386. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  387. ... 32 more
  388. Caused by: java.lang.NullPointerException
  389. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  390. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  391. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  392. ... 36 more
  393. 2014-09-30 08:48:26,754 INFO [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-9fb0b338) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 35, will recycle it and start a new one
  394. 2014-09-30 08:48:26,761 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) VM is already stopped: VM[SecondaryStorageVm|s-35-VM]
  395. 2014-09-30 08:48:26,773 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9fb0b338) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null
  396. 2014-09-30 08:48:26,774 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Destroying vm VM[SecondaryStorageVm|s-35-VM]
  397. 2014-09-30 08:48:26,774 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Cleaning up NICS
  398. 2014-09-30 08:48:26,774 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Cleaning network for vm: 35
  399. 2014-09-30 08:48:26,797 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Asking SecurityGroupProvider to release Nic[8-35-null-205.185.158.241]
  400. 2014-09-30 08:48:26,802 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Asking VirtualRouter to release Nic[8-35-null-205.185.158.241]
  401. 2014-09-30 08:48:26,802 DEBUG [c.c.n.g.DirectNetworkGuru] (secstorage-1:ctx-9fb0b338) Deallocate network: networkId: 204, ip: 205.185.158.241
  402. 2014-09-30 08:48:26,809 DEBUG [c.c.n.g.DirectNetworkGuru] (secstorage-1:ctx-9fb0b338) remove nic 8 secondary ip
  403. 2014-09-30 08:48:26,821 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Removed nic id=8
  404. 2014-09-30 08:48:26,823 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Revoving nic secondary ip entry ...
  405. 2014-09-30 08:48:26,840 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Removed nic id=9
  406. 2014-09-30 08:48:26,842 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Revoving nic secondary ip entry ...
  407. 2014-09-30 08:48:26,867 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Removed nic id=10
  408. 2014-09-30 08:48:26,868 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Revoving nic secondary ip entry ...
  409. 2014-09-30 08:48:26,892 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Removed nic id=11
  410. 2014-09-30 08:48:26,894 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9fb0b338) Revoving nic secondary ip entry ...
  411. 2014-09-30 08:48:26,894 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
  412. 2014-09-30 08:48:26,894 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-9fb0b338) Cleaning storage for vm: 35
  413. 2014-09-30 08:48:26,917 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-9fb0b338) Marking volume that was never created as destroyed: Vol[3|vm=35|ROOT]
  414. 2014-09-30 08:48:26,937 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Expunged VM[SecondaryStorageVm|s-35-VM]
  415. 2014-09-30 08:48:26,948 INFO [c.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-9fb0b338) Primary secondary storage is not even started, wait until next turn
  416. 2014-09-30 08:48:26,969 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-8813b220) Zone 1 is ready to launch secondary storage VM
  417. 2014-09-30 08:48:26,976 INFO [c.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-8813b220) No running secondary storage vms found in datacenter id=1, starting one
  418. 2014-09-30 08:48:26,982 INFO [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-8813b220) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
  419. 2014-09-30 08:48:26,985 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-8813b220) Assign secondary storage vm from a newly started instance for request from data center : 1
  420. 2014-09-30 08:48:27,006 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Found existing network configuration for offering [Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]: Ntwk[204|Guest|6]
  421. 2014-09-30 08:48:27,006 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Releasing lock for Acct[d665d34a-47eb-11e4-8415-0050569b56d4-system]
  422. 2014-09-30 08:48:27,011 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[201|Control|3]
  423. 2014-09-30 08:48:27,011 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Releasing lock for Acct[d665d34a-47eb-11e4-8415-0050569b56d4-system]
  424. 2014-09-30 08:48:27,015 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[200|Management|2]
  425. 2014-09-30 08:48:27,015 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Releasing lock for Acct[d665d34a-47eb-11e4-8415-0050569b56d4-system]
  426. 2014-09-30 08:48:27,020 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Found existing network configuration for offering [Network Offering [4-Storage-System-Storage-Network]: Ntwk[202|Storage|4]
  427. 2014-09-30 08:48:27,020 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Releasing lock for Acct[d665d34a-47eb-11e4-8415-0050569b56d4-system]
  428. 2014-09-30 08:48:27,043 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Allocating entries for VM: VM[SecondaryStorageVm|s-36-VM]
  429. 2014-09-30 08:48:27,050 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Allocating nics for VM[SecondaryStorageVm|s-36-VM]
  430. 2014-09-30 08:48:27,051 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Allocating nic for vm VM[SecondaryStorageVm|s-36-VM] in network Ntwk[204|Guest|6] with requested profile NicProfile[0-0-null-null-null
  431. 2014-09-30 08:48:27,065 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Allocating nic for vm VM[SecondaryStorageVm|s-36-VM] in network Ntwk[201|Control|3] with requested profile null
  432. 2014-09-30 08:48:27,070 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Allocating nic for vm VM[SecondaryStorageVm|s-36-VM] in network Ntwk[200|Management|2] with requested profile null
  433. 2014-09-30 08:48:27,080 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Allocating nic for vm VM[SecondaryStorageVm|s-36-VM] in network Ntwk[202|Storage|4] with requested profile null
  434. 2014-09-30 08:48:27,091 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Allocating disks for VM[SecondaryStorageVm|s-36-VM]
  435. 2014-09-30 08:48:27,100 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Allocation completed for VM: VM[SecondaryStorageVm|s-36-VM]
  436. 2014-09-30 08:48:27,105 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-8813b220) received secondary storage vm alert
  437. 2014-09-30 08:48:27,105 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-8813b220) New secondary storage vm created, zone: Rushford HEND, secStorageVm: s-36-VM, public IP: null, private IP: null
  438. 2014-09-30 08:48:27,121 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
  439. 2014-09-30 08:48:27,121 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Successfully transitioned to start state for VM[SecondaryStorageVm|s-36-VM] reservation id = ec082ec8-d9d8-431b-9546-3138d90c89a6
  440. 2014-09-30 08:48:27,130 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Trying to deploy VM, vm has dcId: 1 and podId: null
  441. 2014-09-30 08:48:27,130 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Deploy avoids pods: null, clusters: null, hosts: null
  442. 2014-09-30 08:48:27,135 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Deploy avoids pods: null, clusters: null, hosts: null
  443. 2014-09-30 08:48:27,137 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@2a0b20fb
  444. 2014-09-30 08:48:27,137 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 536870912
  445. 2014-09-30 08:48:27,137 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Is ROOT volume READY (pool already allocated)?: No
  446. 2014-09-30 08:48:27,137 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-8813b220) Searching all possible resources under this Zone: 1
  447. 2014-09-30 08:48:27,139 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-8813b220) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
  448. 2014-09-30 08:48:27,150 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Checking resources in Cluster: 1 under Pod: 1
  449. 2014-09-30 08:48:27,151 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  450. 2014-09-30 08:48:27,155 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
  451. 2014-09-30 08:48:27,160 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
  452. 2014-09-30 08:48:27,160 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
  453. 2014-09-30 08:48:27,173 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Host: 2 has cpu capability (cpu:24, speed:2001) to support requested CPU: 1 and requested speed: 500
  454. 2014-09-30 08:48:27,173 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  455. 2014-09-30 08:48:27,176 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  456. 2014-09-30 08:48:27,176 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Free CPU: 47524 , Requested CPU: 500
  457. 2014-09-30 08:48:27,176 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Free RAM: 66447331328 , Requested RAM: 536870912
  458. 2014-09-30 08:48:27,176 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Host has enough CPU and RAM available
  459. 2014-09-30 08:48:27,176 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 48024, total with overprovisioning: 48024; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  460. 2014-09-30 08:48:27,176 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 2, used: 1073741824, reserved: 0, total: 67521073152; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  461. 2014-09-30 08:48:27,176 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Found a suitable host, adding to list: 2
  462. 2014-09-30 08:48:27,185 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Host: 1 has cpu capability (cpu:24, speed:2000) to support requested CPU: 1 and requested speed: 500
  463. 2014-09-30 08:48:27,186 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  464. 2014-09-30 08:48:27,189 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  465. 2014-09-30 08:48:27,189 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Free CPU: 48000 , Requested CPU: 500
  466. 2014-09-30 08:48:27,189 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Free RAM: 67521052672 , Requested RAM: 536870912
  467. 2014-09-30 08:48:27,189 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Host has enough CPU and RAM available
  468. 2014-09-30 08:48:27,189 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 48000, total with overprovisioning: 48000; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  469. 2014-09-30 08:48:27,189 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 1, used: 0, reserved: 0, total: 67521052672; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  470. 2014-09-30 08:48:27,189 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Found a suitable host, adding to list: 1
  471. 2014-09-30 08:48:27,189 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-8813b220 FirstFitRoutingAllocator) Host Allocator returning 2 suitable hosts
  472. 2014-09-30 08:48:27,191 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Checking suitable pools for volume (Id, Type): (4,ROOT)
  473. 2014-09-30 08:48:27,191 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) We need to allocate new storagepool for this volume
  474. 2014-09-30 08:48:27,191 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Calling StoragePoolAllocators to find suitable pools
  475. 2014-09-30 08:48:27,194 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (secstorage-1:ctx-8813b220) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  476. 2014-09-30 08:48:27,194 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-8813b220) ClusterScopeStoragePoolAllocator looking for storage pool
  477. 2014-09-30 08:48:27,194 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-8813b220) Looking for pools in dc: 1 pod:1 cluster:1
  478. 2014-09-30 08:48:27,196 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-8813b220) Found pools matching tags: [Pool[1|CLVM]]
  479. 2014-09-30 08:48:27,198 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-8813b220) Removing pool Pool[1|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  480. 2014-09-30 08:48:27,199 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-8813b220) Checking if storage pool is suitable, name: null ,poolId: 1
  481. 2014-09-30 08:48:27,205 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-8813b220) Checking pool 1 for storage, totalSize: 2997945892864, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  482. 2014-09-30 08:48:27,209 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-8813b220) Checking pool: 1 for volume allocation [Vol[4|vm=36|ROOT]], maxSize : 2997945892864, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  483. 2014-09-30 08:48:27,209 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-8813b220) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  484. 2014-09-30 08:48:27,209 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  485. 2014-09-30 08:48:27,209 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Checking if host: 2 can access any suitable storage pool for volume: ROOT
  486. 2014-09-30 08:48:27,210 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Host: 2 can access pool: 1
  487. 2014-09-30 08:48:27,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Found a potential host id: 2 name: Flex-H2.flexhost.local and associated storage pools for this VM
  488. 2014-09-30 08:48:27,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-8813b220) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(4|ROOT-->Pool(1))]
  489. 2014-09-30 08:48:27,212 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) Deployment found - P0=VM[SecondaryStorageVm|s-36-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(4|ROOT-->Pool(1))]
  490. 2014-09-30 08:48:27,234 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 2 host id before state transition: null
  491. 2014-09-30 08:48:27,245 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  492. 2014-09-30 08:48:27,245 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) We are allocating VM, increasing the used capacity of this host:2
  493. 2014-09-30 08:48:27,245 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) Current Used CPU: 500 , Free CPU:47524 ,Requested CPU: 500
  494. 2014-09-30 08:48:27,245 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) Current Used RAM: 1073741824 , Free RAM:66447331328 ,Requested RAM: 536870912
  495. 2014-09-30 08:48:27,245 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) CPU STATS after allocation: for host: 2, old used: 500, old reserved: 0, actual total: 48024, total with overprovisioning: 48024; new used:1000, reserved:0; requested cpu:500,alloc_from_last:false
  496. 2014-09-30 08:48:27,245 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-8813b220) RAM STATS after allocation: for host: 2, old used: 1073741824, old reserved: 0, total: 67521073152; new used: 1610612736, reserved: 0; requested mem: 536870912,alloc_from_last:false
  497. 2014-09-30 08:48:27,252 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8813b220) VM is being created in podId: 1
  498. 2014-09-30 08:48:27,265 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Network id=204 is already implemented
  499. 2014-09-30 08:48:27,293 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Asking SecurityGroupProvider to prepare for Nic[12-36-null-205.185.158.241]
  500. 2014-09-30 08:48:27,297 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Asking VirtualRouter to prepare for Nic[12-36-null-205.185.158.241]
  501. 2014-09-30 08:48:27,305 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Network id=201 is already implemented
  502. 2014-09-30 08:48:27,353 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Network id=200 is already implemented
  503. 2014-09-30 08:48:27,390 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-8813b220) Allocated a nic NicProfile[14-36-ec082ec8-d9d8-431b-9546-3138d90c89a6-192.168.30.12-null for VM[SecondaryStorageVm|s-36-VM]
  504. 2014-09-30 08:48:27,403 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-8813b220) Network id=202 is already implemented
  505. 2014-09-30 08:48:27,441 DEBUG [c.c.n.g.StorageNetworkGuru] (secstorage-1:ctx-8813b220) Allocated a storage nic NicProfile[15-36-ec082ec8-d9d8-431b-9546-3138d90c89a6-10.83.0.22-storage://830 for VM[SecondaryStorageVm|s-36-VM]
  506. 2014-09-30 08:48:27,452 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-8813b220) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-36-VM]
  507. 2014-09-30 08:48:27,470 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-8813b220) template 3 is already in store:2, type:Image
  508. 2014-09-30 08:48:27,475 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-8813b220) template 3 is already in store:1, type:Primary
  509. 2014-09-30 08:48:27,527 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-8813b220) template 3 is already in store:2, type:Image
  510. 2014-09-30 08:48:27,538 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-8813b220) Seq 2-2053047773: Sending { Cmd , MgmtId: 345050433729, via: 2(Flex-H2.flexhost.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2","uuid":"a3383c10-47eb-11e4-8415-0050569b56d4","id":3,"format":"QCOW2","accountId":1,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://Flex-CSMS.flexhost.local/mnt/secondary","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"ce97f9df-7b3b-449d-83d1-ac9e07eae8cb","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"83aa613b-a469-4714-b24d-66a728752471","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_s1c0","port":0,"url":"CLVM://localhost//vg_s1c0/?ROLE=Primary&STOREUUID=83aa613b-a469-4714-b24d-66a728752471"}},"name":"ROOT-36","size":0,"volumeId":4,"vmName":"s-36-VM","accountId":1,"format":"QCOW2","id":4,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  511. 2014-09-30 08:48:27,906 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 2-2053047769: Processing: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error Child process (/bin/mount Flex-CSMS.flexhost.local:/mnt/secondary/template/tmpl/1/3 /mnt/7a4f0f30-05b1-3f2f-9acf-b5827be546c0) unexpected exit status 32: mount.nfs: Connection timed out\n","wait":0}}] }
  512. 2014-09-30 08:48:27,907 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-ca3aa2f5) Seq 2-2053047769: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  513. 2014-09-30 08:48:27,908 ERROR [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Failed to start instance VM[ConsoleProxy|v-33-VM]
  514. java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  515. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  516. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  517. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  518. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  519. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  520. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  521. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  522. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  523. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  524. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  525. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  526. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  527. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  528. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  529. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  530. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  531. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  532. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  533. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  534. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  535. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  536. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  537. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  538. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  539. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  540. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  541. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  542. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  543. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  544. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  545. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  546. at java.lang.Thread.run(Thread.java:745)
  547. Caused by: java.lang.reflect.InvocationTargetException
  548. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  549. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  550. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  551. at java.lang.reflect.Method.invoke(Method.java:606)
  552. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  553. ... 31 more
  554. Caused by: java.lang.NullPointerException
  555. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  556. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  557. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  558. ... 35 more
  559. 2014-09-30 08:48:27,916 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Cleaning up resources for the vm VM[ConsoleProxy|v-33-VM] in Starting state
  560. 2014-09-30 08:48:27,920 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-ca3aa2f5) Seq 2-2053047774: Sending { Cmd , MgmtId: 345050433729, via: 2(Flex-H2.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-33-VM","wait":0}}] }
  561. 2014-09-30 08:48:28,051 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-33257f92) Seq 2-2053047771: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  562. 2014-09-30 08:48:28,129 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) Seq 2-2053047774: Processing: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
  563. 2014-09-30 08:48:28,129 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-ca3aa2f5) Seq 2-2053047774: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { StopAnswer } }
  564. 2014-09-30 08:48:28,159 DEBUG [c.c.n.g.ControlNetworkGuru] (consoleproxy-1:ctx-ca3aa2f5) Released nic: NicProfile[2-33-null-null-null
  565. 2014-09-30 08:48:28,174 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (consoleproxy-1:ctx-ca3aa2f5) Releasing ip address for reservationId=607571f8-f8be-4bbd-b6cd-6cb7abd9cef8, instance=3
  566. 2014-09-30 08:48:28,175 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-ca3aa2f5) Released nic: NicProfile[3-33-null-null-null
  567. 2014-09-30 08:48:28,183 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Successfully released network resources for the vm VM[ConsoleProxy|v-33-VM]
  568. 2014-09-30 08:48:28,183 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Successfully cleanued up resources for the vm VM[ConsoleProxy|v-33-VM] in Starting state
  569. 2014-09-30 08:48:28,248 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2
  570. 2014-09-30 08:48:28,257 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  571. 2014-09-30 08:48:28,257 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Hosts's actual total RAM: 67521073152 and RAM after applying overprovisioning: 67521073152
  572. 2014-09-30 08:48:28,257 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) release cpu from host: 2, old used: 1000,reserved: 0, actual total: 48024, total with overprovisioning: 48024; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
  573. 2014-09-30 08:48:28,257 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) release mem from host: 2, old used: 1610612736,reserved: 0, total: 67521073152; new used: 536870912,reserved:0; movedfromreserved: false,moveToReserveredfalse
  574. 2014-09-30 08:48:28,264 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-ca3aa2f5) Exception while trying to start console proxy
  575. com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Unable to start instance due to InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  576. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1073)
  577. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  578. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  579. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  580. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  581. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  582. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  583. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  584. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  585. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  586. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  587. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  588. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  589. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  590. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  591. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  592. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  593. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  594. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  595. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  596. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  597. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  598. at java.lang.Thread.run(Thread.java:745)
  599. Caused by: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack
  600. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:150)
  601. at org.apache.cloudstack.framework.async.InplaceAsyncCallbackDriver.performCompletionCallback(InplaceAsyncCallbackDriver.java:26)
  602. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.complete(AsyncCallbackDispatcher.java:122)
  603. at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.copyAsync(CloudStackPrimaryDataStoreDriverImpl.java:233)
  604. at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:56)
  605. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:545)
  606. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:608)
  607. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1192)
  608. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1256)
  609. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:963)
  610. ... 22 more
  611. Caused by: java.lang.reflect.InvocationTargetException
  612. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  613. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  614. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  615. at java.lang.reflect.Method.invoke(Method.java:606)
  616. at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.dispatch(AsyncCallbackDispatcher.java:144)
  617. ... 31 more
  618. Caused by: java.lang.NullPointerException
  619. at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:496)
  620. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageCallBack(VolumeServiceImpl.java:559)
  621. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  622. ... 35 more
  623. 2014-09-30 08:48:28,293 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-2b569a86) Zone 1 is ready to launch console proxy
  624. 2014-09-30 08:48:28,293 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-2b569a86) Expand console proxy standby capacity for zone Rushford HEND
  625. 2014-09-30 08:48:28,298 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-2b569a86) Found a stopped console proxy, bring it up to running pool. proxy vm id : 33
  626. 2014-09-30 08:48:28,311 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
  627. 2014-09-30 08:48:28,312 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-2b569a86) Successfully transitioned to start state for VM[ConsoleProxy|v-33-VM] reservation id = 2024bb12-ae12-4e57-b5cf-94787eb56a50
  628. 2014-09-30 08:48:28,319 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-2b569a86) Trying to deploy VM, vm has dcId: 1 and podId: 1
  629. 2014-09-30 08:48:28,319 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-2b569a86) Deploy avoids pods: null, clusters: null, hosts: null
  630. 2014-09-30 08:48:28,325 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Deploy avoids pods: null, clusters: null, hosts: null
  631. 2014-09-30 08:48:28,326 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@2a0b20fb
  632. 2014-09-30 08:48:28,326 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 1073741824
  633. 2014-09-30 08:48:28,326 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Is ROOT volume READY (pool already allocated)?: No
  634. 2014-09-30 08:48:28,326 DEBUG [c.c.d.FirstFitPlanner] (consoleproxy-1:ctx-2b569a86) Searching resources only under specified Pod: 1
  635. 2014-09-30 08:48:28,327 DEBUG [c.c.d.FirstFitPlanner] (consoleproxy-1:ctx-2b569a86) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  636. 2014-09-30 08:48:28,338 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Checking resources in Cluster: 1 under Pod: 1
  637. 2014-09-30 08:48:28,338 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  638. 2014-09-30 08:48:28,343 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) FirstFitAllocator has 2 hosts to check for allocation: [Host[-1-Routing], Host[-2-Routing]]
  639. 2014-09-30 08:48:28,349 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Found 2 hosts for allocation after prioritization: [Host[-1-Routing], Host[-2-Routing]]
  640. 2014-09-30 08:48:28,349 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=1024
  641. 2014-09-30 08:48:28,361 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Host: 1 has cpu capability (cpu:24, speed:2000) to support requested CPU: 1 and requested speed: 500
  642. 2014-09-30 08:48:28,361 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  643. 2014-09-30 08:48:28,364 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  644. 2014-09-30 08:48:28,364 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Free CPU: 48000 , Requested CPU: 500
  645. 2014-09-30 08:48:28,365 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Free RAM: 67521052672 , Requested RAM: 1073741824
  646. 2014-09-30 08:48:28,365 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Host has enough CPU and RAM available
  647. 2014-09-30 08:48:28,365 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 48000, total with overprovisioning: 48000; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  648. 2014-09-30 08:48:28,365 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 1, used: 0, reserved: 0, total: 67521052672; requested mem: 1073741824,alloc_from_last_host?:false ,considerReservedCapacity?: true
  649. 2014-09-30 08:48:28,365 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Found a suitable host, adding to list: 1
  650. 2014-09-30 08:48:28,375 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Host: 2 has cpu capability (cpu:24, speed:2001) to support requested CPU: 1 and requested speed: 500
  651. 2014-09-30 08:48:28,375 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  652. 2014-09-30 08:48:28,379 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Hosts's actual total CPU: 48024 and CPU after applying overprovisioning: 48024
  653. 2014-09-30 08:48:28,379 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Free CPU: 47524 , Requested CPU: 500
  654. 2014-09-30 08:48:28,379 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Free RAM: 66984202240 , Requested RAM: 1073741824
  655. 2014-09-30 08:48:28,379 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Host has enough CPU and RAM available
  656. 2014-09-30 08:48:28,379 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 48024, total with overprovisioning: 48024; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  657. 2014-09-30 08:48:28,379 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 2, used: 536870912, reserved: 0, total: 67521073152; requested mem: 1073741824,alloc_from_last_host?:false ,considerReservedCapacity?: true
  658. 2014-09-30 08:48:28,379 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Found a suitable host, adding to list: 2
  659. 2014-09-30 08:48:28,379 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (consoleproxy-1:ctx-2b569a86 FirstFitRoutingAllocator) Host Allocator returning 2 suitable hosts
  660. 2014-09-30 08:48:28,381 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Checking suitable pools for volume (Id, Type): (1,ROOT)
  661. 2014-09-30 08:48:28,381 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) We need to allocate new storagepool for this volume
  662. 2014-09-30 08:48:28,381 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Calling StoragePoolAllocators to find suitable pools
  663. 2014-09-30 08:48:28,383 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  664. 2014-09-30 08:48:28,383 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) ClusterScopeStoragePoolAllocator looking for storage pool
  665. 2014-09-30 08:48:28,383 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) Looking for pools in dc: 1 pod:1 cluster:1
  666. 2014-09-30 08:48:28,384 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) Found pools matching tags: [Pool[1|CLVM]]
  667. 2014-09-30 08:48:28,386 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) Removing pool Pool[1|CLVM] from avoid set, must have been inserted when searching for another disk's tag
  668. 2014-09-30 08:48:28,388 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) Checking if storage pool is suitable, name: null ,poolId: 1
  669. 2014-09-30 08:48:28,393 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-2b569a86) Checking pool 1 for storage, totalSize: 2997945892864, usedBytes: 0, usedPct: 0.0, disable threshold: 0.85
  670. 2014-09-30 08:48:28,398 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-2b569a86) Checking pool: 1 for volume allocation [Vol[1|vm=33|ROOT]], maxSize : 2997945892864, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  671. 2014-09-30 08:48:28,398 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-2b569a86) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  672. 2014-09-30 08:48:28,398 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  673. 2014-09-30 08:48:28,398 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Checking if host: 1 can access any suitable storage pool for volume: ROOT
  674. 2014-09-30 08:48:28,399 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Host: 1 can access pool: 1
  675. 2014-09-30 08:48:28,400 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Found a potential host id: 1 name: Flex-H1.flexhost.local and associated storage pools for this VM
  676. 2014-09-30 08:48:28,401 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-2b569a86) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1|ROOT-->Pool(1))]
  677. 2014-09-30 08:48:28,401 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-2b569a86) Deployment found - P0=VM[ConsoleProxy|v-33-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1|ROOT-->Pool(1))]
  678. 2014-09-30 08:48:28,418 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 1 host id before state transition: null
  679. 2014-09-30 08:48:28,428 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) Hosts's actual total CPU: 48000 and CPU after applying overprovisioning: 48000
  680. 2014-09-30 08:48:28,428 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) We are allocating VM, increasing the used capacity of this host:1
  681. 2014-09-30 08:48:28,428 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) Current Used CPU: 0 , Free CPU:48000 ,Requested CPU: 500
  682. 2014-09-30 08:48:28,428 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) Current Used RAM: 0 , Free RAM:67521052672 ,Requested RAM: 1073741824
  683. 2014-09-30 08:48:28,428 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) CPU STATS after allocation: for host: 1, old used: 0, old reserved: 0, actual total: 48000, total with overprovisioning: 48000; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
  684. 2014-09-30 08:48:28,428 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-2b569a86) RAM STATS after allocation: for host: 1, old used: 0, old reserved: 0, total: 67521052672; new used: 1073741824, reserved: 0; requested mem: 1073741824,alloc_from_last:false
  685. 2014-09-30 08:48:28,435 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-2b569a86) VM is being created in podId: 1
  686. 2014-09-30 08:48:28,448 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-2b569a86) Network id=204 is already implemented
  687. 2014-09-30 08:48:28,479 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-2b569a86) Asking SecurityGroupProvider to prepare for Nic[1-33-null-205.185.158.240]
  688. 2014-09-30 08:48:28,482 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-2b569a86) Asking VirtualRouter to prepare for Nic[1-33-null-205.185.158.240]
  689. 2014-09-30 08:48:28,491 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-2b569a86) Network id=201 is already implemented
  690. 2014-09-30 08:48:28,560 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-2b569a86) Network id=200 is already implemented
  691. 2014-09-30 08:48:28,598 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-2b569a86) Allocated a nic NicProfile[3-33-2024bb12-ae12-4e57-b5cf-94787eb56a50-192.168.30.14-null for VM[ConsoleProxy|v-33-VM]
  692. 2014-09-30 08:48:28,611 DEBUG [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-2b569a86) Checking if we need to prepare 1 volumes for VM[ConsoleProxy|v-33-VM]
  693. 2014-09-30 08:48:28,630 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-2b569a86) template 3 is already in store:2, type:Image
  694. 2014-09-30 08:48:28,647 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-2b569a86) template 3 is already in store:1, type:Primary
  695. 2014-09-30 08:48:28,698 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-2b569a86) template 3 is already in store:2, type:Image
  696. 2014-09-30 08:48:28,710 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-2b569a86) Seq 1-792595066: Sending { Cmd , MgmtId: 345050433729, via: 1(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2","uuid":"a3383c10-47eb-11e4-8415-0050569b56d4","id":3,"format":"QCOW2","accountId":1,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://Flex-CSMS.flexhost.local/mnt/secondary","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"3b6a23be-02d2-448a-b070-de030c8a5317","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"83aa613b-a469-4714-b24d-66a728752471","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_s1c0","port":0,"url":"CLVM://localhost//vg_s1c0/?ROLE=Primary&STOREUUID=83aa613b-a469-4714-b24d-66a728752471"}},"name":"ROOT-33","size":0,"volumeId":1,"vmName":"v-33-VM","accountId":1,"format":"QCOW2","id":1,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  697. 2014-09-30 08:48:31,782 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-c223ac75) checking health of usage server
  698. 2014-09-30 08:48:31,784 DEBUG [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-c223ac75) usage server running? false, heartbeat: null
  699. 2014-09-30 08:48:31,784 WARN [o.a.c.alerts] (HA-2:ctx-c223ac75) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running
  700. 2014-09-30 08:48:31,786 DEBUG [c.c.a.AlertManagerImpl] (HA-2:ctx-c223ac75) Have already sent: 1 emails for alert type '13' -- skipping send email
  701. 2014-09-30 08:48:31,997 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-f70a21c0) External devices stats collector is running...
  702. 2014-09-30 08:48:32,007 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-c24ca34b) Found 0 running routers.
  703. 2014-09-30 08:48:32,008 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-f5c6fe5e) Snapshot scheduler.poll is being called at 2014-09-30 13:48:32 GMT
  704. 2014-09-30 08:48:32,009 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-70a6c471) Found 0 routers to update status.
  705. 2014-09-30 08:48:32,010 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-f5c6fe5e) Got 0 snapshots to be executed at 2014-09-30 13:48:32 GMT
  706. 2014-09-30 08:48:32,011 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-70a6c471) Found 0 networks to update RvR status.
  707. 2014-09-30 08:48:37,028 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5eb89def) Resetting hosts suitable for reconnect
  708. 2014-09-30 08:48:37,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5eb89def) Completed resetting hosts suitable for reconnect
  709. 2014-09-30 08:48:37,030 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5eb89def) Acquiring hosts for clusters already owned by this management server
  710. 2014-09-30 08:48:37,031 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5eb89def) Completed acquiring hosts for clusters already owned by this management server
  711. 2014-09-30 08:48:37,031 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5eb89def) Acquiring hosts for clusters not owned by any management server
  712. 2014-09-30 08:48:37,032 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5eb89def) Completed acquiring hosts for clusters not owned by any management server
  713. 2014-09-30 08:48:39,992 INFO [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-aacb8c39) Found the following agents behind on ping: [1]
  714. 2014-09-30 08:48:39,995 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-aacb8c39) Ping timeout for host 1, do invstigation
  715. 2014-09-30 08:48:39,997 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-10:ctx-517a5c53) Investigating why host 1 has disconnected with event PingTimeout
  716. 2014-09-30 08:48:39,998 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-10:ctx-517a5c53) checking if agent (1) is alive
  717. 2014-09-30 08:48:40,005 DEBUG [c.c.a.t.Request] (AgentTaskPool-10:ctx-517a5c53) Seq 1-792595067: Sending { Cmd , MgmtId: 345050433729, via: 1(Flex-H1.flexhost.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckHealthCommand":{"wait":50}}] }
  718. 2014-09-30 08:48:40,009 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) Seq 1-792595067: Processing: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckHealthAnswer":{"result":true,"details":"resource is alive","wait":0}}] }
  719. 2014-09-30 08:48:40,009 DEBUG [c.c.a.t.Request] (AgentTaskPool-10:ctx-517a5c53) Seq 1-792595067: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  720. 2014-09-30 08:48:40,009 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-10:ctx-517a5c53) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is alive
  721. 2014-09-30 08:48:40,009 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-10:ctx-517a5c53) agent (1) responded to checkHeathCommand, reporting that agent is Up
  722. 2014-09-30 08:48:40,009 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-10:ctx-517a5c53) The state determined is Up
  723. 2014-09-30 08:48:40,009 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-10:ctx-517a5c53) Agent is determined to be up and running
  724. 2014-09-30 08:48:40,009 DEBUG [c.c.h.Status] (AgentTaskPool-10:ctx-517a5c53) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 1, name = Flex-H1.flexhost.local]
  725. 2014-09-30 08:48:40,037 DEBUG [c.c.h.Status] (AgentTaskPool-10:ctx-517a5c53) Agent status update: [id = 1; name = Flex-H1.flexhost.local; old status = Up; event = Ping; new status = Up; old update count = 1002; new update count = 1003]
  726. 2014-09-30 08:48:41,891 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-588ddc25) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
  727. 2014-09-30 08:48:41,907 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-588ddc25) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
  728. 2014-09-30 08:48:55,635 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-32158824) VmStatsCollector is running...
  729. 2014-09-30 08:49:01,826 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) Ping from 2
  730. 2014-09-30 08:49:01,915 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aaed52c8) Checking if any host reservation can be released ...
  731. 2014-09-30 08:49:01,923 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aaed52c8) Cannot release reservation, Found 1 VMs Running on host 1
  732. 2014-09-30 08:49:01,930 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aaed52c8) Cannot release reservation, Found 1 VMs Running on host 2
  733. 2014-09-30 08:49:01,930 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aaed52c8) Done running HostReservationReleaseChecker ...
  734. 2014-09-30 08:49:02,010 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-df2e4030) Found 0 routers to update status.
  735. 2014-09-30 08:49:02,012 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-df2e4030) Found 0 networks to update RvR status.
  736. 2014-09-30 08:49:02,014 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Running Capacity Checker ...
  737. 2014-09-30 08:49:02,014 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) recalculating system capacity
  738. 2014-09-30 08:49:02,014 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Executing cpu/ram capacity update
  739. 2014-09-30 08:49:02,020 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) Found 1 VMs on host 1
  740. 2014-09-30 08:49:02,022 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) Found 0 VM, not running on host 1
  741. 2014-09-30 08:49:02,024 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) No need to calibrate cpu capacity, host:1 usedCpu: 500 reservedCpu: 0
  742. 2014-09-30 08:49:02,024 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) No need to calibrate memory capacity, host:1 usedMem: 1073741824 reservedMem: 0
  743. 2014-09-30 08:49:02,030 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) Found 1 VMs on host 2
  744. 2014-09-30 08:49:02,039 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) Found 0 VM, not running on host 2
  745. 2014-09-30 08:49:02,042 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) No need to calibrate cpu capacity, host:2 usedCpu: 500 reservedCpu: 0
  746. 2014-09-30 08:49:02,042 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-8ea2c87a) No need to calibrate memory capacity, host:2 usedMem: 536870912 reservedMem: 0
  747. 2014-09-30 08:49:02,042 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Done executing cpu/ram capacity update
  748. 2014-09-30 08:49:02,042 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Executing storage capacity update
  749. 2014-09-30 08:49:02,050 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-8ea2c87a) Successfully set Capacity - 2997945892864 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 1, PodId 1
  750. 2014-09-30 08:49:02,050 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Done executing storage capacity update
  751. 2014-09-30 08:49:02,050 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Executing capacity updates for public ip and Vlans
  752. 2014-09-30 08:49:02,076 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Done capacity updates for public ip and Vlans
  753. 2014-09-30 08:49:02,076 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Executing capacity updates for private ip
  754. 2014-09-30 08:49:02,084 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Done executing capacity updates for private ip
  755. 2014-09-30 08:49:02,084 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Done recalculating system capacity
  756. 2014-09-30 08:49:02,114 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-8ea2c87a) Done running Capacity Checker ...
  757. 2014-09-30 08:49:10,224 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-3cf592d3) HostStatsCollector is running...
  758. 2014-09-30 08:49:10,855 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-3cf592d3) Seq 1-792595068: Received: { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  759. 2014-09-30 08:49:11,496 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-3cf592d3) Seq 2-2053047775: Received: { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!