API tools faq
paste
Advertisement
Guest User

acs-unable-to-connect-to-xenservers

a guest
Feb 16th, 2016
73
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 92.57 KB | None | 0 0
raw download clone embed print report
  1. 2016-02-16 13:07:45,595 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-cd813b4e) Seq 8-4118541859230318673: Timed out on null
  2. 2016-02-16 13:07:45,595 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-cd813b4e) Seq 8-4118541859230318673: Cancelling.
  3. 2016-02-16 13:07:45,595 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-cd813b4e) Operation timed out: Commands 4118541859230318673 to Host 8 timed out after 3600
  4. 2016-02-16 13:07:45,595 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-cd813b4e) Unable to obtain VM statistics.
  5. 2016-02-16 13:07:55,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b04c1938) Begin cleanup expired async-jobs
  6. 2016-02-16 13:07:55,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b04c1938) End cleanup expired async-jobs
  7. 2016-02-16 13:07:55,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-938f1c3d) Found 12 routers to update status.
  8. 2016-02-16 13:07:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-938f1c3d) Found 0 VPC networks to update Redundant State.
  9. 2016-02-16 13:07:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-938f1c3d) Found 0 networks to update RvR status.
  10. 2016-02-16 13:07:55,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-68545537) Found 12 routers to update status.
  11. 2016-02-16 13:07:55,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-68545537) Found 0 VPC networks to update Redundant State.
  12. 2016-02-16 13:07:55,682 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-68545537) Found 0 networks to update RvR status.
  13. 2016-02-16 13:08:05,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b1a0961f) Begin cleanup expired async-jobs
  14. 2016-02-16 13:08:05,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b1a0961f) End cleanup expired async-jobs
  15. 2016-02-16 13:08:13,435 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-0284e75b) Zone 1 is not ready to launch console proxy yet
  16. 2016-02-16 13:08:13,584 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-a803fd46) Zone 1 is not ready to launch secondary storage VM yet
  17. 2016-02-16 13:08:15,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a43ed60c) Begin cleanup expired async-jobs
  18. 2016-02-16 13:08:15,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a43ed60c) End cleanup expired async-jobs
  19. 2016-02-16 13:08:25,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-0e6d9045) Begin cleanup expired async-jobs
  20. 2016-02-16 13:08:25,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-0e6d9045) End cleanup expired async-jobs
  21. 2016-02-16 13:08:25,455 ERROR [c.c.v.UserVmManagerImpl] (UserVm-Scavenger-1:ctx-36e74b16) Caught the Exception in VmIpFetchTask
  22. java.lang.NullPointerException
  23. at com.cloud.vm.UserVmManagerImpl$VmIpFetchTask.runInContext(UserVmManagerImpl.java:2221)
  24. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  25. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  26. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  27. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  28. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  29. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  30. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  31. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  32. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  33. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  34. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  35. at java.lang.Thread.run(Thread.java:745)
  36. 2016-02-16 13:08:25,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-fe03aa32) Found 12 routers to update status.
  37. 2016-02-16 13:08:25,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-fe03aa32) Found 0 VPC networks to update Redundant State.
  38. 2016-02-16 13:08:25,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-fe03aa32) Found 0 networks to update RvR status.
  39. 2016-02-16 13:08:25,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-af0cd408) Found 12 routers to update status.
  40. 2016-02-16 13:08:25,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-af0cd408) Found 0 VPC networks to update Redundant State.
  41. 2016-02-16 13:08:25,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-af0cd408) Found 0 networks to update RvR status.
  42. 2016-02-16 13:08:30,596 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-bc20ff1e) Resetting hosts suitable for reconnect
  43. 2016-02-16 13:08:30,597 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-bc20ff1e) Completed resetting hosts suitable for reconnect
  44. 2016-02-16 13:08:30,597 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-bc20ff1e) Acquiring hosts for clusters already owned by this management server
  45. 2016-02-16 13:08:30,598 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-bc20ff1e) Completed acquiring hosts for clusters already owned by this management server
  46. 2016-02-16 13:08:30,598 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-bc20ff1e) Acquiring hosts for clusters not owned by any management server
  47. 2016-02-16 13:08:30,598 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-bc20ff1e) Completed acquiring hosts for clusters not owned by any management server
  48. 2016-02-16 13:08:35,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6fe76902) Begin cleanup expired async-jobs
  49. 2016-02-16 13:08:35,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6fe76902) End cleanup expired async-jobs
  50. 2016-02-16 13:08:40,667 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-a32409f3) AutoScaling Monitor is running...
  51. 2016-02-16 13:08:41,291 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-ba01bf70) StorageCollector is running...
  52. 2016-02-16 13:08:41,294 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-ba01bf70) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  53. 2016-02-16 13:08:41,294 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-ba01bf70) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  54. 2016-02-16 13:08:41,295 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-ba01bf70) Seq 6-6515864235874975772: Forwarding null to 57177340185274
  55. 2016-02-16 13:08:41,296 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 6-6515864235874975772: Routing from 57177340185273
  56. 2016-02-16 13:08:41,296 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 6-6515864235874975772: Link is closed
  57. 2016-02-16 13:08:41,296 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 6-6515864235874975772: MgmtId 57177340185273: Req: Resource [Host:6] is unreachable: Host 6: Link is closed
  58. 2016-02-16 13:08:41,296 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 6--1: MgmtId 57177340185273: Req: Routing to peer
  59. 2016-02-16 13:08:41,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 6--1: MgmtId 57177340185273: Req: Cancel request received
  60. 2016-02-16 13:08:41,297 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) Seq 6-6515864235874975772: Cancelling.
  61. 2016-02-16 13:08:41,297 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ba01bf70) Seq 6-6515864235874975772: Waiting some more time because this is the current command
  62. 2016-02-16 13:08:41,297 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ba01bf70) Seq 6-6515864235874975772: Waiting some more time because this is the current command
  63. 2016-02-16 13:08:41,297 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-ba01bf70) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  64. 2016-02-16 13:08:41,297 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ba01bf70) Seq 6-6515864235874975772: Timed out on null
  65. 2016-02-16 13:08:41,297 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ba01bf70) Seq 6-6515864235874975772: Cancelling.
  66. 2016-02-16 13:08:41,297 DEBUG [o.a.c.s.RemoteHostEndPoint] (StatsCollector-3:ctx-ba01bf70) Failed to send command, due to Agent:6, com.cloud.exception.OperationTimedoutException: Commands 6515864235874975772 to Host 6 timed out after 3600
  67. 2016-02-16 13:08:41,297 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-ba01bf70) Error trying to retrieve storage stats
  68. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:6, com.cloud.exception.OperationTimedoutException: Commands 6515864235874975772 to Host 6 timed out after 3600
  69. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:133)
  70. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:725)
  71. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  72. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  73. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  74. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  75. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  76. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  77. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  78. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  79. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  80. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  81. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  82. at java.lang.Thread.run(Thread.java:745)
  83. 2016-02-16 13:08:42,089 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-6ceb9f76) HostStatsCollector is running...
  84. 2016-02-16 13:08:42,094 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428434: Forwarding null to 57177340185274
  85. 2016-02-16 13:08:42,095 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-7555351324867428434: Routing from 57177340185273
  86. 2016-02-16 13:08:42,095 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-7555351324867428434: Link is closed
  87. 2016-02-16 13:08:42,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 1-7555351324867428434: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  88. 2016-02-16 13:08:42,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  89. 2016-02-16 13:08:42,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  90. 2016-02-16 13:08:42,096 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-6:null) Seq 1-7555351324867428434: Cancelling.
  91. 2016-02-16 13:08:42,096 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428434: Waiting some more time because this is the current command
  92. 2016-02-16 13:08:42,096 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428434: Waiting some more time because this is the current command
  93. 2016-02-16 13:08:42,096 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-6ceb9f76) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  94. 2016-02-16 13:08:42,096 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428434: Timed out on null
  95. 2016-02-16 13:08:42,096 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428434: Cancelling.
  96. 2016-02-16 13:08:42,096 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Operation timed out: Commands 7555351324867428434 to Host 1 timed out after 3600
  97. 2016-02-16 13:08:42,096 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Unable to obtain host 1 statistics.
  98. 2016-02-16 13:08:42,096 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-6ceb9f76) Received invalid host stats for host: 1
  99. 2016-02-16 13:08:42,100 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247570: Forwarding null to 57177340185274
  100. 2016-02-16 13:08:42,100 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 3-822751356925247570: Routing from 57177340185273
  101. 2016-02-16 13:08:42,100 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 3-822751356925247570: Link is closed
  102. 2016-02-16 13:08:42,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 3-822751356925247570: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  103. 2016-02-16 13:08:42,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  104. 2016-02-16 13:08:42,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  105. 2016-02-16 13:08:42,101 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) Seq 3-822751356925247570: Cancelling.
  106. 2016-02-16 13:08:42,101 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247570: Waiting some more time because this is the current command
  107. 2016-02-16 13:08:42,101 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247570: Waiting some more time because this is the current command
  108. 2016-02-16 13:08:42,101 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-6ceb9f76) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  109. 2016-02-16 13:08:42,101 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247570: Timed out on null
  110. 2016-02-16 13:08:42,101 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247570: Cancelling.
  111. 2016-02-16 13:08:42,101 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Operation timed out: Commands 822751356925247570 to Host 3 timed out after 3600
  112. 2016-02-16 13:08:42,101 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Unable to obtain host 3 statistics.
  113. 2016-02-16 13:08:42,101 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-6ceb9f76) Received invalid host stats for host: 3
  114. 2016-02-16 13:08:42,105 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318674: Forwarding null to 57177340185274
  115. 2016-02-16 13:08:42,105 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 8-4118541859230318674: Routing from 57177340185273
  116. 2016-02-16 13:08:42,105 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 8-4118541859230318674: Link is closed
  117. 2016-02-16 13:08:42,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) Seq 8-4118541859230318674: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  118. 2016-02-16 13:08:42,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  119. 2016-02-16 13:08:42,106 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  120. 2016-02-16 13:08:42,106 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) Seq 8-4118541859230318674: Cancelling.
  121. 2016-02-16 13:08:42,106 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318674: Waiting some more time because this is the current command
  122. 2016-02-16 13:08:42,106 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318674: Waiting some more time because this is the current command
  123. 2016-02-16 13:08:42,106 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-6ceb9f76) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  124. 2016-02-16 13:08:42,106 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318674: Timed out on null
  125. 2016-02-16 13:08:42,106 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318674: Cancelling.
  126. 2016-02-16 13:08:42,106 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Operation timed out: Commands 4118541859230318674 to Host 8 timed out after 3600
  127. 2016-02-16 13:08:42,106 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Unable to obtain host 8 statistics.
  128. 2016-02-16 13:08:42,107 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-6ceb9f76) Received invalid host stats for host: 8
  129. 2016-02-16 13:08:42,109 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428435: Forwarding null to 57177340185274
  130. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-7555351324867428435: Routing from 57177340185273
  131. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-7555351324867428435: Link is closed
  132. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 1-7555351324867428435: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  133. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  134. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  135. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) Seq 1-7555351324867428435: Cancelling.
  136. 2016-02-16 13:08:42,110 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428435: Waiting some more time because this is the current command
  137. 2016-02-16 13:08:42,111 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428435: Waiting some more time because this is the current command
  138. 2016-02-16 13:08:42,111 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-6ceb9f76) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  139. 2016-02-16 13:08:42,111 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428435: Timed out on null
  140. 2016-02-16 13:08:42,111 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 1-7555351324867428435: Cancelling.
  141. 2016-02-16 13:08:42,111 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Operation timed out: Commands 7555351324867428435 to Host 1 timed out after 3600
  142. 2016-02-16 13:08:42,111 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Unable to obtain GPU stats for host acs-compute-1
  143. 2016-02-16 13:08:42,113 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247571: Forwarding null to 57177340185274
  144. 2016-02-16 13:08:42,114 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 3-822751356925247571: Routing from 57177340185273
  145. 2016-02-16 13:08:42,114 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 3-822751356925247571: Link is closed
  146. 2016-02-16 13:08:42,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 3-822751356925247571: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  147. 2016-02-16 13:08:42,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  148. 2016-02-16 13:08:42,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  149. 2016-02-16 13:08:42,115 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-14:null) Seq 3-822751356925247571: Cancelling.
  150. 2016-02-16 13:08:42,115 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247571: Waiting some more time because this is the current command
  151. 2016-02-16 13:08:42,115 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247571: Waiting some more time because this is the current command
  152. 2016-02-16 13:08:42,115 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-6ceb9f76) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  153. 2016-02-16 13:08:42,115 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247571: Timed out on null
  154. 2016-02-16 13:08:42,115 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 3-822751356925247571: Cancelling.
  155. 2016-02-16 13:08:42,115 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Operation timed out: Commands 822751356925247571 to Host 3 timed out after 3600
  156. 2016-02-16 13:08:42,115 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Unable to obtain GPU stats for host acs-compute-2
  157. 2016-02-16 13:08:42,118 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318675: Forwarding null to 57177340185274
  158. 2016-02-16 13:08:42,118 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 8-4118541859230318675: Routing from 57177340185273
  159. 2016-02-16 13:08:42,118 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 8-4118541859230318675: Link is closed
  160. 2016-02-16 13:08:42,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 8-4118541859230318675: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  161. 2016-02-16 13:08:42,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  162. 2016-02-16 13:08:42,119 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  163. 2016-02-16 13:08:42,119 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) Seq 8-4118541859230318675: Cancelling.
  164. 2016-02-16 13:08:42,119 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318675: Waiting some more time because this is the current command
  165. 2016-02-16 13:08:42,119 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318675: Waiting some more time because this is the current command
  166. 2016-02-16 13:08:42,119 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-6ceb9f76) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  167. 2016-02-16 13:08:42,119 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318675: Timed out on null
  168. 2016-02-16 13:08:42,119 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-6ceb9f76) Seq 8-4118541859230318675: Cancelling.
  169. 2016-02-16 13:08:42,119 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Operation timed out: Commands 4118541859230318675 to Host 8 timed out after 3600
  170. 2016-02-16 13:08:42,119 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-6ceb9f76) Unable to obtain GPU stats for host acs-compute-3
  171. 2016-02-16 13:08:43,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-1b4841ee) Zone 1 is not ready to launch console proxy yet
  172. 2016-02-16 13:08:43,584 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-47783e44) Zone 1 is not ready to launch secondary storage VM yet
  173. 2016-02-16 13:08:45,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-63b96122) Begin cleanup expired async-jobs
  174. 2016-02-16 13:08:45,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-63b96122) End cleanup expired async-jobs
  175. 2016-02-16 13:08:45,595 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-223d0112) VmStatsCollector is running...
  176. 2016-02-16 13:08:45,658 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-223d0112) Seq 1-7555351324867428436: Forwarding null to 57177340185274
  177. 2016-02-16 13:08:45,659 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-7555351324867428436: Routing from 57177340185273
  178. 2016-02-16 13:08:45,659 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-7555351324867428436: Link is closed
  179. 2016-02-16 13:08:45,659 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 1-7555351324867428436: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  180. 2016-02-16 13:08:45,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  181. 2016-02-16 13:08:45,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  182. 2016-02-16 13:08:45,660 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) Seq 1-7555351324867428436: Cancelling.
  183. 2016-02-16 13:08:45,660 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 1-7555351324867428436: Waiting some more time because this is the current command
  184. 2016-02-16 13:08:45,660 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 1-7555351324867428436: Waiting some more time because this is the current command
  185. 2016-02-16 13:08:45,660 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-223d0112) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  186. 2016-02-16 13:08:45,660 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 1-7555351324867428436: Timed out on null
  187. 2016-02-16 13:08:45,661 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 1-7555351324867428436: Cancelling.
  188. 2016-02-16 13:08:45,661 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-223d0112) Operation timed out: Commands 7555351324867428436 to Host 1 timed out after 3600
  189. 2016-02-16 13:08:45,661 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-223d0112) Unable to obtain VM statistics.
  190. 2016-02-16 13:08:45,709 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-223d0112) Seq 3-822751356925247572: Forwarding null to 57177340185274
  191. 2016-02-16 13:08:45,710 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 3-822751356925247572: Routing from 57177340185273
  192. 2016-02-16 13:08:45,710 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 3-822751356925247572: Link is closed
  193. 2016-02-16 13:08:45,710 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 3-822751356925247572: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  194. 2016-02-16 13:08:45,710 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  195. 2016-02-16 13:08:45,711 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  196. 2016-02-16 13:08:45,711 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) Seq 3-822751356925247572: Cancelling.
  197. 2016-02-16 13:08:45,711 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 3-822751356925247572: Waiting some more time because this is the current command
  198. 2016-02-16 13:08:45,711 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 3-822751356925247572: Waiting some more time because this is the current command
  199. 2016-02-16 13:08:45,711 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-223d0112) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  200. 2016-02-16 13:08:45,711 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 3-822751356925247572: Timed out on null
  201. 2016-02-16 13:08:45,711 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 3-822751356925247572: Cancelling.
  202. 2016-02-16 13:08:45,711 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-223d0112) Operation timed out: Commands 822751356925247572 to Host 3 timed out after 3600
  203. 2016-02-16 13:08:45,711 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-223d0112) Unable to obtain VM statistics.
  204. 2016-02-16 13:08:45,736 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-223d0112) Seq 8-4118541859230318676: Forwarding null to 57177340185274
  205. 2016-02-16 13:08:45,738 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 8-4118541859230318676: Routing from 57177340185273
  206. 2016-02-16 13:08:45,738 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 8-4118541859230318676: Link is closed
  207. 2016-02-16 13:08:45,738 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 8-4118541859230318676: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  208. 2016-02-16 13:08:45,738 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  209. 2016-02-16 13:08:45,739 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  210. 2016-02-16 13:08:45,739 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) Seq 8-4118541859230318676: Cancelling.
  211. 2016-02-16 13:08:45,739 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 8-4118541859230318676: Waiting some more time because this is the current command
  212. 2016-02-16 13:08:45,739 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 8-4118541859230318676: Waiting some more time because this is the current command
  213. 2016-02-16 13:08:45,739 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-223d0112) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  214. 2016-02-16 13:08:45,739 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 8-4118541859230318676: Timed out on null
  215. 2016-02-16 13:08:45,739 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-223d0112) Seq 8-4118541859230318676: Cancelling.
  216. 2016-02-16 13:08:45,739 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-223d0112) Operation timed out: Commands 4118541859230318676 to Host 8 timed out after 3600
  217. 2016-02-16 13:08:45,739 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-223d0112) Unable to obtain VM statistics.
  218. 2016-02-16 13:08:55,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-244ecfc5) Begin cleanup expired async-jobs
  219. 2016-02-16 13:08:55,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-244ecfc5) End cleanup expired async-jobs
  220. 2016-02-16 13:08:55,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-65643011) Found 12 routers to update status.
  221. 2016-02-16 13:08:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-65643011) Found 0 VPC networks to update Redundant State.
  222. 2016-02-16 13:08:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-65643011) Found 0 networks to update RvR status.
  223. 2016-02-16 13:08:55,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d9994103) Found 12 routers to update status.
  224. 2016-02-16 13:08:55,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d9994103) Found 0 VPC networks to update Redundant State.
  225. 2016-02-16 13:08:55,688 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d9994103) Found 0 networks to update RvR status.
  226. 2016-02-16 13:09:05,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-f24ef092) Begin cleanup expired async-jobs
  227. 2016-02-16 13:09:05,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-f24ef092) End cleanup expired async-jobs
  228. 2016-02-16 13:09:13,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-29e08021) Zone 1 is not ready to launch console proxy yet
  229. 2016-02-16 13:09:13,584 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-d7aa8a49) Zone 1 is not ready to launch secondary storage VM yet
  230. 2016-02-16 13:09:15,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-be25d70a) Begin cleanup expired async-jobs
  231. 2016-02-16 13:09:15,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-be25d70a) End cleanup expired async-jobs
  232. 2016-02-16 13:09:25,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b8508a66) Begin cleanup expired async-jobs
  233. 2016-02-16 13:09:25,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b8508a66) End cleanup expired async-jobs
  234. 2016-02-16 13:09:25,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-273a6372) Found 12 routers to update status.
  235. 2016-02-16 13:09:25,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-273a6372) Found 0 VPC networks to update Redundant State.
  236. 2016-02-16 13:09:25,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-273a6372) Found 0 networks to update RvR status.
  237. 2016-02-16 13:09:25,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-dd379543) Found 12 routers to update status.
  238. 2016-02-16 13:09:25,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-dd379543) Found 0 VPC networks to update Redundant State.
  239. 2016-02-16 13:09:25,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-dd379543) Found 0 networks to update RvR status.
  240. 2016-02-16 13:09:35,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-feae88cd) Begin cleanup expired async-jobs
  241. 2016-02-16 13:09:35,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-feae88cd) End cleanup expired async-jobs
  242. 2016-02-16 13:09:40,668 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-e549e487) AutoScaling Monitor is running...
  243. 2016-02-16 13:09:41,298 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-df8a131f) StorageCollector is running...
  244. 2016-02-16 13:09:41,301 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df8a131f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  245. 2016-02-16 13:09:41,301 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df8a131f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  246. 2016-02-16 13:09:41,302 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df8a131f) Seq 6-6515864235874975773: Forwarding null to 57177340185274
  247. 2016-02-16 13:09:41,302 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 6-6515864235874975773: Routing from 57177340185273
  248. 2016-02-16 13:09:41,302 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 6-6515864235874975773: Link is closed
  249. 2016-02-16 13:09:41,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 6-6515864235874975773: MgmtId 57177340185273: Req: Resource [Host:6] is unreachable: Host 6: Link is closed
  250. 2016-02-16 13:09:41,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 6--1: MgmtId 57177340185273: Req: Routing to peer
  251. 2016-02-16 13:09:41,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) Seq 6--1: MgmtId 57177340185273: Req: Cancel request received
  252. 2016-02-16 13:09:41,303 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-9:null) Seq 6-6515864235874975773: Cancelling.
  253. 2016-02-16 13:09:41,303 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df8a131f) Seq 6-6515864235874975773: Waiting some more time because this is the current command
  254. 2016-02-16 13:09:41,303 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df8a131f) Seq 6-6515864235874975773: Waiting some more time because this is the current command
  255. 2016-02-16 13:09:41,303 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df8a131f) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  256. 2016-02-16 13:09:41,303 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df8a131f) Seq 6-6515864235874975773: Timed out on null
  257. 2016-02-16 13:09:41,304 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df8a131f) Seq 6-6515864235874975773: Cancelling.
  258. 2016-02-16 13:09:41,304 DEBUG [o.a.c.s.RemoteHostEndPoint] (StatsCollector-3:ctx-df8a131f) Failed to send command, due to Agent:6, com.cloud.exception.OperationTimedoutException: Commands 6515864235874975773 to Host 6 timed out after 3600
  259. 2016-02-16 13:09:41,304 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-df8a131f) Error trying to retrieve storage stats
  260. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:6, com.cloud.exception.OperationTimedoutException: Commands 6515864235874975773 to Host 6 timed out after 3600
  261. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:133)
  262. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:725)
  263. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  264. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  265. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  266. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  267. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  268. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  269. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  270. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  271. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  272. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  273. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  274. at java.lang.Thread.run(Thread.java:745)
  275. 2016-02-16 13:09:42,120 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-a9c7d574) HostStatsCollector is running...
  276. 2016-02-16 13:09:42,124 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428437: Forwarding null to 57177340185274
  277. 2016-02-16 13:09:42,125 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-7555351324867428437: Routing from 57177340185273
  278. 2016-02-16 13:09:42,125 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-7555351324867428437: Link is closed
  279. 2016-02-16 13:09:42,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) Seq 1-7555351324867428437: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  280. 2016-02-16 13:09:42,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  281. 2016-02-16 13:09:42,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  282. 2016-02-16 13:09:42,126 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) Seq 1-7555351324867428437: Cancelling.
  283. 2016-02-16 13:09:42,126 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428437: Waiting some more time because this is the current command
  284. 2016-02-16 13:09:42,126 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428437: Waiting some more time because this is the current command
  285. 2016-02-16 13:09:42,126 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-a9c7d574) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  286. 2016-02-16 13:09:42,126 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428437: Timed out on null
  287. 2016-02-16 13:09:42,126 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428437: Cancelling.
  288. 2016-02-16 13:09:42,126 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-a9c7d574) Operation timed out: Commands 7555351324867428437 to Host 1 timed out after 3600
  289. 2016-02-16 13:09:42,126 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-a9c7d574) Unable to obtain host 1 statistics.
  290. 2016-02-16 13:09:42,127 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-a9c7d574) Received invalid host stats for host: 1
  291. 2016-02-16 13:09:42,130 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247573: Forwarding null to 57177340185274
  292. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 3-822751356925247573: Routing from 57177340185273
  293. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 3-822751356925247573: Link is closed
  294. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 3-822751356925247573: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  295. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  296. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  297. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-13:null) Seq 3-822751356925247573: Cancelling.
  298. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247573: Waiting some more time because this is the current command
  299. 2016-02-16 13:09:42,131 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247573: Waiting some more time because this is the current command
  300. 2016-02-16 13:09:42,132 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-a9c7d574) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  301. 2016-02-16 13:09:42,132 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247573: Timed out on null
  302. 2016-02-16 13:09:42,132 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247573: Cancelling.
  303. 2016-02-16 13:09:42,132 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-a9c7d574) Operation timed out: Commands 822751356925247573 to Host 3 timed out after 3600
  304. 2016-02-16 13:09:42,132 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-a9c7d574) Unable to obtain host 3 statistics.
  305. 2016-02-16 13:09:42,132 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-a9c7d574) Received invalid host stats for host: 3
  306. 2016-02-16 13:09:42,135 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318677: Forwarding null to 57177340185274
  307. 2016-02-16 13:09:42,136 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 8-4118541859230318677: Routing from 57177340185273
  308. 2016-02-16 13:09:42,136 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 8-4118541859230318677: Link is closed
  309. 2016-02-16 13:09:42,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 8-4118541859230318677: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  310. 2016-02-16 13:09:42,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  311. 2016-02-16 13:09:42,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  312. 2016-02-16 13:09:42,137 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) Seq 8-4118541859230318677: Cancelling.
  313. 2016-02-16 13:09:42,137 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318677: Waiting some more time because this is the current command
  314. 2016-02-16 13:09:42,137 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318677: Waiting some more time because this is the current command
  315. 2016-02-16 13:09:42,137 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-a9c7d574) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  316. 2016-02-16 13:09:42,137 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318677: Timed out on null
  317. 2016-02-16 13:09:42,137 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318677: Cancelling.
  318. 2016-02-16 13:09:42,137 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-a9c7d574) Operation timed out: Commands 4118541859230318677 to Host 8 timed out after 3600
  319. 2016-02-16 13:09:42,137 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-a9c7d574) Unable to obtain host 8 statistics.
  320. 2016-02-16 13:09:42,137 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-a9c7d574) Received invalid host stats for host: 8
  321. 2016-02-16 13:09:42,140 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428438: Forwarding null to 57177340185274
  322. 2016-02-16 13:09:42,140 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-7555351324867428438: Routing from 57177340185273
  323. 2016-02-16 13:09:42,140 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-7555351324867428438: Link is closed
  324. 2016-02-16 13:09:42,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 1-7555351324867428438: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  325. 2016-02-16 13:09:42,141 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  326. 2016-02-16 13:09:42,141 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  327. 2016-02-16 13:09:42,141 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-5:null) Seq 1-7555351324867428438: Cancelling.
  328. 2016-02-16 13:09:42,141 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428438: Waiting some more time because this is the current command
  329. 2016-02-16 13:09:42,141 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428438: Waiting some more time because this is the current command
  330. 2016-02-16 13:09:42,141 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-a9c7d574) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  331. 2016-02-16 13:09:42,141 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428438: Timed out on null
  332. 2016-02-16 13:09:42,141 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 1-7555351324867428438: Cancelling.
  333. 2016-02-16 13:09:42,141 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-a9c7d574) Operation timed out: Commands 7555351324867428438 to Host 1 timed out after 3600
  334. 2016-02-16 13:09:42,141 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-a9c7d574) Unable to obtain GPU stats for host acs-compute-1
  335. 2016-02-16 13:09:42,144 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247574: Forwarding null to 57177340185274
  336. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 3-822751356925247574: Routing from 57177340185273
  337. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 3-822751356925247574: Link is closed
  338. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 3-822751356925247574: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  339. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  340. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  341. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) Seq 3-822751356925247574: Cancelling.
  342. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247574: Waiting some more time because this is the current command
  343. 2016-02-16 13:09:42,145 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247574: Waiting some more time because this is the current command
  344. 2016-02-16 13:09:42,145 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-a9c7d574) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  345. 2016-02-16 13:09:42,146 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247574: Timed out on null
  346. 2016-02-16 13:09:42,146 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 3-822751356925247574: Cancelling.
  347. 2016-02-16 13:09:42,146 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-a9c7d574) Operation timed out: Commands 822751356925247574 to Host 3 timed out after 3600
  348. 2016-02-16 13:09:42,146 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-a9c7d574) Unable to obtain GPU stats for host acs-compute-2
  349. 2016-02-16 13:09:42,148 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318678: Forwarding null to 57177340185274
  350. 2016-02-16 13:09:42,149 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 8-4118541859230318678: Routing from 57177340185273
  351. 2016-02-16 13:09:42,149 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 8-4118541859230318678: Link is closed
  352. 2016-02-16 13:09:42,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 8-4118541859230318678: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  353. 2016-02-16 13:09:42,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  354. 2016-02-16 13:09:42,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  355. 2016-02-16 13:09:42,150 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-6:null) Seq 8-4118541859230318678: Cancelling.
  356. 2016-02-16 13:09:42,150 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318678: Waiting some more time because this is the current command
  357. 2016-02-16 13:09:42,150 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318678: Waiting some more time because this is the current command
  358. 2016-02-16 13:09:42,150 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-a9c7d574) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  359. 2016-02-16 13:09:42,150 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318678: Timed out on null
  360. 2016-02-16 13:09:42,150 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-a9c7d574) Seq 8-4118541859230318678: Cancelling.
  361. 2016-02-16 13:09:42,150 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-a9c7d574) Operation timed out: Commands 4118541859230318678 to Host 8 timed out after 3600
  362. 2016-02-16 13:09:42,150 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-a9c7d574) Unable to obtain GPU stats for host acs-compute-3
  363. 2016-02-16 13:09:43,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-c3b67b97) Zone 1 is not ready to launch console proxy yet
  364. 2016-02-16 13:09:43,584 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-94c1b911) Zone 1 is not ready to launch secondary storage VM yet
  365. 2016-02-16 13:09:45,195 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-ed7d4576) Begin cleanup expired async-jobs
  366. 2016-02-16 13:09:45,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-ed7d4576) End cleanup expired async-jobs
  367. 2016-02-16 13:09:45,739 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-b65e7e6f) VmStatsCollector is running...
  368. 2016-02-16 13:09:45,803 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 1-7555351324867428439: Forwarding null to 57177340185274
  369. 2016-02-16 13:09:45,804 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-7555351324867428439: Routing from 57177340185273
  370. 2016-02-16 13:09:45,804 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-7555351324867428439: Link is closed
  371. 2016-02-16 13:09:45,804 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 1-7555351324867428439: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  372. 2016-02-16 13:09:45,804 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  373. 2016-02-16 13:09:45,805 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  374. 2016-02-16 13:09:45,805 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) Seq 1-7555351324867428439: Cancelling.
  375. 2016-02-16 13:09:45,805 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 1-7555351324867428439: Waiting some more time because this is the current command
  376. 2016-02-16 13:09:45,805 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 1-7555351324867428439: Waiting some more time because this is the current command
  377. 2016-02-16 13:09:45,805 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-b65e7e6f) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  378. 2016-02-16 13:09:45,805 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 1-7555351324867428439: Timed out on null
  379. 2016-02-16 13:09:45,805 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 1-7555351324867428439: Cancelling.
  380. 2016-02-16 13:09:45,805 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b65e7e6f) Operation timed out: Commands 7555351324867428439 to Host 1 timed out after 3600
  381. 2016-02-16 13:09:45,806 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-b65e7e6f) Unable to obtain VM statistics.
  382. 2016-02-16 13:09:45,853 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 3-822751356925247575: Forwarding null to 57177340185274
  383. 2016-02-16 13:09:45,854 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 3-822751356925247575: Routing from 57177340185273
  384. 2016-02-16 13:09:45,854 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 3-822751356925247575: Link is closed
  385. 2016-02-16 13:09:45,854 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) Seq 3-822751356925247575: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  386. 2016-02-16 13:09:45,855 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  387. 2016-02-16 13:09:45,855 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  388. 2016-02-16 13:09:45,855 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) Seq 3-822751356925247575: Cancelling.
  389. 2016-02-16 13:09:45,855 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 3-822751356925247575: Waiting some more time because this is the current command
  390. 2016-02-16 13:09:45,855 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 3-822751356925247575: Waiting some more time because this is the current command
  391. 2016-02-16 13:09:45,855 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-b65e7e6f) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  392. 2016-02-16 13:09:45,855 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 3-822751356925247575: Timed out on null
  393. 2016-02-16 13:09:45,855 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 3-822751356925247575: Cancelling.
  394. 2016-02-16 13:09:45,856 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b65e7e6f) Operation timed out: Commands 822751356925247575 to Host 3 timed out after 3600
  395. 2016-02-16 13:09:45,856 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-b65e7e6f) Unable to obtain VM statistics.
  396. 2016-02-16 13:09:45,880 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 8-4118541859230318679: Forwarding null to 57177340185274
  397. 2016-02-16 13:09:45,881 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 8-4118541859230318679: Routing from 57177340185273
  398. 2016-02-16 13:09:45,881 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 8-4118541859230318679: Link is closed
  399. 2016-02-16 13:09:45,881 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 8-4118541859230318679: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  400. 2016-02-16 13:09:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  401. 2016-02-16 13:09:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  402. 2016-02-16 13:09:45,882 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) Seq 8-4118541859230318679: Cancelling.
  403. 2016-02-16 13:09:45,882 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 8-4118541859230318679: Waiting some more time because this is the current command
  404. 2016-02-16 13:09:45,882 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 8-4118541859230318679: Waiting some more time because this is the current command
  405. 2016-02-16 13:09:45,882 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-b65e7e6f) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  406. 2016-02-16 13:09:45,882 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 8-4118541859230318679: Timed out on null
  407. 2016-02-16 13:09:45,883 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b65e7e6f) Seq 8-4118541859230318679: Cancelling.
  408. 2016-02-16 13:09:45,883 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b65e7e6f) Operation timed out: Commands 4118541859230318679 to Host 8 timed out after 3600
  409. 2016-02-16 13:09:45,883 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-b65e7e6f) Unable to obtain VM statistics.
  410. 2016-02-16 13:09:55,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-70aa41f9) Begin cleanup expired async-jobs
  411. 2016-02-16 13:09:55,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-70aa41f9) End cleanup expired async-jobs
  412. 2016-02-16 13:09:55,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-60b510df) Found 12 routers to update status.
  413. 2016-02-16 13:09:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-60b510df) Found 0 VPC networks to update Redundant State.
  414. 2016-02-16 13:09:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-60b510df) Found 0 networks to update RvR status.
  415. 2016-02-16 13:09:55,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-17a967b1) Found 12 routers to update status.
  416. 2016-02-16 13:09:55,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-17a967b1) Found 0 VPC networks to update Redundant State.
  417. 2016-02-16 13:09:55,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-17a967b1) Found 0 networks to update RvR status.
  418. 2016-02-16 13:10:00,596 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-6b815b59) Resetting hosts suitable for reconnect
  419. 2016-02-16 13:10:00,597 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-6b815b59) Completed resetting hosts suitable for reconnect
  420. 2016-02-16 13:10:00,606 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-6b815b59) Acquiring hosts for clusters already owned by this management server
  421. 2016-02-16 13:10:00,606 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-6b815b59) Completed acquiring hosts for clusters already owned by this management server
  422. 2016-02-16 13:10:00,606 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-6b815b59) Acquiring hosts for clusters not owned by any management server
  423. 2016-02-16 13:10:00,607 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-6b815b59) Completed acquiring hosts for clusters not owned by any management server
  424. 2016-02-16 13:10:05,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-8c4604ec) Begin cleanup expired async-jobs
  425. 2016-02-16 13:10:05,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-8c4604ec) End cleanup expired async-jobs
  426. 2016-02-16 13:10:13,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-c3a75a74) Zone 1 is not ready to launch console proxy yet
  427. 2016-02-16 13:10:13,585 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-e3946029) Zone 1 is not ready to launch secondary storage VM yet
  428. 2016-02-16 13:10:15,195 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-06b62246) Begin cleanup expired async-jobs
  429. 2016-02-16 13:10:15,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-06b62246) End cleanup expired async-jobs
  430. 2016-02-16 13:10:25,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b443da29) Begin cleanup expired async-jobs
  431. 2016-02-16 13:10:25,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-b443da29) End cleanup expired async-jobs
  432. 2016-02-16 13:10:25,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-636287a8) Found 12 routers to update status.
  433. 2016-02-16 13:10:25,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-636287a8) Found 0 VPC networks to update Redundant State.
  434. 2016-02-16 13:10:25,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-636287a8) Found 0 networks to update RvR status.
  435. 2016-02-16 13:10:25,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d9583ee) Found 12 routers to update status.
  436. 2016-02-16 13:10:25,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d9583ee) Found 0 VPC networks to update Redundant State.
  437. 2016-02-16 13:10:25,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d9583ee) Found 0 networks to update RvR status.
  438. 2016-02-16 13:10:35,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6497fc7a) Begin cleanup expired async-jobs
  439. 2016-02-16 13:10:35,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6497fc7a) End cleanup expired async-jobs
  440. 2016-02-16 13:10:40,669 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-3afca4f6) AutoScaling Monitor is running...
  441. 2016-02-16 13:10:41,304 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-b31c639e) StorageCollector is running...
  442. 2016-02-16 13:10:41,307 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-b31c639e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  443. 2016-02-16 13:10:41,307 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-b31c639e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  444. 2016-02-16 13:10:41,308 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-b31c639e) Seq 6-6515864235874975774: Forwarding null to 57177340185274
  445. 2016-02-16 13:10:41,309 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 6-6515864235874975774: Routing from 57177340185273
  446. 2016-02-16 13:10:41,309 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 6-6515864235874975774: Link is closed
  447. 2016-02-16 13:10:41,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 6-6515864235874975774: MgmtId 57177340185273: Req: Resource [Host:6] is unreachable: Host 6: Link is closed
  448. 2016-02-16 13:10:41,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 6--1: MgmtId 57177340185273: Req: Routing to peer
  449. 2016-02-16 13:10:41,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 6--1: MgmtId 57177340185273: Req: Cancel request received
  450. 2016-02-16 13:10:41,310 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-14:null) Seq 6-6515864235874975774: Cancelling.
  451. 2016-02-16 13:10:41,310 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-b31c639e) Seq 6-6515864235874975774: Waiting some more time because this is the current command
  452. 2016-02-16 13:10:41,310 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-b31c639e) Seq 6-6515864235874975774: Waiting some more time because this is the current command
  453. 2016-02-16 13:10:41,310 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-b31c639e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  454. 2016-02-16 13:10:41,310 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-b31c639e) Seq 6-6515864235874975774: Timed out on null
  455. 2016-02-16 13:10:41,310 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-b31c639e) Seq 6-6515864235874975774: Cancelling.
  456. 2016-02-16 13:10:41,310 DEBUG [o.a.c.s.RemoteHostEndPoint] (StatsCollector-3:ctx-b31c639e) Failed to send command, due to Agent:6, com.cloud.exception.OperationTimedoutException: Commands 6515864235874975774 to Host 6 timed out after 3600
  457. 2016-02-16 13:10:41,310 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-b31c639e) Error trying to retrieve storage stats
  458. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:6, com.cloud.exception.OperationTimedoutException: Commands 6515864235874975774 to Host 6 timed out after 3600
  459. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:133)
  460. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:725)
  461. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  462. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  463. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  464. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  465. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  466. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  467. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  468. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  469. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  470. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  471. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  472. at java.lang.Thread.run(Thread.java:745)
  473. 2016-02-16 13:10:42,151 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-f2889d19) HostStatsCollector is running...
  474. 2016-02-16 13:10:42,155 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428440: Forwarding null to 57177340185274
  475. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-7555351324867428440: Routing from 57177340185273
  476. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-7555351324867428440: Link is closed
  477. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 1-7555351324867428440: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  478. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  479. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  480. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) Seq 1-7555351324867428440: Cancelling.
  481. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428440: Waiting some more time because this is the current command
  482. 2016-02-16 13:10:42,156 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428440: Waiting some more time because this is the current command
  483. 2016-02-16 13:10:42,157 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-f2889d19) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  484. 2016-02-16 13:10:42,157 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428440: Timed out on null
  485. 2016-02-16 13:10:42,157 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428440: Cancelling.
  486. 2016-02-16 13:10:42,157 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-f2889d19) Operation timed out: Commands 7555351324867428440 to Host 1 timed out after 3600
  487. 2016-02-16 13:10:42,157 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-f2889d19) Unable to obtain host 1 statistics.
  488. 2016-02-16 13:10:42,157 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-f2889d19) Received invalid host stats for host: 1
  489. 2016-02-16 13:10:42,160 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247576: Forwarding null to 57177340185274
  490. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 3-822751356925247576: Routing from 57177340185273
  491. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 3-822751356925247576: Link is closed
  492. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 3-822751356925247576: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  493. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  494. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  495. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) Seq 3-822751356925247576: Cancelling.
  496. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247576: Waiting some more time because this is the current command
  497. 2016-02-16 13:10:42,161 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247576: Waiting some more time because this is the current command
  498. 2016-02-16 13:10:42,162 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-f2889d19) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  499. 2016-02-16 13:10:42,162 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247576: Timed out on null
  500. 2016-02-16 13:10:42,162 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247576: Cancelling.
  501. 2016-02-16 13:10:42,162 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-f2889d19) Operation timed out: Commands 822751356925247576 to Host 3 timed out after 3600
  502. 2016-02-16 13:10:42,162 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-f2889d19) Unable to obtain host 3 statistics.
  503. 2016-02-16 13:10:42,162 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-f2889d19) Received invalid host stats for host: 3
  504. 2016-02-16 13:10:42,165 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318680: Forwarding null to 57177340185274
  505. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 8-4118541859230318680: Routing from 57177340185273
  506. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 8-4118541859230318680: Link is closed
  507. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 8-4118541859230318680: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  508. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  509. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  510. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) Seq 8-4118541859230318680: Cancelling.
  511. 2016-02-16 13:10:42,166 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318680: Waiting some more time because this is the current command
  512. 2016-02-16 13:10:42,167 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318680: Waiting some more time because this is the current command
  513. 2016-02-16 13:10:42,167 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-f2889d19) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  514. 2016-02-16 13:10:42,167 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318680: Timed out on null
  515. 2016-02-16 13:10:42,167 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318680: Cancelling.
  516. 2016-02-16 13:10:42,167 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-f2889d19) Operation timed out: Commands 4118541859230318680 to Host 8 timed out after 3600
  517. 2016-02-16 13:10:42,167 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-f2889d19) Unable to obtain host 8 statistics.
  518. 2016-02-16 13:10:42,167 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-f2889d19) Received invalid host stats for host: 8
  519. 2016-02-16 13:10:42,169 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428441: Forwarding null to 57177340185274
  520. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-7555351324867428441: Routing from 57177340185273
  521. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-7555351324867428441: Link is closed
  522. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 1-7555351324867428441: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  523. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  524. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  525. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) Seq 1-7555351324867428441: Cancelling.
  526. 2016-02-16 13:10:42,170 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428441: Waiting some more time because this is the current command
  527. 2016-02-16 13:10:42,171 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428441: Waiting some more time because this is the current command
  528. 2016-02-16 13:10:42,171 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-f2889d19) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  529. 2016-02-16 13:10:42,171 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428441: Timed out on null
  530. 2016-02-16 13:10:42,171 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 1-7555351324867428441: Cancelling.
  531. 2016-02-16 13:10:42,171 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-f2889d19) Operation timed out: Commands 7555351324867428441 to Host 1 timed out after 3600
  532. 2016-02-16 13:10:42,171 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-f2889d19) Unable to obtain GPU stats for host acs-compute-1
  533. 2016-02-16 13:10:42,173 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247577: Forwarding null to 57177340185274
  534. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 3-822751356925247577: Routing from 57177340185273
  535. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 3-822751356925247577: Link is closed
  536. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 3-822751356925247577: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  537. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  538. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  539. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-9:null) Seq 3-822751356925247577: Cancelling.
  540. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247577: Waiting some more time because this is the current command
  541. 2016-02-16 13:10:42,174 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247577: Waiting some more time because this is the current command
  542. 2016-02-16 13:10:42,174 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-f2889d19) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  543. 2016-02-16 13:10:42,175 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247577: Timed out on null
  544. 2016-02-16 13:10:42,175 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 3-822751356925247577: Cancelling.
  545. 2016-02-16 13:10:42,175 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-f2889d19) Operation timed out: Commands 822751356925247577 to Host 3 timed out after 3600
  546. 2016-02-16 13:10:42,175 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-f2889d19) Unable to obtain GPU stats for host acs-compute-2
  547. 2016-02-16 13:10:42,177 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318681: Forwarding null to 57177340185274
  548. 2016-02-16 13:10:42,177 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 8-4118541859230318681: Routing from 57177340185273
  549. 2016-02-16 13:10:42,177 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 8-4118541859230318681: Link is closed
  550. 2016-02-16 13:10:42,177 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) Seq 8-4118541859230318681: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  551. 2016-02-16 13:10:42,178 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  552. 2016-02-16 13:10:42,178 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  553. 2016-02-16 13:10:42,178 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) Seq 8-4118541859230318681: Cancelling.
  554. 2016-02-16 13:10:42,178 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318681: Waiting some more time because this is the current command
  555. 2016-02-16 13:10:42,178 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318681: Waiting some more time because this is the current command
  556. 2016-02-16 13:10:42,178 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-f2889d19) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  557. 2016-02-16 13:10:42,178 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318681: Timed out on null
  558. 2016-02-16 13:10:42,178 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-f2889d19) Seq 8-4118541859230318681: Cancelling.
  559. 2016-02-16 13:10:42,178 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-f2889d19) Operation timed out: Commands 4118541859230318681 to Host 8 timed out after 3600
  560. 2016-02-16 13:10:42,179 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-f2889d19) Unable to obtain GPU stats for host acs-compute-3
  561. 2016-02-16 13:10:43,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-25875a94) Zone 1 is not ready to launch console proxy yet
  562. 2016-02-16 13:10:43,584 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-491b921b) Zone 1 is not ready to launch secondary storage VM yet
  563. 2016-02-16 13:10:45,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-f41edcf3) Begin cleanup expired async-jobs
  564. 2016-02-16 13:10:45,199 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-f41edcf3) End cleanup expired async-jobs
  565. 2016-02-16 13:10:45,883 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-61f92883) VmStatsCollector is running...
  566. 2016-02-16 13:10:45,944 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-61f92883) Seq 1-7555351324867428442: Forwarding null to 57177340185274
  567. 2016-02-16 13:10:45,945 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-7555351324867428442: Routing from 57177340185273
  568. 2016-02-16 13:10:45,945 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-7555351324867428442: Link is closed
  569. 2016-02-16 13:10:45,945 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 1-7555351324867428442: MgmtId 57177340185273: Req: Resource [Host:1] is unreachable: Host 1: Link is closed
  570. 2016-02-16 13:10:45,945 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 1--1: MgmtId 57177340185273: Req: Routing to peer
  571. 2016-02-16 13:10:45,946 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) Seq 1--1: MgmtId 57177340185273: Req: Cancel request received
  572. 2016-02-16 13:10:45,946 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-13:null) Seq 1-7555351324867428442: Cancelling.
  573. 2016-02-16 13:10:45,946 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 1-7555351324867428442: Waiting some more time because this is the current command
  574. 2016-02-16 13:10:45,946 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 1-7555351324867428442: Waiting some more time because this is the current command
  575. 2016-02-16 13:10:45,946 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-61f92883) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  576. 2016-02-16 13:10:45,946 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 1-7555351324867428442: Timed out on null
  577. 2016-02-16 13:10:45,946 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 1-7555351324867428442: Cancelling.
  578. 2016-02-16 13:10:45,946 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-61f92883) Operation timed out: Commands 7555351324867428442 to Host 1 timed out after 3600
  579. 2016-02-16 13:10:45,946 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-61f92883) Unable to obtain VM statistics.
  580. 2016-02-16 13:10:45,992 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-61f92883) Seq 3-822751356925247578: Forwarding null to 57177340185274
  581. 2016-02-16 13:10:45,993 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 3-822751356925247578: Routing from 57177340185273
  582. 2016-02-16 13:10:45,993 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 3-822751356925247578: Link is closed
  583. 2016-02-16 13:10:45,993 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 3-822751356925247578: MgmtId 57177340185273: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  584. 2016-02-16 13:10:45,994 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) Seq 3--1: MgmtId 57177340185273: Req: Routing to peer
  585. 2016-02-16 13:10:45,994 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) Seq 3--1: MgmtId 57177340185273: Req: Cancel request received
  586. 2016-02-16 13:10:45,994 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) Seq 3-822751356925247578: Cancelling.
  587. 2016-02-16 13:10:45,994 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 3-822751356925247578: Waiting some more time because this is the current command
  588. 2016-02-16 13:10:45,994 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 3-822751356925247578: Waiting some more time because this is the current command
  589. 2016-02-16 13:10:45,994 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-61f92883) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  590. 2016-02-16 13:10:45,994 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 3-822751356925247578: Timed out on null
  591. 2016-02-16 13:10:45,995 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 3-822751356925247578: Cancelling.
  592. 2016-02-16 13:10:45,995 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-61f92883) Operation timed out: Commands 822751356925247578 to Host 3 timed out after 3600
  593. 2016-02-16 13:10:45,995 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-61f92883) Unable to obtain VM statistics.
  594. 2016-02-16 13:10:46,019 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-61f92883) Seq 8-4118541859230318682: Forwarding null to 57177340185274
  595. 2016-02-16 13:10:46,020 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 8-4118541859230318682: Routing from 57177340185273
  596. 2016-02-16 13:10:46,020 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 8-4118541859230318682: Link is closed
  597. 2016-02-16 13:10:46,020 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 8-4118541859230318682: MgmtId 57177340185273: Req: Resource [Host:8] is unreachable: Host 8: Link is closed
  598. 2016-02-16 13:10:46,021 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 8--1: MgmtId 57177340185273: Req: Routing to peer
  599. 2016-02-16 13:10:46,021 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 8--1: MgmtId 57177340185273: Req: Cancel request received
  600. 2016-02-16 13:10:46,021 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-5:null) Seq 8-4118541859230318682: Cancelling.
  601. 2016-02-16 13:10:46,021 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 8-4118541859230318682: Waiting some more time because this is the current command
  602. 2016-02-16 13:10:46,021 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 8-4118541859230318682: Waiting some more time because this is the current command
  603. 2016-02-16 13:10:46,021 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-61f92883) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  604. 2016-02-16 13:10:46,021 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 8-4118541859230318682: Timed out on null
  605. 2016-02-16 13:10:46,021 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-61f92883) Seq 8-4118541859230318682: Cancelling.
  606. 2016-02-16 13:10:46,022 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-61f92883) Operation timed out: Commands 4118541859230318682 to Host 8 timed out after 3600
  607. 2016-02-16 13:10:46,022 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-61f92883) Unable to obtain VM statistics.
  608. 2016-02-16 13:10:55,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9952e259) Begin cleanup expired async-jobs
  609. 2016-02-16 13:10:55,214 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9952e259) End cleanup expired async-jobs
  610. 2016-02-16 13:10:55,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-00049ad0) Found 12 routers to update status.
  611. 2016-02-16 13:10:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-00049ad0) Found 0 VPC networks to update Redundant State.
  612. 2016-02-16 13:10:55,586 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-00049ad0) Found 0 networks to update RvR status.
  613. 2016-02-16 13:10:55,680 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-b2be871d) Found 12 routers to update status.
  614. 2016-02-16 13:10:55,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-b2be871d) Found 0 VPC networks to update Redundant State.
  615. 2016-02-16 13:10:55,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-b2be871d) Found 0 networks to update RvR status.
  616. 2016-02-16 13:11:05,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6bd31adc) Begin cleanup expired async-jobs
  617. 2016-02-16 13:11:05,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6bd31adc) End cleanup expired async-jobs
  618. 2016-02-16 13:11:13,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-795034da) Zone 1 is not ready to launch console proxy yet
  619. 2016-02-16 13:11:13,585 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-114143a9) Zone 1 is not ready to launch secondary storage VM yet
  620. 2016-02-16 13:11:15,195 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-53851cd1) Begin cleanup expired async-jobs
  621. 2016-02-16 13:11:15,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-53851cd1) End cleanup expired async-jobs
  622. ^C
  623. root@acs-management-1:~# tail -f /var/log/cloudstack/management/management-server.log
  624. 2016-02-16 15:03:00,751 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) Seq 8-4118541859230319018: Cancelling.
  625. 2016-02-16 15:03:00,751 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-11fd7a14) Seq 8-4118541859230319018: Waiting some more time because this is the current command
  626. 2016-02-16 15:03:00,751 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-11fd7a14) Seq 8-4118541859230319018: Waiting some more time because this is the current command
  627. 2016-02-16 15:03:00,751 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-11fd7a14) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  628. 2016-02-16 15:03:00,751 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-11fd7a14) Seq 8-4118541859230319018: Timed out on null
  629. 2016-02-16 15:03:00,752 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-11fd7a14) Seq 8-4118541859230319018: Cancelling.
  630. 2016-02-16 15:03:00,752 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-11fd7a14) Operation timed out: Commands 4118541859230319018 to Host 8 timed out after 3600
  631. 2016-02-16 15:03:00,752 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-11fd7a14) Unable to obtain VM statistics.
  632. 2016-02-16 15:03:05,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a6d6133c) Begin cleanup expired async-jobs
  633. 2016-02-16 15:03:05,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a6d6133c) End cleanup expired async-jobs
  634. 2016-02-16 15:03:13,434 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-e9c502a5) Zone 1 is not ready to launch console proxy yet
  635. 2016-02-16 15:03:13,584 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-6030509f) Zone 1 is not ready to launch secondary storage VM yet
  636. 2016-02-16 15:03:15,196 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-62f1a89b) Begin cleanup expired async-jobs
  637. 2016-02-16 15:03:15,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-62f1a89b) End cleanup expired async-jobs
  638. 2016-02-16 15:03:25,195 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-dfffd706) Begin cleanup expired async-jobs
  639. 2016-02-16 15:03:25,198 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-dfffd706) End cleanup expired async-jobs
  640. 2016-02-16 15:03:25,584 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a6bda86a) Found 12 routers to update status.
  641. 2016-02-16 15:03:25,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a6bda86a) Found 0 VPC networks to update Redundant State.
  642. 2016-02-16 15:03:25,585 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a6bda86a) Found 0 networks to update RvR status.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!