API tools faq
paste
Guest User

Untitled

a guest
Sep 24th, 2013
129
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 96.42 KB | None | 0 0
raw download clone embed print report
  1. 2013-09-24 11:21:21,161 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) KVMInvestigator was able to determine host 13 is in Down
  2. 2013-09-24 11:21:21,161 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) The state determined is Down
  3. 2013-09-24 11:21:21,161 ERROR [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Host is down: 13-ad112.mydomain.net. Starting HA on the VMs
  4. 2013-09-24 11:21:21,161 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Host 13 is disconnecting with event HostDown
  5. 2013-09-24 11:21:21,162 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) The next status of agent 13is Down, current status is Up
  6. 2013-09-24 11:21:21,162 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Deregistering link for 13 with state Down
  7. 2013-09-24 11:21:21,162 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Remove Agent : 13
  8. 2013-09-24 11:21:21,162 DEBUG [agent.manager.ConnectedAgentAttache] (AgentTaskPool-1:null) Processing Disconnect.
  9. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492229: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  10. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492256: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  11. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 13-682492229: Waiting some more time because this is the current command
  12. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492255: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  13. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492256: Waiting some more time because this is the current command
  14. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492212: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  15. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 13-682492229: Waiting some more time because this is the current command
  16. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492253: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  17. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 13-682492212: Waiting some more time because this is the current command
  18. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 13-682492255: Waiting some more time because this is the current command
  19. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492256: Waiting some more time because this is the current command
  20. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 13-682492255: Waiting some more time because this is the current command
  21. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 13-682492212: Waiting some more time because this is the current command
  22. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492254: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  23. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492253: Waiting some more time because this is the current command
  24. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492254: Waiting some more time because this is the current command
  25. 2013-09-24 11:21:21,163 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  26. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492254: Waiting some more time because this is the current command
  27. 2013-09-24 11:21:21,164 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 13-682492229: Timed out on null
  28. 2013-09-24 11:21:21,164 DEBUG [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 13-682492229: Cancelling.
  29. 2013-09-24 11:21:21,164 DEBUG [cloud.storage.StorageManagerImpl] (StatsCollector-2:null) Unable to send storage pool command to Pool[6|NetworkFilesystem] via 13
  30. com.cloud.exception.OperationTimedoutException: Commands 682492229 to Host 13 timed out after 3600
  31. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
  32. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
  33. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439)
  34. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1061)
  35. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:441)
  36. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:455)
  37. at com.cloud.server.StatsCollector$StorageCollector.run(StatsCollector.java:562)
  38. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  39. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  40. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  41. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  42. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  43. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  44. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  45. at java.lang.Thread.run(Thread.java:679)
  46. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492253: Waiting some more time because this is the current command
  47. 2013-09-24 11:21:21,163 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492259: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  48. 2013-09-24 11:21:21,163 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  49. 2013-09-24 11:21:21,163 INFO [utils.exception.CSExceptionErrorCode] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  50. 2013-09-24 11:21:21,163 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-7:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  51. 2013-09-24 11:21:21,164 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 13-682492212: Timed out on null
  52. 2013-09-24 11:21:21,164 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492259: Waiting some more time because this is the current command
  53. 2013-09-24 11:21:21,164 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492251: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  54. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682491906: Sending disconnect to class com.cloud.network.security.SecurityGroupListener
  55. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492234: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  56. 2013-09-24 11:21:21,164 INFO [utils.exception.CSExceptionErrorCode] (RouterMonitor-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  57. 2013-09-24 11:21:21,164 INFO [utils.exception.CSExceptionErrorCode] (RouterMonitor-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  58. 2013-09-24 11:21:21,165 WARN [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492253: Timed out on null
  59. 2013-09-24 11:21:21,165 WARN [agent.manager.AgentAttache] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 13-682492255: Timed out on Seq 13-682492255: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":10,"srcIp":"185.15.184.209","protocol":"tcp","srcPortRange":[1,65535],"revoked":true,"alreadyAdded":false,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":13,"srcIp":"185.15.184.209","protocol":"icmp","revoked":true,"alreadyAdded":false,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","icmpType":-1,"icmpCode":-1,"trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"router.guest.ip":"10.1.1.166","firewall.egress.default":"false","zone.network.type":"Advanced","router.ip":"169.254.2.49","router.name":"r-10-VM"},"wait":0}}] }
  60. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-6:null) Seq 13-682492251: Waiting some more time because this is the current command
  61. 2013-09-24 11:21:21,165 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-6:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  62. 2013-09-24 11:21:21,165 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-6:null) Seq 13-682492251: Timed out on Seq 13-682492251: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.49","router.name":"r-10-VM"},"wait":30}}] }
  63. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492207: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  64. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (catalina-exec-4:null) Seq 13-682492234: Waiting some more time because this is the current command
  65. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492259: Waiting some more time because this is the current command
  66. 2013-09-24 11:21:21,165 DEBUG [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 13-682492212: Cancelling.
  67. 2013-09-24 11:21:21,165 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492256: Timed out on Seq 13-682492256: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.49","router.name":"r-10-VM"},"wait":30}}] }
  68. 2013-09-24 11:21:21,166 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-6:null) Seq 13-682492251: Cancelling.
  69. 2013-09-24 11:21:21,166 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 682492212 to Host 13 timed out after 3600
  70. 2013-09-24 11:21:21,166 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-8:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  71. 2013-09-24 11:21:21,166 DEBUG [agent.manager.AgentAttache] (catalina-exec-4:null) Seq 13-682492234: Waiting some more time because this is the current command
  72. 2013-09-24 11:21:21,166 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 13-682492207: Waiting some more time because this is the current command
  73. 2013-09-24 11:21:21,166 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492250: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  74. 2013-09-24 11:21:21,167 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-5:null) Seq 13-682492250: Waiting some more time because this is the current command
  75. 2013-09-24 11:21:21,165 WARN [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492254: Timed out on null
  76. 2013-09-24 11:21:21,167 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-5:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  77. 2013-09-24 11:21:21,167 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492252: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  78. 2013-09-24 11:21:21,167 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 13-682492207: Waiting some more time because this is the current command
  79. 2013-09-24 11:21:21,167 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492259: Timed out on Seq 13-682492259: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.49","router.name":"r-10-VM"},"wait":30}}] }
  80. 2013-09-24 11:21:21,167 INFO [utils.exception.CSExceptionErrorCode] (catalina-exec-4:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  81. 2013-09-24 11:21:21,167 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 13 statistics.
  82. 2013-09-24 11:21:21,167 DEBUG [agent.manager.AgentAttache] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 13-682492255: Cancelling.
  83. 2013-09-24 11:21:21,167 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-6:null) Operation timed out: Commands 682492251 to Host 13 timed out after 60
  84. 2013-09-24 11:21:21,168 WARN [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Timed Out
  85. com.cloud.exception.OperationTimedoutException: Commands 682492255 to Host 13 timed out after 3600
  86. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
  87. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
  88. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:1010)
  89. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3519)
  90. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendFirewallRules(VirtualNetworkApplianceManagerImpl.java:3752)
  91. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$8.execute(VirtualNetworkApplianceManagerImpl.java:3641)
  92. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3796)
  93. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3620)
  94. at com.cloud.network.element.VirtualRouterElement.applyFWRules(VirtualRouterElement.java:241)
  95. at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:554)
  96. at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:3226)
  97. at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:510)
  98. at com.cloud.network.firewall.FirewallManagerImpl.applyFirewallRules(FirewallManagerImpl.java:641)
  99. at com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:780)
  100. at com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1241)
  101. at com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1221)
  102. at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
  103. at org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd.execute(DisableStaticNatCmd.java:83)
  104. at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
  105. at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
  106. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  107. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  108. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  109. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  110. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  111. at java.lang.Thread.run(Thread.java:679)
  112. 2013-09-24 11:21:21,168 DEBUG [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492253: Cancelling.
  113. 2013-09-24 11:21:21,168 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 13
  114. 2013-09-24 11:21:21,168 WARN [agent.manager.AgentAttache] (catalina-exec-4:null) Seq 13-682492234: Timed out on Seq 13-682492234: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.GetVncPortCommand":{"id":10,"name":"r-10-VM","wait":0}}] }
  115. 2013-09-24 11:21:21,168 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  116. 2013-09-24 11:21:21,168 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-5:null) Seq 13-682492250: Timed out on Seq 13-682492250: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.129","router.name":"r-7-VM"},"wait":30}}] }
  117. 2013-09-24 11:21:21,168 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-2:null) Seq 13-682492252: Waiting some more time because this is the current command
  118. 2013-09-24 11:21:21,168 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492258: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  119. 2013-09-24 11:21:21,170 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-2:null) Seq 13-682492252: Waiting some more time because this is the current command
  120. 2013-09-24 11:21:21,169 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 13-682492207: Timed out on null
  121. 2013-09-24 11:21:21,169 WARN [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Unable to apply firewall rules on disconnected router r-10-VM
  122. com.cloud.exception.AgentUnavailableException: Resource [Host:13] is unreachable: Host 13: Unable to send commands to virtual router
  123. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3522)
  124. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendFirewallRules(VirtualNetworkApplianceManagerImpl.java:3752)
  125. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$8.execute(VirtualNetworkApplianceManagerImpl.java:3641)
  126. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3796)
  127. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3620)
  128. at com.cloud.network.element.VirtualRouterElement.applyFWRules(VirtualRouterElement.java:241)
  129. at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:554)
  130. at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:3226)
  131. at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:510)
  132. at com.cloud.network.firewall.FirewallManagerImpl.applyFirewallRules(FirewallManagerImpl.java:641)
  133. at com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:780)
  134. at com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1241)
  135. at com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1221)
  136. at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
  137. at org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd.execute(DisableStaticNatCmd.java:83)
  138. at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
  139. at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
  140. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  141. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  142. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  143. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  144. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  145. at java.lang.Thread.run(Thread.java:679)
  146. Caused by: com.cloud.exception.OperationTimedoutException: Commands 682492255 to Host 13 timed out after 3600
  147. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
  148. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
  149. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:1010)
  150. at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3519)
  151. ... 26 more
  152. 2013-09-24 11:21:21,169 WARN [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Operation timed out: Commands 682492253 to Host 13 timed out after 3600
  153. 2013-09-24 11:21:21,169 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492259: Cancelling.
  154. 2013-09-24 11:21:21,168 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-6:null) Unable to update router r-10-VM's status
  155. 2013-09-24 11:21:21,171 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-8:null) Operation timed out: Commands 682492259 to Host 13 timed out after 60
  156. 2013-09-24 11:21:21,171 DEBUG [agent.manager.AgentAttache] (RouterMonitor-1:null) Seq 13-682492254: Cancelling.
  157. 2013-09-24 11:21:21,171 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492256: Cancelling.
  158. 2013-09-24 11:21:21,171 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Applying firewall rules in network Ntwk[206|Guest|61]
  159. 2013-09-24 11:21:21,170 INFO [utils.exception.CSExceptionErrorCode] (AgentTaskPool-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  160. 2013-09-24 11:21:21,170 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492258: Waiting some more time because this is the current command
  161. 2013-09-24 11:21:21,170 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492249: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  162. 2013-09-24 11:21:21,172 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492258: Waiting some more time because this is the current command
  163. 2013-09-24 11:21:21,172 WARN [agent.manager.AgentAttache] (AgentTaskPool-2:null) Seq 13-682492252: Timed out on Seq 13-682492252: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckHealthCommand":{"wait":50}}] }
  164. 2013-09-24 11:21:21,171 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-7:null) Operation timed out: Commands 682492256 to Host 13 timed out after 60
  165. 2013-09-24 11:21:21,171 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 13-682492207: Cancelling.
  166. 2013-09-24 11:21:21,171 WARN [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Operation timed out: Commands 682492254 to Host 13 timed out after 3600
  167. 2013-09-24 11:21:21,171 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-8:null) Unable to update router r-10-VM's status
  168. 2013-09-24 11:21:21,172 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-5:null) Seq 13-682492250: Cancelling.
  169. 2013-09-24 11:21:21,172 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 682492207 to Host 13 timed out after 3600
  170. 2013-09-24 11:21:21,172 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-7:null) Unable to update router r-10-VM's status
  171. 2013-09-24 11:21:21,172 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-7:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  172. 2013-09-24 11:21:21,172 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492257: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  173. 2013-09-24 11:21:21,172 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-6:null) Seq 13-682492249: Waiting some more time because this is the current command
  174. 2013-09-24 11:21:21,173 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492258: Timed out on Seq 13-682492258: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.129","router.name":"r-7-VM"},"wait":30}}] }
  175. 2013-09-24 11:21:21,173 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 13-682492248: Sending disconnect to class com.cloud.agent.manager.SynchronousListener
  176. 2013-09-24 11:21:21,173 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492257: Waiting some more time because this is the current command
  177. 2013-09-24 11:21:21,172 WARN [cloud.vm.UserVmManagerImpl] (StatsCollector-3:null) Unable to obtain VM statistics.
  178. 2013-09-24 11:21:21,172 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-5:null) Operation timed out: Commands 682492250 to Host 13 timed out after 60
  179. 2013-09-24 11:21:21,172 DEBUG [agent.manager.AgentAttache] (catalina-exec-4:null) Seq 13-682492234: Cancelling.
  180. 2013-09-24 11:21:21,173 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-5:null) Unable to update router r-7-VM's status
  181. 2013-09-24 11:21:21,173 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492257: Waiting some more time because this is the current command
  182. 2013-09-24 11:21:21,173 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer_EnhancerByCloudStack_b48d7d49
  183. 2013-09-24 11:21:21,173 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-5:null) Seq 13-682492248: Waiting some more time because this is the current command
  184. 2013-09-24 11:21:21,173 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-6:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  185. 2013-09-24 11:21:21,174 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-5:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  186. 2013-09-24 11:21:21,174 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl_EnhancerByCloudStack_7b5207bf
  187. 2013-09-24 11:21:21,174 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl_EnhancerByCloudStack_8ce31952
  188. 2013-09-24 11:21:21,174 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
  189. 2013-09-24 11:21:21,174 INFO [utils.exception.CSExceptionErrorCode] (RedundantRouterStatusMonitor-8:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  190. 2013-09-24 11:21:21,174 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492257: Timed out on Seq 13-682492257: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.129","router.name":"r-7-VM"},"wait":30}}] }
  191. 2013-09-24 11:21:21,173 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-7:null) Seq 13-682492258: Cancelling.
  192. 2013-09-24 11:21:21,173 WARN [agent.manager.AgentManagerImpl] (catalina-exec-4:null) Operation timed out: Commands 682492234 to Host 13 timed out after 3600
  193. 2013-09-24 11:21:21,174 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-2:null) Seq 13-682492252: Cancelling.
  194. 2013-09-24 11:21:21,174 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-7:null) Operation timed out: Commands 682492258 to Host 13 timed out after 60
  195. 2013-09-24 11:21:21,174 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-5:null) Seq 13-682492248: Timed out on Seq 13-682492248: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.129","router.name":"r-7-VM"},"wait":30}}] }
  196. 2013-09-24 11:21:21,174 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
  197. 2013-09-24 11:21:21,174 WARN [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-6:null) Seq 13-682492249: Timed out on Seq 13-682492249: { Cmd , MgmtId: 159303462925110, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.49","router.name":"r-10-VM"},"wait":30}}] }
  198. 2013-09-24 11:21:21,175 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
  199. 2013-09-24 11:21:21,174 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-7:null) Unable to update router r-7-VM's status
  200. 2013-09-24 11:21:21,174 WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Operation timed out: Commands 682492252 to Host 13 timed out after 100
  201. 2013-09-24 11:21:21,174 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-8:null) Seq 13-682492257: Cancelling.
  202. 2013-09-24 11:21:21,174 DEBUG [cloud.servlet.ConsoleProxyServlet] (catalina-exec-4:null) Port info null
  203. 2013-09-24 11:21:21,176 INFO [cloud.servlet.ConsoleProxyServlet] (catalina-exec-4:null) Parse host info returned from executing GetVNCPortCommand. host info: null
  204. 2013-09-24 11:21:21,175 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-6:null) Seq 13-682492249: Cancelling.
  205. 2013-09-24 11:21:21,175 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-8:null) Operation timed out: Commands 682492257 to Host 13 timed out after 60
  206. 2013-09-24 11:21:21,175 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_b9a3ca56
  207. 2013-09-24 11:21:21,177 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
  208. 2013-09-24 11:21:21,177 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-8:null) Unable to update router r-7-VM's status
  209. 2013-09-24 11:21:21,177 DEBUG [agent.manager.AgentAttache] (RedundantRouterStatusMonitor-5:null) Seq 13-682492248: Cancelling.
  210. 2013-09-24 11:21:21,177 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-5:null) Operation timed out: Commands 682492248 to Host 13 timed out after 60
  211. 2013-09-24 11:21:21,177 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-5:null) Unable to update router r-7-VM's status
  212. 2013-09-24 11:21:21,177 WARN [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-6:null) Operation timed out: Commands 682492249 to Host 13 timed out after 60
  213. 2013-09-24 11:21:21,177 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-2:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  214. 2013-09-24 11:21:21,177 WARN [network.router.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-6:null) Unable to update router r-10-VM's status
  215. 2013-09-24 11:21:21,177 DEBUG [cloud.servlet.ConsoleProxyServlet] (catalina-exec-4:null) Compose console url: https://185-15-184-62.realhostip.com/ajax?token=DLSukproRDTNy4gODkVd28mpeBnj46uLsdr6495-HxTR2qR9aQsaA-_bHDpaHSSJSj70l_gZ6z9-BSTmpEfff_RxfRULgJ4L7mUknchGN3xC_crRqFi7m0enaqIlPoDoYbjZGo-aX1DgFy5YeeqZZ-WHX-pzoiZN9IPXzwPdBGbOA_uV_fHgl12IGui6JoytqFWaaTHOb2yKU1UUrkFohQ
  216. 2013-09-24 11:21:21,177 DEBUG [cloud.servlet.ConsoleProxyServlet] (catalina-exec-4:null) the console url is :: <html><title>r-10-VM</title><frameset><frame src="https://185-15-184-62.realhostip.com/ajax?token=DLSukproRDTNy4gODkVd28mpeBnj46uLsdr6495-HxTR2qR9aQsaA-_bHDpaHSSJSj70l_gZ6z9-BSTmpEfff_RxfRULgJ4L7mUknchGN3xC_crRqFi7m0enaqIlPoDoYbjZGo-aX1DgFy5YeeqZZ-WHX-pzoiZN9IPXzwPdBGbOA_uV_fHgl12IGui6JoytqFWaaTHOb2yKU1UUrkFohQ"></frame></frameset></html>
  217. 2013-09-24 11:21:21,177 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  218. 2013-09-24 11:21:21,177 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_531536f7
  219. 2013-09-24 11:21:21,177 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  220. 2013-09-24 11:21:21,178 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_cdf8ff1a
  221. 2013-09-24 11:21:21,178 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
  222. 2013-09-24 11:21:21,177 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-2:null) XenServerInvestigator unable to determine the state of the host. Moving on.
  223. 2013-09-24 11:21:21,178 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-2:null) checking if agent (13) is alive
  224. 2013-09-24 11:21:21,179 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
  225. 2013-09-24 11:21:21,179 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor
  226. 2013-09-24 11:21:21,179 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
  227. 2013-09-24 11:21:21,179 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
  228. 2013-09-24 11:21:21,179 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
  229. 2013-09-24 11:21:21,179 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentTaskPool-1:null) Disconnected called on 13 with status Down
  230. 2013-09-24 11:21:21,179 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
  231. 2013-09-24 11:21:21,179 DEBUG [cloud.host.Status] (AgentTaskPool-1:null) Transition:[Resource state = Enabled, Agent event = HostDown, Host id = 13, name = ad112.mydomain.net]
  232. 2013-09-24 11:21:21,181 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150870: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":10,"srcIp":"185.15.184.209","protocol":"tcp","srcPortRange":[1,65535],"revoked":true,"alreadyAdded":false,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":13,"srcIp":"185.15.184.209","protocol":"icmp","revoked":true,"alreadyAdded":false,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","icmpType":-1,"icmpCode":-1,"trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"router.guest.ip":"10.1.1.210","firewall.egress.default":"false","zone.network.type":"Advanced","router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":0}}] }
  233. 2013-09-24 11:21:21,183 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-2:null) sending ping from (2) to agent's host ip address (172.16.0.241)
  234. 2013-09-24 11:21:21,184 DEBUG [agent.transport.Request] (AgentTaskPool-2:null) Seq 2-2128150871: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.PingTestCommand":{"_computingHostIp":"172.16.0.241","wait":20}}] }
  235. 2013-09-24 11:21:21,220 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-6:null) Seq 2-2128150872: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":30}}] }
  236. 2013-09-24 11:21:21,221 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 2-2128150867: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  237. 2013-09-24 11:21:21,222 DEBUG [cloud.host.Status] (AgentTaskPool-1:null) Agent status update: [id = 13; name = ad112.mydomain.net; old status = Up; event = HostDown; new status = Down; old update count = 8; new update count = 9]
  238. 2013-09-24 11:21:21,222 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentTaskPool-1:null) Notifying other nodes of to disconnect
  239. 2013-09-24 11:21:21,227 WARN [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Scheduling restart for VMs on host 13
  240. 2013-09-24 11:21:21,229 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-5:null) Seq 2-2128150873: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.169","router.name":"r-8-VM"},"wait":30}}] }
  241. 2013-09-24 11:21:21,234 WARN [apache.cloudstack.alerts] (AgentTaskPool-1:null) alertType:: 7 // dataCenterId:: 2 // podId:: 2 // clusterId:: null // message:: Host is down, name: ad112.mydomain.net (id:13), availability zone: FR EQUINIX, pod: pod0
  242. 2013-09-24 11:21:21,235 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-8:null) Seq 2-2128150874: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":30}}] }
  243. 2013-09-24 11:21:21,245 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-7:null) Seq 2-2128150875: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.169","router.name":"r-8-VM"},"wait":30}}] }
  244. 2013-09-24 11:21:21,253 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-7:null) Seq 2-2128150876: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":30}}] }
  245. 2013-09-24 11:21:21,262 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-8:null) Seq 2-2128150877: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.169","router.name":"r-8-VM"},"wait":30}}] }
  246. 2013-09-24 11:21:21,264 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 5-test2
  247. 2013-09-24 11:21:21,270 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-6:null) Seq 2-2128150878: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":30}}] }
  248. 2013-09-24 11:21:21,280 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-5:null) Seq 2-2128150879: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.2.169","router.name":"r-8-VM"},"wait":30}}] }
  249. 2013-09-24 11:21:21,290 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[User|test2]
  250. 2013-09-24 11:21:21,290 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 6-ac0f8f8d-0603-46ce-8242-5b08ece29de2
  251. 2013-09-24 11:21:21,305 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-1) Processing HAWork[1-HA-5-Running-Investigating]
  252. 2013-09-24 11:21:21,309 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-1) HA on VM[User|test2]
  253. 2013-09-24 11:21:21,312 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-3:work-1) Unable to reach the agent for VM[User|test2]: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  254. 2013-09-24 11:21:21,312 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-1) SimpleInvestigator found VM[User|test2]to be alive? null
  255. 2013-09-24 11:21:21,312 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-1) XenServerInvestigator found VM[User|test2]to be alive? null
  256. 2013-09-24 11:21:21,313 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-3:work-1) testing if VM[User|test2] is alive
  257. 2013-09-24 11:21:21,319 DEBUG [agent.transport.Request] (HA-Worker-3:work-1) Seq 2-2128150880: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.PingTestCommand":{"_routerIp":"169.254.2.188","_privateIp":"10.1.1.185","wait":20}}] }
  258. 2013-09-24 11:21:21,323 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]
  259. 2013-09-24 11:21:21,323 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 7-r-7-VM
  260. 2013-09-24 11:21:21,339 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) Processing HAWork[4-HA-6-Running-Investigating]
  261. 2013-09-24 11:21:21,342 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) HA on VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]
  262. 2013-09-24 11:21:21,343 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-4:work-4) Unable to reach the agent for VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  263. 2013-09-24 11:21:21,343 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) SimpleInvestigator found VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]to be alive? null
  264. 2013-09-24 11:21:21,344 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) XenServerInvestigator found VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]to be alive? null
  265. 2013-09-24 11:21:21,344 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-4) testing if VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] is alive
  266. 2013-09-24 11:21:21,347 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-4:work-4) Can not send command com.cloud.agent.api.PingTestCommand due to Host 13 is not up
  267. 2013-09-24 11:21:21,347 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-4) VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] could not be pinged, returning that it is unknown
  268. 2013-09-24 11:21:21,348 DEBUG [agent.transport.Request] (HA-Worker-4:work-4) Seq 2-2128150881: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.PingTestCommand":{"_routerIp":"169.254.2.167","_privateIp":"10.1.1.121","wait":20}}] }
  269. 2013-09-24 11:21:21,348 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 2-2128150868: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-8-VM","bytesSent":0,"bytesReceived":0,"result":true,"details":"","wait":0}}] }
  270. 2013-09-24 11:21:21,348 DEBUG [agent.transport.Request] (RouterMonitor-1:null) Seq 2-2128150868: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
  271. 2013-09-24 11:21:21,349 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Details from executing class com.cloud.agent.api.NetworkUsageCommand:
  272. 2013-09-24 11:21:21,349 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Recieved and Sent bytes are both 0. Not updating user_statistics
  273. 2013-09-24 11:21:21,352 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Can not send command com.cloud.agent.api.NetworkUsageCommand due to Host 13 is not up
  274. 2013-09-24 11:21:21,356 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[DomainRouter|r-7-VM]
  275. 2013-09-24 11:21:21,356 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 10-r-10-VM
  276. 2013-09-24 11:21:21,372 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) Processing HAWork[7-HA-7-Running-Investigating]
  277. 2013-09-24 11:21:21,375 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) HA on VM[DomainRouter|r-7-VM]
  278. 2013-09-24 11:21:21,379 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-0:work-7) Unable to reach the agent for VM[DomainRouter|r-7-VM]: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  279. 2013-09-24 11:21:21,379 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) SimpleInvestigator found VM[DomainRouter|r-7-VM]to be alive? null
  280. 2013-09-24 11:21:21,379 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) XenServerInvestigator found VM[DomainRouter|r-7-VM]to be alive? null
  281. 2013-09-24 11:21:21,379 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-0:work-7) Not a User Vm, unable to determine state of VM[DomainRouter|r-7-VM] returning null
  282. 2013-09-24 11:21:21,379 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) null found VM[DomainRouter|r-7-VM]to be alive? null
  283. 2013-09-24 11:21:21,379 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-7) Testing if VM[DomainRouter|r-7-VM] is alive
  284. 2013-09-24 11:21:21,381 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-7) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[DomainRouter|r-7-VM] returning null
  285. 2013-09-24 11:21:21,381 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) null found VM[DomainRouter|r-7-VM]to be alive? null
  286. 2013-09-24 11:21:21,385 DEBUG [agent.transport.Request] (HA-Worker-0:work-7) Seq 2-2128150883: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","privateNetwork":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false}},"wait":20}}] }
  287. 2013-09-24 11:21:21,390 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[DomainRouter|r-10-VM]
  288. 2013-09-24 11:21:21,390 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 12-rrvm2
  289. 2013-09-24 11:21:21,405 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) Processing HAWork[10-HA-10-Running-Investigating]
  290. 2013-09-24 11:21:21,408 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) HA on VM[DomainRouter|r-10-VM]
  291. 2013-09-24 11:21:21,411 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-2:work-10) Unable to reach the agent for VM[DomainRouter|r-10-VM]: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  292. 2013-09-24 11:21:21,411 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) SimpleInvestigator found VM[DomainRouter|r-10-VM]to be alive? null
  293. 2013-09-24 11:21:21,411 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) XenServerInvestigator found VM[DomainRouter|r-10-VM]to be alive? null
  294. 2013-09-24 11:21:21,411 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:work-10) Not a User Vm, unable to determine state of VM[DomainRouter|r-10-VM] returning null
  295. 2013-09-24 11:21:21,411 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) null found VM[DomainRouter|r-10-VM]to be alive? null
  296. 2013-09-24 11:21:21,411 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:work-10) Testing if VM[DomainRouter|r-10-VM] is alive
  297. 2013-09-24 11:21:21,413 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:work-10) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[DomainRouter|r-10-VM] returning null
  298. 2013-09-24 11:21:21,413 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) null found VM[DomainRouter|r-10-VM]to be alive? null
  299. 2013-09-24 11:21:21,415 DEBUG [agent.transport.Request] (HA-Worker-2:work-10) Seq 2-2128150884: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","privateNetwork":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false}},"wait":20}}] }
  300. 2013-09-24 11:21:21,416 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 2-2128150870: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 0, [{"com.cloud.agent.api.routing.SetFirewallRulesAnswer":{"result":true,"wait":0}}] }
  301. 2013-09-24 11:21:21,416 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150870: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 0, { SetFirewallRulesAnswer } }
  302. 2013-09-24 11:21:21,416 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) About to stop the router r-10-VM due to: Unable to apply firewall rules on disconnected router
  303. 2013-09-24 11:21:21,416 WARN [apache.cloudstack.alerts] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) alertType:: 9 // dataCenterId:: 2 // podId:: 2 // clusterId:: null // message:: Virtual router r-10-VM would be stopped after connecting back, due to Unable to apply firewall rules on disconnected router
  304. 2013-09-24 11:21:21,439 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[User|rrvm2]
  305. 2013-09-24 11:21:21,440 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentTaskPool-1:null) Notifying other nodes of to disconnect
  306. 2013-09-24 11:21:21,472 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-13) Processing HAWork[13-HA-12-Running-Investigating]
  307. 2013-09-24 11:21:21,474 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-13) HA on VM[User|rrvm2]
  308. 2013-09-24 11:21:21,476 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-1:work-13) Unable to reach the agent for VM[User|rrvm2]: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  309. 2013-09-24 11:21:21,476 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-13) SimpleInvestigator found VM[User|rrvm2]to be alive? null
  310. 2013-09-24 11:21:21,476 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-13) XenServerInvestigator found VM[User|rrvm2]to be alive? null
  311. 2013-09-24 11:21:21,476 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-1:work-13) testing if VM[User|rrvm2] is alive
  312. 2013-09-24 11:21:21,478 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq 2-2128150872: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  313. 2013-09-24 11:21:21,478 DEBUG [agent.transport.Request] (AgentManager-Handler-14:null) Seq 2-2128150873: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  314. 2013-09-24 11:21:21,478 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-6:null) Seq 2-2128150872: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  315. 2013-09-24 11:21:21,478 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-5:null) Seq 2-2128150873: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  316. 2013-09-24 11:21:21,478 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 2-2128150869: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-8-VM","bytesSent":0,"bytesReceived":0,"result":true,"details":"","wait":0}}] }
  317. 2013-09-24 11:21:21,478 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-5:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  318. 2013-09-24 11:21:21,478 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-6:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  319. 2013-09-24 11:21:21,478 DEBUG [agent.transport.Request] (RouterMonitor-1:null) Seq 2-2128150869: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
  320. 2013-09-24 11:21:21,478 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Details from executing class com.cloud.agent.api.NetworkUsageCommand:
  321. 2013-09-24 11:21:21,478 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Recieved and Sent bytes are both 0. Not updating user_statistics
  322. 2013-09-24 11:21:21,482 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-1:work-13) Can not send command com.cloud.agent.api.PingTestCommand due to Host 13 is not up
  323. 2013-09-24 11:21:21,482 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-1:work-13) VM[User|rrvm2] could not be pinged, returning that it is unknown
  324. 2013-09-24 11:21:21,484 DEBUG [agent.transport.Request] (HA-Worker-1:work-13) Seq 2-2128150885: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.PingTestCommand":{"_routerIp":"169.254.2.167","_privateIp":"10.1.1.84","wait":20}}] }
  325. 2013-09-24 11:21:21,484 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Can not send command com.cloud.agent.api.NetworkUsageCommand due to Host 13 is not up
  326. 2013-09-24 11:21:21,503 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150887: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.BumpUpPriorityCommand":{"accessDetails":{"router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":0}}] }
  327. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 2-2128150874: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  328. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-8:null) Seq 2-2128150874: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  329. 2013-09-24 11:21:21,566 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-8:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  330. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 2-2128150875: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  331. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 2-2128150876: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  332. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-7:null) Seq 2-2128150875: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  333. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-7:null) Seq 2-2128150876: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  334. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 2-2128150877: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  335. 2013-09-24 11:21:21,566 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-7:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  336. 2013-09-24 11:21:21,566 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-7:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  337. 2013-09-24 11:21:21,566 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-8:null) Seq 2-2128150877: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  338. 2013-09-24 11:21:21,566 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-8:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  339. 2013-09-24 11:21:21,634 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq 2-2128150878: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  340. 2013-09-24 11:21:21,634 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-6:null) Seq 2-2128150878: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  341. 2013-09-24 11:21:21,634 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-6:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  342. 2013-09-24 11:21:21,644 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 2-2128150879: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status: MASTER&Bumped: NO","wait":0}}] }
  343. 2013-09-24 11:21:21,644 DEBUG [agent.transport.Request] (RedundantRouterStatusMonitor-5:null) Seq 2-2128150879: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { CheckRouterAnswer } }
  344. 2013-09-24 11:21:21,644 DEBUG [agent.manager.AgentManagerImpl] (RedundantRouterStatusMonitor-5:null) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
  345. 2013-09-24 11:21:21,674 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq 2-2128150883: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  346. 2013-09-24 11:21:21,674 DEBUG [agent.transport.Request] (HA-Worker-0:work-7) Seq 2-2128150883: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  347. 2013-09-24 11:21:21,674 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) KVMInvestigator found VM[DomainRouter|r-7-VM]to be alive? null
  348. 2013-09-24 11:21:21,674 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) Fencing off VM that we don't know the state of
  349. 2013-09-24 11:21:21,674 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-0:work-7) Don't know how to fence non XenServer hosts KVM
  350. 2013-09-24 11:21:21,675 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) Fencer null returned null
  351. 2013-09-24 11:21:21,677 DEBUG [agent.transport.Request] (HA-Worker-0:work-7) Seq 2-2128150888: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"r-7-VM","hostGuid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","hostIp":"172.16.0.241","inSeq":false,"wait":0}}] }
  352. 2013-09-24 11:21:21,716 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 2-2128150884: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  353. 2013-09-24 11:21:21,716 DEBUG [agent.transport.Request] (HA-Worker-2:work-10) Seq 2-2128150884: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  354. 2013-09-24 11:21:21,716 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) KVMInvestigator found VM[DomainRouter|r-10-VM]to be alive? null
  355. 2013-09-24 11:21:21,716 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) Fencing off VM that we don't know the state of
  356. 2013-09-24 11:21:21,716 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-2:work-10) Don't know how to fence non XenServer hosts KVM
  357. 2013-09-24 11:21:21,717 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) Fencer null returned null
  358. 2013-09-24 11:21:21,719 DEBUG [agent.transport.Request] (HA-Worker-2:work-10) Seq 2-2128150889: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"r-10-VM","hostGuid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","hostIp":"172.16.0.241","inSeq":false,"wait":0}}] }
  359. 2013-09-24 11:21:21,809 DEBUG [agent.transport.Request] (AgentManager-Handler-6:null) Seq 2-2128150882: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-11-VM","bytesSent":20216,"bytesReceived":53492,"result":true,"details":"","wait":0}}] }
  360. 2013-09-24 11:21:21,809 DEBUG [agent.transport.Request] (RouterMonitor-1:null) Seq 2-2128150882: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
  361. 2013-09-24 11:21:21,810 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Details from executing class com.cloud.agent.api.NetworkUsageCommand:
  362. 2013-09-24 11:21:21,897 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 2-2128150886: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-11-VM","bytesSent":20216,"bytesReceived":53492,"result":true,"details":"","wait":0}}] }
  363. 2013-09-24 11:21:21,897 DEBUG [agent.transport.Request] (RouterMonitor-1:null) Seq 2-2128150886: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
  364. 2013-09-24 11:21:21,897 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Details from executing class com.cloud.agent.api.NetworkUsageCommand:
  365. 2013-09-24 11:21:21,898 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Router stats changed from the time NetworkUsageCommand was sent. Ignoring current answer. Router: r-11-VM Rcvd: 53492Sent: 20216
  366. 2013-09-24 11:21:21,898 DEBUG [db.Transaction.Transaction] (RouterMonitor-1:null) Rolling back the transaction: Time = 1 Name = -VirtualNetworkApplianceManagerImpl$NetworkUsageTask.run:900-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRunAndReset:351-FutureTask.runAndReset:178-ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201:165-ScheduledThreadPoolExecutor$ScheduledFutureTask.run:267-ThreadPoolExecutor.runWorker:1146-ThreadPoolExecutor$Worker.run:615-Thread.run:679; called by -Transaction.rollback:898-Transaction.removeUpTo:841-Transaction.close:665-VirtualNetworkApplianceManagerImpl$NetworkUsageTask.run:955-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRunAndReset:351-FutureTask.runAndReset:178-ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201:165-ScheduledThreadPoolExecutor$ScheduledFutureTask.run:267-ThreadPoolExecutor.runWorker:1146-ThreadPoolExecutor$Worker.run:615-Thread.run:679
  367. 2013-09-24 11:21:22,004 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 2-2128150887: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  368. 2013-09-24 11:21:22,004 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150887: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  369. 2013-09-24 11:21:22,055 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 2-2128150888: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.FenceAnswer":{"result":true,"wait":0}}] }
  370. 2013-09-24 11:21:22,055 DEBUG [agent.transport.Request] (HA-Worker-0:work-7) Seq 2-2128150888: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { FenceAnswer } }
  371. 2013-09-24 11:21:22,055 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) Fencer KVMFenceBuilder returned true
  372. 2013-09-24 11:21:22,066 DEBUG [network.firewall.FirewallManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Successfully released firewall rules for ip id=830 and # of rules now = 0
  373. 2013-09-24 11:21:22,067 DEBUG [network.rules.RulesManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Releasing 0 port forwarding rules for ip id=830
  374. 2013-09-24 11:21:22,067 DEBUG [network.rules.RulesManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Releasing 0 static nat rules for ip id=830
  375. 2013-09-24 11:21:22,068 DEBUG [network.rules.RulesManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) There are no port forwarding rules to apply for ip id=830
  376. 2013-09-24 11:21:22,069 DEBUG [network.rules.RulesManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) There are no static nat rules to apply for ip id=830
  377. 2013-09-24 11:21:22,077 INFO [cloud.network.NetworkManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Let VirtualRouter handle StaticNat in network 206
  378. 2013-09-24 11:21:22,080 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-7) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 13 new host id: 13 host id before state transition: 13
  379. 2013-09-24 11:21:22,083 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Applying static nat rules in network Ntwk[206|Guest|61]
  380. 2013-09-24 11:21:22,084 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Router r-10-VM is stop pending, so not sending apply static nat rules commands to the backend
  381. 2013-09-24 11:21:22,084 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Applying static nat rules in network Ntwk[206|Guest|61]
  382. 2013-09-24 11:21:22,088 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-0:work-7) Can not send command com.cloud.agent.api.NetworkUsageCommand due to Host 13 is not up
  383. 2013-09-24 11:21:22,091 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150890: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.1.1.121","id":0,"srcIp":"185.15.184.209","revoked":true,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0,"defaultEgressPolicy":false}],"accessDetails":{"router.guest.ip":"10.1.1.210","zone.network.type":"Advanced","router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":0}}] }
  384. 2013-09-24 11:21:22,092 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-7) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  385. 2013-09-24 11:21:22,092 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-7) Unable to actually stop VM[DomainRouter|r-7-VM] but continue with release because it's a force stop
  386. 2013-09-24 11:21:22,092 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-7) VM[DomainRouter|r-7-VM] is stopped on the host. Proceeding to release resource held.
  387. 2013-09-24 11:21:22,097 DEBUG [cloud.network.NetworkModelImpl] (HA-Worker-0:work-7) Service SecurityGroup is not supported in the network id=205
  388. 2013-09-24 11:21:22,108 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq 2-2128150889: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.FenceAnswer":{"result":true,"wait":0}}] }
  389. 2013-09-24 11:21:22,108 DEBUG [agent.transport.Request] (HA-Worker-2:work-10) Seq 2-2128150889: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { FenceAnswer } }
  390. 2013-09-24 11:21:22,108 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) Fencer KVMFenceBuilder returned true
  391. 2013-09-24 11:21:22,117 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-0:work-7) Asking VirtualRouter to release Nic[49-7-4b4ba8a8-f16c-4210-8e1d-8d5eadaa528a-10.1.1.246]
  392. 2013-09-24 11:21:22,121 DEBUG [network.guru.ControlNetworkGuru] (HA-Worker-0:work-7) Released nic: NicProfile[52-7-null-null-null
  393. 2013-09-24 11:21:22,130 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-2:work-10) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 13 new host id: 13 host id before state transition: 13
  394. 2013-09-24 11:21:22,136 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-2:work-10) Can not send command com.cloud.agent.api.NetworkUsageCommand due to Host 13 is not up
  395. 2013-09-24 11:21:22,139 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-2:work-10) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  396. 2013-09-24 11:21:22,139 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-2:work-10) Unable to actually stop VM[DomainRouter|r-10-VM] but continue with release because it's a force stop
  397. 2013-09-24 11:21:22,140 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-2:work-10) VM[DomainRouter|r-10-VM] is stopped on the host. Proceeding to release resource held.
  398. 2013-09-24 11:21:22,143 DEBUG [cloud.network.NetworkModelImpl] (HA-Worker-2:work-10) Service SecurityGroup is not supported in the network id=206
  399. 2013-09-24 11:21:22,166 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-2:work-10) Asking VirtualRouter to release Nic[70-10-6cb295d2-fac5-4a6b-b10b-ddc13e737642-10.1.1.166]
  400. 2013-09-24 11:21:22,169 DEBUG [network.guru.ControlNetworkGuru] (HA-Worker-2:work-10) Released nic: NicProfile[73-10-null-null-null
  401. 2013-09-24 11:21:22,180 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-7) Successfully released network resources for the vm VM[DomainRouter|r-7-VM]
  402. 2013-09-24 11:21:22,181 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-7) Successfully released storage resources for the vm VM[DomainRouter|r-7-VM]
  403. 2013-09-24 11:21:22,214 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-7) VM state transitted from :Stopping to Stopped with event: OperationSucceededvm's original host id: 13 new host id: null host id before state transition: 13
  404. 2013-09-24 11:21:22,220 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-7) Hosts's actual total CPU: 83232 and CPU after applying overprovisioning: 83232
  405. 2013-09-24 11:21:22,220 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-7) Hosts's actual total RAM: 270846586880 and RAM after applying overprovisioning: 270846590976
  406. 2013-09-24 11:21:22,220 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-7) release cpu from host: 13, old used: 6500,reserved: 0, actual total: 83232, total with overprovisioning: 83232; new used: 6000,reserved:500; movedfromreserved: false,moveToReserveredtrue
  407. 2013-09-24 11:21:22,220 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-7) release mem from host: 13, old used: 2952790016,reserved: 0, total: 270846590976; new used: 2818572288,reserved:134217728; movedfromreserved: false,moveToReserveredtrue
  408. 2013-09-24 11:21:22,230 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-2:work-10) Successfully released network resources for the vm VM[DomainRouter|r-10-VM]
  409. 2013-09-24 11:21:22,230 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-2:work-10) Successfully released storage resources for the vm VM[DomainRouter|r-10-VM]
  410. 2013-09-24 11:21:22,264 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-2:work-10) VM state transitted from :Stopping to Stopped with event: OperationSucceededvm's original host id: 13 new host id: null host id before state transition: 13
  411. 2013-09-24 11:21:22,267 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-2:work-10) Hosts's actual total CPU: 83232 and CPU after applying overprovisioning: 83232
  412. 2013-09-24 11:21:22,267 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-2:work-10) Hosts's actual total RAM: 270846586880 and RAM after applying overprovisioning: 270846590976
  413. 2013-09-24 11:21:22,267 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-2:work-10) release cpu from host: 13, old used: 6000,reserved: 500, actual total: 83232, total with overprovisioning: 83232; new used: 5500,reserved:1000; movedfromreserved: false,moveToReserveredtrue
  414. 2013-09-24 11:21:22,267 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-2:work-10) release mem from host: 13, old used: 2818572288,reserved: 134217728, total: 270846590976; new used: 2684354560,reserved:268435456; movedfromreserved: false,moveToReserveredtrue
  415. 2013-09-24 11:21:22,281 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) VM is not HA enabled so we're done.
  416. 2013-09-24 11:21:22,281 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-7) Completed HAWork[7-HA-7-Running-Scheduled]
  417. 2013-09-24 11:21:22,287 DEBUG [agent.transport.Request] (AgentManager-Handler-14:null) Seq 2-2128150890: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 0, [{"com.cloud.agent.api.routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}] }
  418. 2013-09-24 11:21:22,288 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150890: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 0, { SetStaticNatRulesAnswer } }
  419. 2013-09-24 11:21:22,295 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Router r-10-VM is in Stopped, so not sending apply ip association commands to the backend
  420. 2013-09-24 11:21:22,295 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Applying ip association in network Ntwk[206|Guest|61]
  421. 2013-09-24 11:21:22,308 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150891: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100001, [{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":2,"publicIp":"185.15.184.72","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"221","vlanGateway":"185.15.184.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:ee:c6:00:01:35","networkRate":200,"trafficType":"Public","networkName":"cloudbrpublic"},{"accountId":2,"publicIp":"185.15.184.209","sourceNat":false,"add":false,"oneToOneNat":true,"firstIP":false,"vlanId":"221","vlanGateway":"185.15.184.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:92:ba:00:01:35","networkRate":200,"trafficType":"Public","networkName":"cloudbrpublic"}],"accessDetails":{"router.guest.ip":"10.1.1.210","zone.network.type":"Advanced","router.ip":"169.254.2.167","router.name":"r-11-VM"},"wait":0}}] }
  422. 2013-09-24 11:21:22,331 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) VM is not HA enabled so we're done.
  423. 2013-09-24 11:21:22,332 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-10) Completed HAWork[10-HA-10-Running-Scheduled]
  424. 2013-09-24 11:21:23,167 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 2-2128150891: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 0, [{"com.cloud.agent.api.routing.IpAssocAnswer":{"results":["185.15.184.72 - success","185.15.184.209 - success"],"result":true,"wait":0}}] }
  425. 2013-09-24 11:21:23,167 DEBUG [agent.transport.Request] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Seq 2-2128150891: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 0, { IpAssocAnswer } }
  426. 2013-09-24 11:21:23,168 DEBUG [network.rules.RulesManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Successfully released rules for ip id=830 and # of rules now = 0
  427. 2013-09-24 11:21:23,248 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Complete async job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ], jobStatus: 1, resultCode: 0, result: org.apache.cloudstack.api.response.SuccessResponse@54792174
  428. 2013-09-24 11:21:23,281 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Sync queue (4) is currently empty
  429. 2013-09-24 11:21:23,282 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Done executing org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd for job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]
  430. 2013-09-24 11:21:23,282 WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-25:job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ]) Unable to unregister active job [ 148 ] = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ] from JMX monitoring
  431. 2013-09-24 11:21:24,218 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 2-2128150871: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"Unable to ping computing host, exiting","wait":0}}] }
  432. 2013-09-24 11:21:24,218 DEBUG [agent.transport.Request] (AgentTaskPool-2:null) Seq 2-2128150871: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  433. 2013-09-24 11:21:24,218 DEBUG [cloud.ha.AbstractInvestigatorImpl] (AgentTaskPool-2:null) host (172.16.0.241) cannot be pinged, returning null ('I don't know')
  434. 2013-09-24 11:21:24,219 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-2:null) could not reach agent, could not reach agent's host, returning that we don't have enough information
  435. 2013-09-24 11:21:24,219 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-2:null) null unable to determine the state of the host. Moving on.
  436. 2013-09-24 11:21:24,220 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-2:null) null unable to determine the state of the host. Moving on.
  437. 2013-09-24 11:21:24,224 DEBUG [agent.transport.Request] (AgentTaskPool-2:null) Seq 2-2128150892: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","privateNetwork":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false}},"wait":20}}] }
  438. 2013-09-24 11:21:24,313 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 7-872: Processing Seq 7-872: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": [\n {\n \"id\": 6,\n \"clientInfo\": \"\",\n \"host\": \"172.16.0.240\",\n \"port\": 5906,\n \"tag\": \"d3725c9c-8632-4208-b43e-13fbec488f2a\",\n \"createTime\": 1380021508368,\n \"lastUsedTime\": 1380021682738\n },\n {\n \"id\": 7,\n \"clientInfo\": \"\",\n \"host\": \"172.16.0.240\",\n \"port\": 5905,\n \"tag\": \"504ea5f2-9302-4626-b9f7-c99baaa7921d\",\n \"createTime\": 1380021531048,\n \"lastUsedTime\": 1380021557611\n }\n ]\n}","wait":0}}] }
  439. 2013-09-24 11:21:24,316 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 2-2128150892: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  440. 2013-09-24 11:21:24,316 DEBUG [agent.transport.Request] (AgentTaskPool-2:null) Seq 2-2128150892: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  441. 2013-09-24 11:21:24,316 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-2:null) KVMInvestigator was able to determine host 13 is in Down
  442. 2013-09-24 11:21:24,316 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) The state determined is Down
  443. 2013-09-24 11:21:24,317 ERROR [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Host is down: 13-ad112.mydomain.net. Starting HA on the VMs
  444. 2013-09-24 11:21:24,317 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Host 13 is disconnecting with event HostDown
  445. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Host 13 is already Down
  446. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Deregistering link for 13 with state Down
  447. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Remove Agent : 13
  448. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer_EnhancerByCloudStack_b48d7d49
  449. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl_EnhancerByCloudStack_7b5207bf
  450. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl_EnhancerByCloudStack_8ce31952
  451. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
  452. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
  453. 2013-09-24 11:21:24,318 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
  454. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_b9a3ca56
  455. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
  456. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  457. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_531536f7
  458. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  459. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_cdf8ff1a
  460. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
  461. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
  462. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor
  463. 2013-09-24 11:21:24,319 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
  464. 2013-09-24 11:21:24,320 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
  465. 2013-09-24 11:21:24,320 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
  466. 2013-09-24 11:21:24,320 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentTaskPool-2:null) Disconnected called on 13 with status Down
  467. 2013-09-24 11:21:24,320 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
  468. 2013-09-24 11:21:24,320 DEBUG [cloud.host.Status] (AgentTaskPool-2:null) Transition:[Resource state = Enabled, Agent event = HostDown, Host id = 13, name = ad112.mydomain.net]
  469. 2013-09-24 11:21:24,320 DEBUG [cloud.host.Status] (AgentTaskPool-2:null) Cannot transit agent status with event HostDown for host 13, name=ad112.mydomain.net, mangement server id is 159303462925110
  470. 2013-09-24 11:21:24,320 ERROR [agent.manager.AgentManagerImpl] (AgentTaskPool-2:null) Exception caught while handling disconnect:
  471. com.cloud.utils.exception.CloudRuntimeException: Cannot transit agent status with event HostDown for host 13, mangement server id is 159303462925110,Unable to transition to a new state from Down via HostDown
  472. at com.cloud.agent.manager.AgentManagerImpl.agentStatusTransitTo(AgentManagerImpl.java:1410)
  473. at com.cloud.agent.manager.AgentManagerImpl.disconnectAgent(AgentManagerImpl.java:1425)
  474. at com.cloud.agent.manager.AgentManagerImpl.handleDisconnectWithoutInvestigation(AgentManagerImpl.java:856)
  475. at com.cloud.agent.manager.ClusteredAgentManagerImpl.handleDisconnect(ClusteredAgentManagerImpl.java:298)
  476. at com.cloud.agent.manager.ClusteredAgentManagerImpl.handleDisconnectWithoutInvestigation(ClusteredAgentManagerImpl.java:287)
  477. at com.cloud.agent.manager.AgentManagerImpl.handleDisconnectWithInvestigation(AgentManagerImpl.java:931)
  478. at com.cloud.agent.manager.ClusteredAgentManagerImpl.handleDisconnect(ClusteredAgentManagerImpl.java:300)
  479. at com.cloud.agent.manager.ClusteredAgentManagerImpl.handleDisconnectWithInvestigation(ClusteredAgentManagerImpl.java:292)
  480. at com.cloud.agent.manager.AgentManagerImpl$DisconnectTask.run(AgentManagerImpl.java:955)
  481. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  482. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  483. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  484. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  485. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  486. at java.lang.Thread.run(Thread.java:679)
  487. 2013-09-24 11:21:24,377 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 7-872: Sending Seq 7-872: { Ans: , MgmtId: 159303462925110, via: 7, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
  488. 2013-09-24 11:21:24,927 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-21:null) Async job-148 = [ 7e115e1a-2cfc-4c4c-a0eb-b115a16cae6c ] completed
  489. 2013-09-24 11:21:25,140 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 2-2128150881: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"PING 10.1.1.121 (10.1.1.121): 56 data bytes64 bytes from 10.1.1.210: Destination Host UnreachableVr HL TOS Len ID Flg off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 5d23 10.1.1.210 10.1.1.121 --- 10.1.1.121 ping statistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable to ping the vm, exiting","wait":0}}] }
  490. 2013-09-24 11:21:25,140 DEBUG [agent.transport.Request] (HA-Worker-4:work-4) Seq 2-2128150881: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  491. 2013-09-24 11:21:25,140 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-4:work-4) Details from executing class com.cloud.agent.api.PingTestCommand: PING 10.1.1.121 (10.1.1.121): 56 data bytes64 bytes from 10.1.1.210: Destination Host UnreachableVr HL TOS Len ID Flg off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 5d23 10.1.1.210 10.1.1.121 --- 10.1.1.121 ping statistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable to ping the vm, exiting
  492. 2013-09-24 11:21:25,140 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-4) VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] could not be pinged, returning that it is unknown
  493. 2013-09-24 11:21:25,140 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-4) Returning null since we're unable to determine state of VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]
  494. 2013-09-24 11:21:25,140 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) null found VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]to be alive? null
  495. 2013-09-24 11:21:25,140 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-4:work-4) Not a System Vm, unable to determine state of VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] returning null
  496. 2013-09-24 11:21:25,140 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-4:work-4) Testing if VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] is alive
  497. 2013-09-24 11:21:25,142 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-4:work-4) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] returning null
  498. 2013-09-24 11:21:25,142 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) null found VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]to be alive? null
  499. 2013-09-24 11:21:25,144 DEBUG [agent.transport.Request] (HA-Worker-4:work-4) Seq 2-2128150893: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","privateNetwork":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"172.16.0.241","netmask":"255.255.255.0","mac":"90:e2:ba:39:f3:f6","isSecurityGroupEnabled":false}},"wait":20}}] }
  500. 2013-09-24 11:21:25,233 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq 2-2128150893: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  501. 2013-09-24 11:21:25,233 DEBUG [agent.transport.Request] (HA-Worker-4:work-4) Seq 2-2128150893: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { Answer } }
  502. 2013-09-24 11:21:25,233 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) KVMInvestigator found VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]to be alive? null
  503. 2013-09-24 11:21:25,233 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) Fencing off VM that we don't know the state of
  504. 2013-09-24 11:21:25,233 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-4:work-4) Don't know how to fence non XenServer hosts KVM
  505. 2013-09-24 11:21:25,234 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) Fencer null returned null
  506. 2013-09-24 11:21:25,236 DEBUG [agent.transport.Request] (HA-Worker-4:work-4) Seq 2-2128150894: Sending { Cmd , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-6-VM","hostGuid":"8e0e92ae-81b5-390b-a48c-3f76c0df89cd-LibvirtComputingResource","hostIp":"172.16.0.241","inSeq":false,"wait":0}}] }
  507. 2013-09-24 11:21:25,326 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 2-2128150894: Processing: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.FenceAnswer":{"result":true,"wait":0}}] }
  508. 2013-09-24 11:21:25,326 DEBUG [agent.transport.Request] (HA-Worker-4:work-4) Seq 2-2128150894: Received: { Ans: , MgmtId: 159303462925110, via: 2, Ver: v1, Flags: 10, { FenceAnswer } }
  509. 2013-09-24 11:21:25,326 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) Fencer KVMFenceBuilder returned true
  510. 2013-09-24 11:21:25,347 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-4) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 13 new host id: 13 host id before state transition: 13
  511. 2013-09-24 11:21:25,349 DEBUG [cloud.vm.UserVmManagerImpl] (HA-Worker-4:work-4) Collect vm disk statistics from host before stopping Vm
  512. 2013-09-24 11:21:25,351 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-4:work-4) Can not send command com.cloud.agent.api.GetVmDiskStatsCommand due to Host 13 is not up
  513. 2013-09-24 11:21:25,353 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-4) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:13] is unreachable: Host 13: Host with specified id is not in the right state: Down
  514. 2013-09-24 11:21:25,353 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-4) Unable to actually stop VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] but continue with release because it's a force stop
  515. 2013-09-24 11:21:25,355 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-4) VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2] is stopped on the host. Proceeding to release resource held.
  516. 2013-09-24 11:21:25,359 DEBUG [cloud.network.NetworkModelImpl] (HA-Worker-4:work-4) Service SecurityGroup is not supported in the network id=206
  517. 2013-09-24 11:21:25,361 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-4) Changing active number of nics for network id=206 on -1
  518. 2013-09-24 11:21:25,383 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-4) Asking VirtualRouter to release Nic[91-6-c175b15e-3f0a-4f34-9f09-b8b20189e5e1-10.1.1.121]
  519. 2013-09-24 11:21:25,383 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-4) Successfully released network resources for the vm VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]
  520. 2013-09-24 11:21:25,383 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-4) Successfully released storage resources for the vm VM[User|ac0f8f8d-0603-46ce-8242-5b08ece29de2]
  521. 2013-09-24 11:21:25,405 DEBUG [cloud.network.NetworkModelImpl] (HA-Worker-4:work-4) Service SecurityGroup is not supported in the network id=206
  522. 2013-09-24 11:21:25,408 DEBUG [cloud.network.NetworkModelImpl] (HA-Worker-4:work-4) Service SecurityGroup is not supported in the network id=206
  523. 2013-09-24 11:21:25,439 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-4) VM state transitted from :Stopping to Stopped with event: OperationSucceededvm's original host id: 13 new host id: null host id before state transition: 13
  524. 2013-09-24 11:21:25,444 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-4) Hosts's actual total CPU: 83232 and CPU after applying overprovisioning: 83232
  525. 2013-09-24 11:21:25,444 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-4) Hosts's actual total RAM: 270846586880 and RAM after applying overprovisioning: 270846590976
  526. 2013-09-24 11:21:25,444 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-4) release cpu from host: 13, old used: 5500,reserved: 1000, actual total: 83232, total with overprovisioning: 83232; new used: 3000,reserved:3500; movedfromreserved: false,moveToReserveredtrue
  527. 2013-09-24 11:21:25,444 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-4) release mem from host: 13, old used: 2684354560,reserved: 268435456, total: 270846590976; new used: 1610612736,reserved:1342177280; movedfromreserved: false,moveToReserveredtrue
  528. 2013-09-24 11:21:25,500 ERROR [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-4) Terminating HAWork[4-HA-6-Running-Scheduled]
  529. java.lang.NullPointerException
  530. at com.cloud.storage.VolumeManagerImpl.canVmRestartOnAnotherServer(VolumeManagerImpl.java:2641)
  531. at com.cloud.ha.HighAvailabilityManagerImpl.restart(HighAvailabilityManagerImpl.java:516)
  532. at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:831)
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!