API tools faq
paste
cweise

XenServer - CS Disaster

cweise
Jun 13th, 2012
54
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 48.59 KB | None | 0 0
raw download clone embed print report
  1. 2012-06-13 01:09:25,790 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-313:null) Seq 6-1254951110: Executing request
  2. 2012-06-13 01:09:26,183 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) External load balancer devices stats collector is running...
  3. 2012-06-13 01:09:26,198 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 205 is not configured for external networking, so skipping usage check
  4. .
  5. 2012-06-13 01:09:26,200 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 219 is not configured for external networking, so skipping usage check
  6. .
  7. 2012-06-13 01:09:26,205 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 210 is not configured for external networking, so skipping usage check
  8. .
  9. 2012-06-13 01:09:26,207 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 211 is not configured for external networking, so skipping usage check
  10. .
  11. 2012-06-13 01:09:26,211 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 215 is not configured for external networking, so skipping usage check
  12. .
  13. 2012-06-13 01:09:26,215 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 213 is not configured for external networking, so skipping usage check
  14. .
  15. 2012-06-13 01:09:26,218 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 214 is not configured for external networking, so skipping usage check
  16. .
  17. 2012-06-13 01:09:26,220 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 218 is not configured for external networking, so skipping usage check
  18. .
  19. 2012-06-13 01:09:26,223 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 224 is not configured for external networking, so skipping usage check
  20. .
  21. 2012-06-13 01:09:26,227 DEBUG [cloud.network.ExternalLoadBalancerUsageManagerImpl] (ExternalNetworkMonitor-1:null) Network 221 is not configured for external networking, so skipping usage check
  22. .
  23. 2012-06-13 01:09:26,227 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Ping from 4
  24. 2012-06-13 01:09:26,236 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-4:null) checking health of usage server
  25. 2012-06-13 01:09:26,238 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-4:null) usage server running? true, heartbeat: Wed Jun 13 01:08:39 PDT 2012
  26. 2012-06-13 01:09:26,242 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-4:null) Ping from 3
  27. 2012-06-13 01:09:26,396 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-313:null) Seq 6-1254951110: Response Received:
  28. 2012-06-13 01:09:26,396 DEBUG [agent.transport.Request] (DirectAgent-313:null) Seq 6-1254951110: Processing: { Ans: , MgmtId: 33970771068361, via: 6, Ver: v1, Flags: 10, [{"NetworkUsageAnswer"
  29. :{"routerName":"r-14-VM","bytesSent":73596,"bytesReceived":647418,"result":true,"details":"","wait":0}}] }
  30. 2012-06-13 01:09:26,396 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-313:null) Cleanup succeeded. Details
  31. 2012-06-13 01:09:26,396 DEBUG [agent.transport.Request] (RouterMonitor-1:null) Seq 6-1254951110: Received: { Ans: , MgmtId: 33970771068361, via: 6, Ver: v1, Flags: 10, { NetworkUsageAnswer } }
  32. 2012-06-13 01:09:26,396 DEBUG [cloud.vm.VirtualMachineManagerImpl] (RouterMonitor-1:null) Cleanup succeeded. Details
  33. 2012-06-13 01:09:26,396 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Details from executing class com.cloud.agent.api.NetworkUsageCommand:
  34. 2012-06-13 01:09:26,408 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-1907821087: Executing request
  35. 2012-06-13 01:09:35,557 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-313:null) Ping from 1
  36. 2012-06-13 01:09:36,028 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-411:null) Seq 1-2140078082: Executing request
  37. 2012-06-13 01:09:36,630 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-411:null) Seq 1-2140078082: Response Received:
  38. 2012-06-13 01:09:36,630 DEBUG [agent.transport.Request] (DirectAgent-411:null) Seq 1-2140078082: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"ClusterSyncAnswer":
  39. {"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}] }
  40. 2012-06-13 01:09:37,888 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-459:null) Seq 5-1907818498: Executing request
  41. 2012-06-13 01:09:47,485 WARN [xen.resource.CitrixResourceBase] (DirectAgent-313:null) can not ping xenserver 859c89dc-5894-4ec6-9bca-c66c5dd18b04
  42. 2012-06-13 01:09:47,485 WARN [agent.manager.DirectAgentAttache] (DirectAgent-313:null) Unable to get current status on 5
  43. 2012-06-13 01:09:47,489 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Investigating why host 5 has disconnected with event AgentDisconnected
  44. 2012-06-13 01:09:47,489 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) checking if agent (5) is alive
  45. 2012-06-13 01:09:47,492 DEBUG [agent.transport.Request] (AgentTaskPool-9:null) Seq 5-1907821088: Sending { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  46. 2012-06-13 01:09:47,492 DEBUG [agent.transport.Request] (AgentTaskPool-9:null) Seq 5-1907821088: Executing: { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  47. 2012-06-13 01:09:47,492 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-28:null) Seq 5-1907821088: Executing request
  48. 2012-06-13 01:09:48,481 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-28:null) Seq 5-1907821088: Response Received:
  49. 2012-06-13 01:09:48,481 DEBUG [agent.transport.Request] (DirectAgent-28:null) Seq 5-1907821088: Processing: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, [{"CheckHealthAnswer":{"result":false,"details":"resource is not alive","wait":0}}] }
  50.  
  51. 2012-06-13 01:09:48,482 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-28:null) Cleanup failed due to resource is not alive
  52. 2012-06-13 01:09:48,482 DEBUG [agent.transport.Request] (AgentTaskPool-9:null) Seq 5-1907821088: Received: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  53. 2012-06-13 01:09:48,482 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-9:null) Cleanup failed due to resource is not alive
  54. 2012-06-13 01:09:48,482 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is not alive
  55. 2012-06-13 01:09:48,484 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-9:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  56. 2012-06-13 01:09:48,484 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-9:null) VmwareInvestigator unable to determine the state of the host. Moving on.
  57. 2012-06-13 01:09:48,491 DEBUG [agent.transport.Request] (AgentTaskPool-9:null) Seq 1-2140080426: Sending { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  58. 2012-06-13 01:09:48,492 DEBUG [agent.transport.Request] (AgentTaskPool-9:null) Seq 1-2140080426: Executing: { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  59. 2012-06-13 01:09:48,492 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-466:null) Seq 1-2140080426: Executing request
  60. 2012-06-13 01:09:48,996 DEBUG [xen.resource.XenServer56Resource] (DirectAgent-466:null) Heart beat is still going
  61. 2012-06-13 01:09:48,997 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-466:null) Seq 1-2140080426: Response Received:
  62. 2012-06-13 01:09:48,997 DEBUG [agent.transport.Request] (DirectAgent-466:null) Seq 1-2140080426: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"CheckOnHostAnswer":{"determined":true,"alive":true,"result":true,"details":"Heartbeat is still going","wait":0}}] }
  63. 2012-06-13 01:09:48,997 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-466:null) Cleanup succeeded. Details Heartbeat is still going
  64. 2012-06-13 01:09:48,997 DEBUG [agent.transport.Request] (AgentTaskPool-9:null) Seq 1-2140080426: Received: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, { CheckOnHostAnswer } }
  65. 2012-06-13 01:09:48,997 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-9:null) Cleanup succeeded. Details Heartbeat is still going
  66. 2012-06-13 01:09:48,997 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Details from executing class com.cloud.agent.api.CheckOnHostCommand: Heartbeat is still going
  67. 2012-06-13 01:09:48,998 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-9:null) XenServerInvestigator was able to determine host 5 is in Up
  68. 2012-06-13 01:09:48,998 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) The state determined is Up
  69. 2012-06-13 01:09:48,998 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Agent is determined to be up and running
  70. 2012-06-13 01:09:48,998 DEBUG [cloud.host.Status] (AgentTaskPool-9:null) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 5, name = AGC-A2X2]
  71. 2012-06-13 01:09:49,010 DEBUG [cloud.host.Status] (AgentTaskPool-9:null) Agent status update: [id = 5; name = AGC-A2X2; old status = Up; event = Ping; new status = Up; old update count = 120; new update count = 121]
  72. 2012-06-13 01:09:55,170 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
  73. 2012-06-13 01:09:55,365 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
  74. 2012-06-13 01:09:55,701 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Running Capacity Checker ...
  75. 2012-06-13 01:09:55,701 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) recalculating system capacity
  76. 2012-06-13 01:09:55,701 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing cpu/ram capacity update
  77. 2012-06-13 01:09:55,717 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 11 VMs on host 1
  78. 2012-06-13 01:09:55,720 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 2 VM, not running on host 1
  79. 2012-06-13 01:09:55,722 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) No need to calibrate cpu capacity, host:1 usedCpu: 10500 reservedCpu: 0
  80. 2012-06-13 01:09:55,722 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) No need to calibrate memory capacity, host:1 usedMem: 40131100672 reservedMem: 0
  81. 2012-06-13 01:09:55,735 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 10 VMs on host 5
  82. 2012-06-13 01:09:55,739 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 4 VM, not running on host 5
  83. 2012-06-13 01:09:55,742 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) No need to calibrate cpu capacity, host:5 usedCpu: 7500 reservedCpu: 0
  84. 2012-06-13 01:09:55,742 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) No need to calibrate memory capacity, host:5 usedMem: 22951231488 reservedMem: 0
  85. 2012-06-13 01:09:55,750 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 4 VMs on host 6
  86. 2012-06-13 01:09:55,755 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 4 VM, not running on host 6
  87. 2012-06-13 01:09:55,757 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) No need to calibrate cpu capacity, host:6 usedCpu: 3500 reservedCpu: 0
  88. 2012-06-13 01:09:55,757 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) No need to calibrate memory capacity, host:6 usedMem: 11408506880 reservedMem: 0
  89. 2012-06-13 01:09:55,757 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done executing cpu/ram capacity update
  90. 2012-06-13 01:09:55,757 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing storage capacity update
  91. 2012-06-13 01:09:55,776 DEBUG [cloud.storage.StorageManagerImpl] (CapacityChecker:null) Successfully set Capacity - 5497545555968 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 200, PodId 1
  92. 2012-06-13 01:09:55,783 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 9 routers.
  93.  
  94. 2012-06-13 01:09:55,783 DEBUG [cloud.storage.StorageManagerImpl] (CapacityChecker:null) Successfully set Capacity - 590914519040 for capacity type - 9 , DataCenterId - 1, HostOrPoolId - 201, PodId 1
  95. 2012-06-13 01:09:55,789 DEBUG [cloud.storage.StorageManagerImpl] (CapacityChecker:null) Successfully set Capacity - 590914519040 for capacity type - 9 , DataCenterId - 1, HostOrPoolId - 202, PodId 1
  96. 2012-06-13 01:09:55,795 DEBUG [cloud.storage.StorageManagerImpl] (CapacityChecker:null) Successfully set Capacity - 590914519040 for capacity type - 9 , DataCenterId - 1, HostOrPoolId - 203, PodId 1
  97. 2012-06-13 01:09:55,795 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done executing storage capacity update
  98. 2012-06-13 01:09:55,795 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing capacity updates for public ip and Vlans
  99. 2012-06-13 01:09:55,815 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done capacity updates for public ip and Vlans
  100. 2012-06-13 01:09:55,815 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing capacity updates for private ip
  101. 2012-06-13 01:09:55,822 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done executing capacity updates for private ip
  102. 2012-06-13 01:09:55,822 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done recalculating system capacity
  103. 2012-06-13 01:09:55,841 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done running Capacity Checker ...
  104. 2012-06-13 01:10:06,582 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-28:null) Ping from 6
  105. 2012-06-13 01:10:06,750 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-78:null) Seq 6-1254948866: Executing request
  106. 2012-06-13 01:10:06,987 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-78:null) Seq 6-1254948866: Response Received:
  107. 2012-06-13 01:10:06,987 DEBUG [agent.transport.Request] (DirectAgent-78:null) Seq 6-1254948866: Processing: { Ans: , MgmtId: 33970771068361, via: 6, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
  108. 2012-06-13 01:10:06,987 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-78:null) Cleanup succeeded. Details null
  109. 2012-06-13 01:10:06,987 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-78:null) Cleanup succeeded. Details null
  110. 2012-06-13 01:10:25,170 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
  111. 2012-06-13 01:10:25,365 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
  112. 2012-06-13 01:10:25,781 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 9 routers.
  113. 2012-06-13 01:10:26,240 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) Ping from 3
  114. 2012-06-13 01:10:26,262 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-15:null) Ping from 4
  115. 2012-06-13 01:10:35,405 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-28:null) Ping from 1
  116. 2012-06-13 01:10:36,028 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-78:null) Seq 1-2140078082: Executing request
  117. 2012-06-13 01:10:36,626 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-78:null) Seq 1-2140078082: Response Received:
  118. 2012-06-13 01:10:36,627 DEBUG [agent.transport.Request] (DirectAgent-78:null) Seq 1-2140078082: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}] }
  119. 2012-06-13 01:10:42,476 WARN [xen.resource.CitrixResourceBase] (DirectAgent-28:null) can not ping xenserver 859c89dc-5894-4ec6-9bca-c66c5dd18b04
  120. 2012-06-13 01:10:42,476 WARN [agent.manager.DirectAgentAttache] (DirectAgent-28:null) Unable to get current status on 5
  121. 2012-06-13 01:10:42,478 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:null) Investigating why host 5 has disconnected with event AgentDisconnected
  122. 2012-06-13 01:10:42,478 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-10:null) checking if agent (5) is alive
  123. 2012-06-13 01:10:42,496 DEBUG [agent.transport.Request] (AgentTaskPool-10:null) Seq 5-1907821089: Sending { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  124. 2012-06-13 01:10:42,496 DEBUG [agent.transport.Request] (AgentTaskPool-10:null) Seq 5-1907821089: Executing: { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  125. 2012-06-13 01:10:42,497 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-390:null) Seq 5-1907821089: Executing request
  126. 2012-06-13 01:10:45,498 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-390:null) Seq 5-1907821089: Response Received:
  127. 2012-06-13 01:10:45,498 DEBUG [agent.transport.Request] (DirectAgent-390:null) Seq 5-1907821089: Processing: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, [{"CheckHealthAnswer":{"result":false,"details":"resource is not alive","wait":0}}] }
  128. 2012-06-13 01:10:45,498 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-390:null) Cleanup failed due to resource is not alive
  129. 2012-06-13 01:10:45,498 DEBUG [agent.transport.Request] (AgentTaskPool-10:null) Seq 5-1907821089: Received: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  130. 2012-06-13 01:10:45,498 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-10:null) Cleanup failed due to resource is not alive
  131. 2012-06-13 01:10:45,499 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-10:null) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is not alive
  132. 2012-06-13 01:10:45,501 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-10:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  133. 2012-06-13 01:10:45,501 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-10:null) VmwareInvestigator unable to determine the state of the host. Moving on.
  134.  
  135. 2012-06-13 01:10:45,507 DEBUG [agent.transport.Request] (AgentTaskPool-10:null) Seq 1-2140080427: Sending { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  136. 2012-06-13 01:10:45,507 DEBUG [agent.transport.Request] (AgentTaskPool-10:null) Seq 1-2140080427: Executing: { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  137. 2012-06-13 01:10:45,507 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-372:null) Seq 1-2140080427: Executing request
  138. 2012-06-13 01:10:45,909 DEBUG [xen.resource.XenServer56Resource] (DirectAgent-372:null) Heart beat is still going
  139. 2012-06-13 01:10:45,909 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-372:null) Seq 1-2140080427: Response Received:
  140. 2012-06-13 01:10:45,909 DEBUG [agent.transport.Request] (DirectAgent-372:null) Seq 1-2140080427: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"CheckOnHostAnswer":{"determined":true,"alive":true,"result":true,"details":"Heartbeat is still going","wait":0}}] }
  141. 2012-06-13 01:10:45,909 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-372:null) Cleanup succeeded. Details Heartbeat is still going
  142. 2012-06-13 01:10:45,909 DEBUG [agent.transport.Request] (AgentTaskPool-10:null) Seq 1-2140080427: Received: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, { CheckOnHostAnswer } }
  143. 2012-06-13 01:10:45,909 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-10:null) Cleanup succeeded. Details Heartbeat is still going
  144. 2012-06-13 01:10:45,909 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-10:null) Details from executing class com.cloud.agent.api.CheckOnHostCommand: Heartbeat is still going
  145. 2012-06-13 01:10:45,909 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-10:null) XenServerInvestigator was able to determine host 5 is in Up
  146. 2012-06-13 01:10:45,909 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:null) The state determined is Up
  147. 2012-06-13 01:10:45,909 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:null) Agent is determined to be up and running
  148. 2012-06-13 01:10:45,909 DEBUG [cloud.host.Status] (AgentTaskPool-10:null) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 5, name = AGC-A2X2]
  149. 2012-06-13 01:10:45,927 DEBUG [cloud.host.Status] (AgentTaskPool-10:null) Agent status update: [id = 5; name = AGC-A2X2; old status = Up; event = Ping; new status = Up; old update count = 121; new update count = 122]
  150. 2012-06-13 01:10:49,467 DEBUG [cloud.network.NetworkManagerImpl] (Network-Scavenger-1:null) Network 224 is still free but it's not time to shutdown yet: 1308178173
  151. 2012-06-13 01:10:55,170 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
  152. 2012-06-13 01:10:55,364 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
  153. 2012-06-13 01:10:55,782 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 9 routers.
  154. 2012-06-13 01:11:06,586 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-390:null) Ping from 6
  155. 2012-06-13 01:11:06,750 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-81:null) Seq 6-1254948866: Executing request
  156. 2012-06-13 01:11:06,972 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-81:null) Seq 6-1254948866: Response Received:
  157. 2012-06-13 01:11:06,973 DEBUG [agent.transport.Request] (DirectAgent-81:null) Seq 6-1254948866: Processing: { Ans: , MgmtId: 33970771068361, via: 6, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
  158. 2012-06-13 01:11:06,973 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-81:null) Cleanup succeeded. Details null
  159. 2012-06-13 01:11:06,973 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-81:null) Cleanup succeeded. Details null
  160. 2012-06-13 01:11:25,170 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
  161. 2012-06-13 01:11:25,365 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
  162. 2012-06-13 01:11:25,781 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 9 routers.
  163. 2012-06-13 01:11:26,236 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-1:null) Ping from 3
  164. 2012-06-13 01:11:26,259 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) Ping from 4
  165. 2012-06-13 01:11:26,909 INFO [agent.manager.AgentMonitor] (AgentMonitor:null) Found the following agents behind on ping: [5]
  166. 2012-06-13 01:11:26,912 DEBUG [cloud.host.Status] (AgentMonitor:null) Ping timeout for host 5, do invstigation
  167. 2012-06-13 01:11:26,914 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-11:null) Investigating why host 5 has disconnected with event PingTimeout
  168. 2012-06-13 01:11:26,914 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-11:null) checking if agent (5) is alive
  169. 2012-06-13 01:11:26,918 DEBUG [agent.transport.Request] (AgentTaskPool-11:null) Seq 5-1907821090: Sending { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  170. 2012-06-13 01:11:26,918 DEBUG [agent.transport.Request] (AgentTaskPool-11:null) Seq 5-1907821090: Executing: { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  171. 2012-06-13 01:11:26,918 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-453:null) Seq 5-1907821090: Executing request
  172. 2012-06-13 01:11:29,920 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-453:null) Seq 5-1907821090: Response Received:
  173. 2012-06-13 01:11:29,920 DEBUG [agent.transport.Request] (DirectAgent-453:null) Seq 5-1907821090: Processing: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, [{"CheckHealthAnswer":{"result":false,"details":"resource is not alive","wait":0}}] }
  174. 2012-06-13 01:11:29,920 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-453:null) Cleanup failed due to resource is not alive
  175. 2012-06-13 01:11:29,920 DEBUG [agent.transport.Request] (AgentTaskPool-11:null) Seq 5-1907821090: Received: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  176. 2012-06-13 01:11:29,920 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-11:null) Cleanup failed due to resource is not alive
  177. 2012-06-13 01:11:29,921 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-11:null) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is not alive
  178. 2012-06-13 01:11:29,923 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-11:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  179.  
  180. 2012-06-13 01:11:29,923 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-11:null) VmwareInvestigator unable to determine the state of the host. Moving on.
  181. 2012-06-13 01:11:29,928 DEBUG [agent.transport.Request] (AgentTaskPool-11:null) Seq 1-2140080428: Sending { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  182. 2012-06-13 01:11:29,929 DEBUG [agent.transport.Request] (AgentTaskPool-11:null) Seq 1-2140080428: Executing: { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  183. 2012-06-13 01:11:29,929 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-462:null) Seq 1-2140080428: Executing request
  184. 2012-06-13 01:11:30,322 DEBUG [xen.resource.XenServer56Resource] (DirectAgent-462:null) Heart beat is still going
  185. 2012-06-13 01:11:30,322 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-462:null) Seq 1-2140080428: Response Received:
  186. 2012-06-13 01:11:30,322 DEBUG [agent.transport.Request] (DirectAgent-462:null) Seq 1-2140080428: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"CheckOnHostAnswer":{"determined":true,"alive":true,"result":true,"details":"Heartbeat is still going","wait":0}}] }
  187. 2012-06-13 01:11:30,323 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-462:null) Cleanup succeeded. Details Heartbeat is still going
  188. 2012-06-13 01:11:30,323 DEBUG [agent.transport.Request] (AgentTaskPool-11:null) Seq 1-2140080428: Received: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, { CheckOnHostAnswer } }
  189. 2012-06-13 01:11:30,323 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-11:null) Cleanup succeeded. Details Heartbeat is still going
  190. 2012-06-13 01:11:30,323 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-11:null) Details from executing class com.cloud.agent.api.CheckOnHostCommand: Heartbeat is still going
  191. 2012-06-13 01:11:30,323 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-11:null) XenServerInvestigator was able to determine host 5 is in Up
  192. 2012-06-13 01:11:30,323 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-11:null) The state determined is Up
  193. 2012-06-13 01:11:30,323 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-11:null) Agent is determined to be up and running
  194. 2012-06-13 01:11:30,323 DEBUG [cloud.host.Status] (AgentTaskPool-11:null) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 5, name = AGC-A2X2]
  195. 2012-06-13 01:11:30,334 DEBUG [cloud.host.Status] (AgentTaskPool-11:null) Agent status update: [id = 5; name = AGC-A2X2; old status = Up; event = Ping; new status = Up; old update count = 122; new update count = 123]
  196. 2012-06-13 01:11:35,032 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-201:null) Seq 1-2140078083: Executing request
  197. 2012-06-13 01:11:35,032 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-201:null) Seq 1-2140078083: Response Received:
  198. 2012-06-13 01:11:35,032 DEBUG [agent.transport.Request] (DirectAgent-201:null) Seq 1-2140078083: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
  199. 2012-06-13 01:11:35,032 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-201:null) Cleanup succeeded. Details null
  200. 2012-06-13 01:11:35,412 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-453:null) Ping from 1
  201. 2012-06-13 01:11:36,028 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-262:null) Seq 1-2140078082: Executing request
  202. 2012-06-13 01:11:36,646 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-262:null) Seq 1-2140078082: Response Received:
  203. 2012-06-13 01:11:36,646 DEBUG [agent.transport.Request] (DirectAgent-262:null) Seq 1-2140078082: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}] }
  204. 2012-06-13 01:11:42,476 WARN [xen.resource.CitrixResourceBase] (DirectAgent-13:null) can not ping xenserver 859c89dc-5894-4ec6-9bca-c66c5dd18b04
  205. 2012-06-13 01:11:42,476 WARN [agent.manager.DirectAgentAttache] (DirectAgent-13:null) Unable to get current status on 5
  206. 2012-06-13 01:11:42,478 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Investigating why host 5 has disconnected with event AgentDisconnected
  207. 2012-06-13 01:11:42,478 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) checking if agent (5) is alive
  208. 2012-06-13 01:11:42,481 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 5-1907821091: Sending { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  209. 2012-06-13 01:11:42,482 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 5-1907821091: Executing: { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  210. 2012-06-13 01:11:42,482 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-99:null) Seq 5-1907821091: Executing request
  211. 2012-06-13 01:11:45,483 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-99:null) Seq 5-1907821091: Response Received:
  212. 2012-06-13 01:11:45,483 DEBUG [agent.transport.Request] (DirectAgent-99:null) Seq 5-1907821091: Processing: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, [{"CheckHealthAnswer":{"result":false,"details":"resource is not alive","wait":0}}] }
  213. 2012-06-13 01:11:45,483 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-99:null) Cleanup failed due to resource is not alive
  214. 2012-06-13 01:11:45,483 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 5-1907821091: Received: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  215. 2012-06-13 01:11:45,483 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-12:null) Cleanup failed due to resource is not alive
  216. 2012-06-13 01:11:45,483 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is not alive
  217. 2012-06-13 01:11:45,485 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-12:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  218. 2012-06-13 01:11:45,485 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-12:null) VmwareInvestigator unable to determine the state of the host. Moving on.
  219. 2012-06-13 01:11:45,491 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 1-2140080429: Sending { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  220.  
  221. 2012-06-13 01:11:45,492 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-488:null) Seq 1-2140080429: Executing request
  222. 2012-06-13 01:11:45,921 DEBUG [xen.resource.XenServer56Resource] (DirectAgent-488:null) Heart beat is still going
  223. 2012-06-13 01:11:45,921 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-488:null) Seq 1-2140080429: Response Received:
  224. 2012-06-13 01:11:45,921 DEBUG [agent.transport.Request] (DirectAgent-488:null) Seq 1-2140080429: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"CheckOnHostAnswer":{"determined":true,"alive":true,"result":true,"details":"Heartbeat is still going","wait":0}}] }
  225. 2012-06-13 01:11:45,921 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-488:null) Cleanup succeeded. Details Heartbeat is still going
  226. 2012-06-13 01:11:45,921 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 1-2140080429: Received: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, { CheckOnHostAnswer } }
  227. 2012-06-13 01:11:45,922 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-12:null) Cleanup succeeded. Details Heartbeat is still going
  228. 2012-06-13 01:11:45,922 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Details from executing class com.cloud.agent.api.CheckOnHostCommand: Heartbeat is still going
  229. 2012-06-13 01:11:45,922 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-12:null) XenServerInvestigator was able to determine host 5 is in Up
  230. 2012-06-13 01:11:45,922 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) The state determined is Up
  231. 2012-06-13 01:11:45,922 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Agent is determined to be up and running
  232. 2012-06-13 01:11:45,922 DEBUG [cloud.host.Status] (AgentTaskPool-12:null) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 5, name = AGC-A2X2]
  233. 2012-06-13 01:11:45,933 DEBUG [cloud.host.Status] (AgentTaskPool-12:null) Agent status update: [id = 5; name = AGC-A2X2; old status = Up; event = Ping; new status = Up; old update count = 123; new update count = 124]
  234. 2012-06-13 01:11:55,170 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
  235. 2012-06-13 01:11:55,366 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
  236. 2012-06-13 01:11:55,781 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 9 routers.
  237. 2012-06-13 01:12:06,585 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-333:null) Ping from 6
  238. 2012-06-13 01:12:06,750 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-188:null) Seq 6-1254948866: Executing request
  239. 2012-06-13 01:12:06,979 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-188:null) Seq 6-1254948866: Response Received:
  240. 2012-06-13 01:12:06,979 DEBUG [agent.transport.Request] (DirectAgent-188:null) Seq 6-1254948866: Processing: { Ans: , MgmtId: 33970771068361, via: 6, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
  241. 2012-06-13 01:12:06,979 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-188:null) Cleanup succeeded. Details null
  242. 2012-06-13 01:12:06,979 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-188:null) Cleanup succeeded. Details null
  243. 2012-06-13 01:12:18,006 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-195:null) Catch Exception: com.xensource.xenapi.Types$CannotContactHost Can't connect host 10.0.1.55 due to Cannot forward messages because the host cannot be contacted. The host may be switched off or there may be network connectivity problems.
  244. 2012-06-13 01:12:18,006 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-195:null) Trying to reset master of slave 10.0.1.55 to 10.0.1.54
  245. 2012-06-13 01:12:24,009 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-195:null) localLogout has problem Failed to read server's response: No route to host
  246. 2012-06-13 01:12:24,009 WARN [agent.manager.DirectAgentAttache] (DirectAgent-195:null) Seq 5-1907821084: Exception Caught while executing command
  247. com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.0.1.55 to 10.0.1.54 due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host
  248. at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:439)
  249. at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:651)
  250. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:4870)
  251. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:4420)
  252. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:449)
  253. at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
  254. at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
  255. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  256. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  257. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  258. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
  259. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
  260. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  261. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  262. at java.lang.Thread.run(Thread.java:679)
  263. 2012-06-13 01:12:24,010 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-195:null) Seq 5-1907821084: Response Received:
  264. 2012-06-13 01:12:24,010 DEBUG [agent.transport.Request] (DirectAgent-195:null) Seq 5-1907821084: Processing: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.0.1.55 to 10.0.1.54 due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host","wait":0}}] }
  265. 2012-06-13 01:12:24,010 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-195:null) Cleanup failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.0.1.55 to 10.0.1.54 due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host
  266. 2012-06-13 01:12:24,010 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 5-1907821084: Received: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, { Answer } }
  267. 2012-06-13 01:12:24,011 WARN [cloud.vm.VirtualMachineManagerImpl] (StatsCollector-2:null) Cleanup failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of slave 10.0.1.55 to 10.0.1.54 due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host
  268.  
  269. 2012-06-13 01:12:24,017 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-13:null) Seq 6-1254951111: Executing request
  270. 2012-06-13 01:12:25,170 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
  271. 2012-06-13 01:12:25,366 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
  272. 2012-06-13 01:12:25,767 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-13:null) Seq 6-1254951111: Response Received:
  273. 2012-06-13 01:12:25,767 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 6-1254951111: Received: { Ans: , MgmtId: 33970771068361, via: 6, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  274. 2012-06-13 01:12:25,767 DEBUG [cloud.vm.VirtualMachineManagerImpl] (StatsCollector-2:null) Cleanup succeeded. Details null
  275. 2012-06-13 01:12:25,767 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-13:null) Cleanup succeeded. Details null
  276. 2012-06-13 01:12:25,781 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 9 routers.
  277. 2012-06-13 01:12:26,232 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 3
  278. 2012-06-13 01:12:26,254 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 4
  279. 2012-06-13 01:12:26,914 INFO [agent.manager.AgentMonitor] (AgentMonitor:null) Found the following agents behind on ping: [5]
  280. 2012-06-13 01:12:26,917 DEBUG [cloud.host.Status] (AgentMonitor:null) Ping timeout for host 5, do invstigation
  281. 2012-06-13 01:12:26,919 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-13:null) Investigating why host 5 has disconnected with event PingTimeout
  282. 2012-06-13 01:12:26,919 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-13:null) checking if agent (5) is alive
  283. 2012-06-13 01:12:26,924 DEBUG [agent.transport.Request] (AgentTaskPool-13:null) Seq 5-1907821092: Sending { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  284. 2012-06-13 01:12:26,924 DEBUG [agent.transport.Request] (AgentTaskPool-13:null) Seq 5-1907821092: Executing: { Cmd , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  285. 2012-06-13 01:12:26,924 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-450:null) Seq 5-1907821092: Executing request
  286. 2012-06-13 01:12:27,010 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-450:null) Seq 5-1907821092: Response Received:
  287. 2012-06-13 01:12:27,010 DEBUG [agent.transport.Request] (DirectAgent-450:null) Seq 5-1907821092: Processing: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, [{"CheckHealthAnswer":{"result":false,"details":"resource is not alive","wait":0}}] }
  288. 2012-06-13 01:12:27,010 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-450:null) Cleanup failed due to resource is not alive
  289. 2012-06-13 01:12:27,010 DEBUG [agent.transport.Request] (AgentTaskPool-13:null) Seq 5-1907821092: Received: { Ans: , MgmtId: 33970771068361, via: 5, Ver: v1, Flags: 10, { CheckHealthAnswer } }
  290. 2012-06-13 01:12:27,010 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-13:null) Cleanup failed due to resource is not alive
  291. 2012-06-13 01:12:27,011 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-13:null) Details from executing class com.cloud.agent.api.CheckHealthCommand: resource is not alive
  292. 2012-06-13 01:12:27,013 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-13:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  293. 2012-06-13 01:12:27,013 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-13:null) VmwareInvestigator unable to determine the state of the host. Moving on.
  294. 2012-06-13 01:12:27,031 DEBUG [agent.transport.Request] (AgentTaskPool-13:null) Seq 1-2140080430: Sending { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  295. 2012-06-13 01:12:27,031 DEBUG [agent.transport.Request] (AgentTaskPool-13:null) Seq 1-2140080430: Executing: { Cmd , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 100011, [{"CheckOnHostCommand":{"host":{"guid":"859c89dc-5894-4ec6-9bca-c66c5dd18b04","privateNetwork":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.0.50.55","netmask":"255.255.255.0","mac":"fe:ff:ff:ff:ff:ff","isSecurityGroupEnabled":false},"storageNetwork2":{"ip":"10.0.1.55","netmask":"255.255.255.0","mac":"18:03:73:0e:43:81","isSecurityGroupEnabled":false}},"wait":20}}] }
  296. 2012-06-13 01:12:27,031 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-408:null) Seq 1-2140080430: Executing request
  297. 2012-06-13 01:12:27,479 DEBUG [xen.resource.XenServer56Resource] (DirectAgent-408:null) Heart beat is still going
  298. 2012-06-13 01:12:27,479 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-408:null) Seq 1-2140080430: Response Received:
  299. 2012-06-13 01:12:27,479 DEBUG [agent.transport.Request] (DirectAgent-408:null) Seq 1-2140080430: Processing: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, [{"CheckOnHostAnswer":{"determined":true,"alive":true,"result":true,"details":"Heartbeat is still going","wait":0}}] }
  300. 2012-06-13 01:12:27,479 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-408:null) Cleanup succeeded. Details Heartbeat is still going
  301. 2012-06-13 01:12:27,479 DEBUG [agent.transport.Request] (AgentTaskPool-13:null) Seq 1-2140080430: Received: { Ans: , MgmtId: 33970771068361, via: 1, Ver: v1, Flags: 10, { CheckOnHostAnswer } }
  302. 2012-06-13 01:12:27,479 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-13:null) Cleanup succeeded. Details Heartbeat is still going
  303. 2012-06-13 01:12:27,479 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-13:null) Details from executing class com.cloud.agent.api.CheckOnHostCommand: Heartbeat is still going
  304. 2012-06-13 01:12:27,479 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-13:null) XenServerInvestigator was able to determine host 5 is in Up
  305. 2012-06-13 01:12:27,479 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-13:null) The state determined is Up
  306. 2012-06-13 01:12:27,479 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-13:null) Agent is determined to be up and running
  307. 2012-06-13 01:12:27,480 DEBUG [cloud.host.Status] (AgentTaskPool-13:null) Transition:[Resource state = Enabled, Agent event = Ping, Host id = 5, name = AGC-A2X2]
  308. 2012-06-13 01:12:27,491 DEBUG [cloud.host.Status] (AgentTaskPool-13:null) Agent status update: [id = 5; name = AGC-A2X2; old status = Up; event = Ping; new status = Up; old update count = 124; new update count = 125]
  309. 2012-06-13 01:12:28,324 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-430:null) Catch Exception: com.xensource.xenapi.Types$CannotContactHost Can't connect host 10.0.1.55 due to Cannot forward messages because the host cannot be contacted. The host may be switched off or there may be network connectivity problems.
  310. 2012-06-13 01:12:28,325 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-430:null) Trying to reset master of slave 10.0.1.55 to 10.0.1.54
  311. 2012-06-13 01:12:34,328 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-430:null) localLogout has problem Failed to read server's response: No route to host
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!