API tools faq
paste
teulaert

cs-ha-with-sg-ping-disabled

teulaert
Aug 5th, 2013
106
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 46.42 KB | None | 0 0
raw download clone embed print report
  1. 2013-08-05 11:32:49,442 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Investigating why host 7 has disconnected with event AgentDisconnected
  2. 2013-08-05 11:32:49,443 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) checking if agent (7) is alive
  3. 2013-08-05 11:32:49,448 DEBUG [agent.transport.Request] (AgentTaskPool-1:null) Seq 7-1858602374: Sending { Cmd , MgmtId: 90520732148205, via: 7, Ver: v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
  4. 2013-08-05 11:32:49,451 INFO [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 7-1858602374: Unable to send due to Resource [Host:7] is unreachable: Host 7: Channel is closed
  5. 2013-08-05 11:32:49,451 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 7-1858602374: Cancelling.
  6. 2013-08-05 11:32:49,451 WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Resource [Host:7] is unreachable: Host 7: Channel is closed
  7. 2013-08-05 11:32:49,452 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) SimpleInvestigator unable to determine the state of the host. Moving on.
  8. 2013-08-05 11:32:49,452 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) XenServerInvestigator unable to determine the state of the host. Moving on.
  9. 2013-08-05 11:32:49,453 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-1:null) checking if agent (7) is alive
  10. 2013-08-05 11:32:49,454 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-1:null) sending ping from (1) to agent's host ip address (192.168.122.32)
  11. 2013-08-05 11:32:49,456 DEBUG [agent.transport.Request] (AgentTaskPool-1:null) Seq 1-204538710: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"PingTestCommand":{"_computingHostIp":"192.168.122.32","wait":20}}] }
  12. 2013-08-05 11:32:49,506 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 1-204538710: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
  13. 2013-08-05 11:32:49,507 DEBUG [agent.transport.Request] (AgentTaskPool-1:null) Seq 1-204538710: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { Answer } }
  14. 2013-08-05 11:32:49,507 DEBUG [cloud.ha.AbstractInvestigatorImpl] (AgentTaskPool-1:null) host (192.168.122.32) has been successfully pinged, returning that host is up
  15. 2013-08-05 11:32:49,507 DEBUG [cloud.ha.UserVmDomRInvestigator] (AgentTaskPool-1:null) ping from (1) to agent's host ip address (192.168.122.32) successful, returning that agent is disconnected
  16. 2013-08-05 11:32:49,507 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) null was able to determine host 7 is in Disconnected
  17. 2013-08-05 11:32:49,507 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) The state determined is Disconnected
  18. 2013-08-05 11:32:49,508 WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Agent is disconnected but the host is still up: 7-n2
  19. 2013-08-05 11:32:49,512 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Host 7 is disconnecting with event AgentDisconnected
  20. 2013-08-05 11:32:49,514 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) The next status of agent 7is Alert, current status is Up
  21. 2013-08-05 11:32:49,514 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Deregistering link for 7 with state Alert
  22. 2013-08-05 11:32:49,514 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Remove Agent : 7
  23. 2013-08-05 11:32:49,514 DEBUG [agent.manager.ConnectedAgentAttache] (AgentTaskPool-1:null) Processing Disconnect.
  24. 2013-08-05 11:32:49,514 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq 7-1858600962: Sending disconnect to class com.cloud.network.security.SecurityGroupListener
  25. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer_EnhancerByCloudStack_dc08546e
  26. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl_EnhancerByCloudStack_b45df077
  27. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
  28. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
  29. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
  30. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_e11ea17b
  31. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
  32. 2013-08-05 11:32:49,515 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  33. 2013-08-05 11:32:49,516 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_7a900e1c
  34. 2013-08-05 11:32:49,516 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  35. 2013-08-05 11:32:49,516 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_f573d63f
  36. 2013-08-05 11:32:49,516 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
  37. 2013-08-05 11:32:49,517 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
  38. 2013-08-05 11:32:49,518 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor
  39. 2013-08-05 11:32:49,518 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
  40. 2013-08-05 11:32:49,518 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
  41. 2013-08-05 11:32:49,518 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
  42. 2013-08-05 11:32:49,518 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentTaskPool-1:null) Disconnected called on 7 with status Alert
  43. 2013-08-05 11:32:49,518 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
  44. 2013-08-05 11:32:49,520 DEBUG [cloud.host.Status] (AgentTaskPool-1:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 7, name = n2]
  45. 2013-08-05 11:32:49,552 DEBUG [cloud.host.Status] (AgentTaskPool-1:null) Agent status update: [id = 7; name = n2; old status = Up; event = AgentDisconnected; new status = Alert; old update count = 99; new update count = 100]
  46. 2013-08-05 11:32:49,552 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentTaskPool-1:null) Notifying other nodes of to disconnect
  47. 2013-08-05 11:32:49,555 WARN [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Scheduling restart for VMs on host 7
  48. 2013-08-05 11:32:49,562 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 19-ca8d1be8-7927-4eff-b9f8-5a68e389e733
  49. 2013-08-05 11:32:49,567 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  50. 2013-08-05 11:32:49,572 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) Processing HAWork[104-HA-19-Running-Investigating]
  51. 2013-08-05 11:32:49,575 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) HA on VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  52. 2013-08-05 11:32:49,576 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 23-28a63a18-9969-4546-9463-f1c8828831b3
  53. 2013-08-05 11:32:49,581 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  54. 2013-08-05 11:32:49,585 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) Processing HAWork[105-HA-23-Running-Investigating]
  55. 2013-08-05 11:32:49,588 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) HA on VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  56. 2013-08-05 11:32:49,589 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Notifying HA Mgr of to restart vm 25-c88924e9-a8c9-4705-acc8-3237ffcf009d
  57. 2013-08-05 11:32:49,591 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-0:work-104) Unable to reach the agent for VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  58. 2013-08-05 11:32:49,594 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) SimpleInvestigator found VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]to be alive? null
  59. 2013-08-05 11:32:49,594 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) XenServerInvestigator found VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]to be alive? null
  60. 2013-08-05 11:32:49,594 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-0:work-104) testing if VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] is alive
  61. 2013-08-05 11:32:49,597 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]
  62. 2013-08-05 11:32:49,602 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) Processing HAWork[106-HA-25-Running-Investigating]
  63. 2013-08-05 11:32:49,604 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) HA on VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]
  64. 2013-08-05 11:32:49,606 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentTaskPool-1:null) Notifying other nodes of to disconnect
  65. 2013-08-05 11:32:49,596 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-1:work-105) Unable to reach the agent for VM[User|28a63a18-9969-4546-9463-f1c8828831b3]: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  66. 2013-08-05 11:32:49,606 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) SimpleInvestigator found VM[User|28a63a18-9969-4546-9463-f1c8828831b3]to be alive? null
  67. 2013-08-05 11:32:49,607 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) XenServerInvestigator found VM[User|28a63a18-9969-4546-9463-f1c8828831b3]to be alive? null
  68. 2013-08-05 11:32:49,607 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-1:work-105) testing if VM[User|28a63a18-9969-4546-9463-f1c8828831b3] is alive
  69. 2013-08-05 11:32:49,612 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-2:work-106) Unable to reach the agent for VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  70. 2013-08-05 11:32:49,612 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) SimpleInvestigator found VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]to be alive? null
  71. 2013-08-05 11:32:49,612 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) XenServerInvestigator found VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]to be alive? null
  72. 2013-08-05 11:32:49,613 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:work-106) testing if VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d] is alive
  73. 2013-08-05 11:32:49,623 DEBUG [agent.transport.Request] (HA-Worker-0:work-104) Seq 1-204538711: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"PingTestCommand":{"_routerIp":"169.254.1.243","_privateIp":"192.168.122.112","wait":20}}] }
  74. 2013-08-05 11:32:49,628 DEBUG [agent.transport.Request] (HA-Worker-2:work-106) Seq 1-204538712: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"PingTestCommand":{"_routerIp":"169.254.1.243","_privateIp":"192.168.122.170","wait":20}}] }
  75. 2013-08-05 11:32:49,632 DEBUG [agent.transport.Request] (HA-Worker-1:work-105) Seq 1-204538713: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"PingTestCommand":{"_routerIp":"169.254.1.243","_privateIp":"192.168.122.133","wait":20}}] }
  76. 2013-08-05 11:32:54,097 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-204538711: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"PING 192.168.122.112 (192.168.122.112): 56 data bytes64 bytes from 192.168.122.151: Destination Host UnreachableVr HL TOS Len ID Flg off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 50c4 192.168.122.151 192.168.122.112 --- 192.168.122.112 ping statistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable to ping the vm, exiting","wait":0}}] }
  77. 2013-08-05 11:32:54,098 DEBUG [agent.transport.Request] (HA-Worker-0:work-104) Seq 1-204538711: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { Answer } }
  78. 2013-08-05 11:32:54,098 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-0:work-104) Details from executing class com.cloud.agent.api.PingTestCommand: PING 192.168.122.112 (192.168.122.112): 56 data bytes64 bytes from 192.168.122.151: Destination Host UnreachableVr HL TOS Len ID Flg off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 50c4 192.168.122.151 192.168.122.112 --- 192.168.122.112 ping statistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable to ping the vm, exiting
  79. 2013-08-05 11:32:54,099 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-0:work-104) VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] could not be pinged, returning that it is unknown
  80. 2013-08-05 11:32:54,099 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-0:work-104) Returning null since we're unable to determine state of VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  81. 2013-08-05 11:32:54,099 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) null found VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]to be alive? null
  82. 2013-08-05 11:32:54,100 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-104) Not a System Vm, unable to determine state of VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] returning null
  83. 2013-08-05 11:32:54,101 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-104) Testing if VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] is alive
  84. 2013-08-05 11:32:54,107 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-104) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] returning null
  85. 2013-08-05 11:32:54,107 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) null found VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]to be alive? null
  86. 2013-08-05 11:32:54,108 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) Fencing off VM that we don't know the state of
  87. 2013-08-05 11:32:54,108 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-0:work-104) Don't know how to fence non XenServer hosts KVM
  88. 2013-08-05 11:32:54,108 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) Fencer null returned null
  89. 2013-08-05 11:32:54,115 DEBUG [agent.transport.Request] (HA-Worker-0:work-104) Seq 1-204538714: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"FenceCommand":{"vmName":"i-2-19-VM","hostGuid":"945a8003-3d66-335b-b49d-dc4e6e40f46e-LibvirtComputingResource","hostIp":"192.168.122.32","inSeq":false,"wait":0}}] }
  90. 2013-08-05 11:32:54,201 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 1-204538714: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"FenceAnswer":{"result":false,"details":"Heart is still beating...","wait":0}}] }
  91. 2013-08-05 11:32:54,202 DEBUG [agent.transport.Request] (HA-Worker-0:work-104) Seq 1-204538714: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { FenceAnswer } }
  92. 2013-08-05 11:32:54,202 DEBUG [cloud.ha.KVMFencer] (HA-Worker-0:work-104) Unable to fence off VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] on Host[-7-Routing]
  93. 2013-08-05 11:32:54,203 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) Fencer KVMFenceBuilder returned false
  94. 2013-08-05 11:32:54,203 DEBUG [ovm.hypervisor.OvmFencer] (HA-Worker-0:work-104) Don't know how to fence non Ovm hosts KVM
  95. 2013-08-05 11:32:54,203 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) Fencer OvmFenceBuilder returned null
  96. 2013-08-05 11:32:54,204 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) We were unable to fence off the VM VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  97. 2013-08-05 11:32:54,210 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-104) Rescheduling HAWork[104-HA-19-Running-Investigating] to try again at Mon Aug 05 11:43:08 CEST 2013
  98. 2013-08-05 11:32:54,372 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running...
  99. 2013-08-05 11:32:54,443 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 10-312148543: Received: { Ans: , MgmtId: 90520732148205, via: 10, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  100. 2013-08-05 11:32:54,610 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-204538715: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  101. 2013-08-05 11:32:57,535 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) SeqA 9-49969: Processing Seq 9-49969: { Cmd , MgmtId: -1, via: 9, Ver: v1, Flags: 11, [{"ConsoleProxyLoadReportCommand":{"_proxyVmId":28,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] }
  102. 2013-08-05 11:32:57,538 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) SeqA 9-49969: Sending Seq 9-49969: { Ans: , MgmtId: 90520732148205, via: 9, Ver: v1, Flags: 100010, [{"AgentControlAnswer":{"result":true,"wait":0}}] }
  103. 2013-08-05 11:32:58,383 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 1-204538713: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"timeout","wait":0}}] }
  104. 2013-08-05 11:32:58,383 DEBUG [agent.transport.Request] (HA-Worker-1:work-105) Seq 1-204538713: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { Answer } }
  105. 2013-08-05 11:32:58,383 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-1:work-105) Details from executing class com.cloud.agent.api.PingTestCommand: timeout
  106. 2013-08-05 11:32:58,384 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-1:work-105) VM[User|28a63a18-9969-4546-9463-f1c8828831b3] could not be pinged, returning that it is unknown
  107. 2013-08-05 11:32:58,384 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-1:work-105) Returning null since we're unable to determine state of VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  108. 2013-08-05 11:32:58,384 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) null found VM[User|28a63a18-9969-4546-9463-f1c8828831b3]to be alive? null
  109. 2013-08-05 11:32:58,385 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-1:work-105) Not a System Vm, unable to determine state of VM[User|28a63a18-9969-4546-9463-f1c8828831b3] returning null
  110. 2013-08-05 11:32:58,385 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-1:work-105) Testing if VM[User|28a63a18-9969-4546-9463-f1c8828831b3] is alive
  111. 2013-08-05 11:32:58,388 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-1:work-105) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[User|28a63a18-9969-4546-9463-f1c8828831b3] returning null
  112. 2013-08-05 11:32:58,388 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) null found VM[User|28a63a18-9969-4546-9463-f1c8828831b3]to be alive? null
  113. 2013-08-05 11:32:58,389 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null) Seq 1-204538712: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"timeout","wait":0}}] }
  114. 2013-08-05 11:32:58,390 DEBUG [agent.transport.Request] (HA-Worker-2:work-106) Seq 1-204538712: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { Answer } }
  115. 2013-08-05 11:32:58,390 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-2:work-106) Details from executing class com.cloud.agent.api.PingTestCommand: timeout
  116. 2013-08-05 11:32:58,390 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:work-106) VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d] could not be pinged, returning that it is unknown
  117. 2013-08-05 11:32:58,391 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:work-106) Returning null since we're unable to determine state of VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]
  118. 2013-08-05 11:32:58,391 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) null found VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]to be alive? null
  119. 2013-08-05 11:32:58,391 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:work-106) Not a System Vm, unable to determine state of VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d] returning null
  120. 2013-08-05 11:32:58,391 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:work-106) Testing if VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d] is alive
  121. 2013-08-05 11:32:58,394 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:work-106) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d] returning null
  122. 2013-08-05 11:32:58,394 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) null found VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]to be alive? null
  123. 2013-08-05 11:32:58,398 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) Fencing off VM that we don't know the state of
  124. 2013-08-05 11:32:58,398 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-1:work-105) Don't know how to fence non XenServer hosts KVM
  125. 2013-08-05 11:32:58,401 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) Fencer null returned null
  126. 2013-08-05 11:32:58,404 DEBUG [agent.transport.Request] (HA-Worker-1:work-105) Seq 1-204538716: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"FenceCommand":{"vmName":"i-2-23-VM","hostGuid":"945a8003-3d66-335b-b49d-dc4e6e40f46e-LibvirtComputingResource","hostIp":"192.168.122.32","inSeq":false,"wait":0}}] }
  127. 2013-08-05 11:32:58,404 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) Fencing off VM that we don't know the state of
  128. 2013-08-05 11:32:58,405 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-2:work-106) Don't know how to fence non XenServer hosts KVM
  129. 2013-08-05 11:32:58,405 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) Fencer null returned null
  130. 2013-08-05 11:32:58,408 DEBUG [agent.transport.Request] (HA-Worker-2:work-106) Seq 1-204538717: Sending { Cmd , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 100011, [{"FenceCommand":{"vmName":"i-2-25-VM","hostGuid":"945a8003-3d66-335b-b49d-dc4e6e40f46e-LibvirtComputingResource","hostIp":"192.168.122.32","inSeq":false,"wait":0}}] }
  131. 2013-08-05 11:32:58,535 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 1-204538716: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"FenceAnswer":{"result":false,"details":"Heart is still beating...","wait":0}}] }
  132. 2013-08-05 11:32:58,535 DEBUG [agent.transport.Request] (HA-Worker-1:work-105) Seq 1-204538716: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { FenceAnswer } }
  133. 2013-08-05 11:32:58,535 DEBUG [cloud.ha.KVMFencer] (HA-Worker-1:work-105) Unable to fence off VM[User|28a63a18-9969-4546-9463-f1c8828831b3] on Host[-7-Routing]
  134. 2013-08-05 11:32:58,536 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) Fencer KVMFenceBuilder returned false
  135. 2013-08-05 11:32:58,535 DEBUG [agent.transport.Request] (AgentManager-Handler-4:null) Seq 1-204538717: Processing: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, [{"FenceAnswer":{"result":false,"details":"Heart is still beating...","wait":0}}] }
  136. 2013-08-05 11:32:58,536 DEBUG [ovm.hypervisor.OvmFencer] (HA-Worker-1:work-105) Don't know how to fence non Ovm hosts KVM
  137. 2013-08-05 11:32:58,537 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) Fencer OvmFenceBuilder returned null
  138. 2013-08-05 11:32:58,537 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) We were unable to fence off the VM VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  139. 2013-08-05 11:32:58,539 DEBUG [agent.transport.Request] (HA-Worker-2:work-106) Seq 1-204538717: Received: { Ans: , MgmtId: 90520732148205, via: 1, Ver: v1, Flags: 10, { FenceAnswer } }
  140. 2013-08-05 11:32:58,539 DEBUG [cloud.ha.KVMFencer] (HA-Worker-2:work-106) Unable to fence off VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d] on Host[-7-Routing]
  141. 2013-08-05 11:32:58,545 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) Fencer KVMFenceBuilder returned false
  142. 2013-08-05 11:32:58,547 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-105) Rescheduling HAWork[105-HA-23-Running-Investigating] to try again at Mon Aug 05 11:43:12 CEST 2013
  143. 2013-08-05 11:32:58,548 DEBUG [ovm.hypervisor.OvmFencer] (HA-Worker-2:work-106) Don't know how to fence non Ovm hosts KVM
  144. 2013-08-05 11:32:58,549 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) Fencer OvmFenceBuilder returned null
  145. 2013-08-05 11:32:58,551 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) We were unable to fence off the VM VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]
  146. 2013-08-05 11:32:58,555 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:work-106) Rescheduling HAWork[106-HA-25-Running-Investigating] to try again at Mon Aug 05 11:43:12 CEST 2013
  147.  
  148. 2013-08-05 11:43:49,650 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) Processing HAWork[105-HA-23-Running-Investigating]
  149. 2013-08-05 11:43:49,656 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) Processing HAWork[104-HA-19-Running-Investigating]
  150. 2013-08-05 11:43:49,661 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) HA on VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  151. 2013-08-05 11:43:49,661 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) HA on VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  152. 2013-08-05 11:43:49,666 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-3:work-105) Unable to reach the agent for VM[User|28a63a18-9969-45
  153. 46-9463-f1c8828831b3]: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  154. 2013-08-05 11:43:49,667 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) SimpleInvestigator found VM[User|28a63a18-9969-4546
  155. -9463-f1c8828831b3]to be alive? null
  156. 2013-08-05 11:43:49,667 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) XenServerInvestigator found VM[User|28a63a18-9969-4
  157. 546-9463-f1c8828831b3]to be alive? null
  158. 2013-08-05 11:43:49,667 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-3:work-105) testing if VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  159. is alive
  160. 2013-08-05 11:43:49,667 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-4:work-104) Unable to reach the agent for VM[User|ca8d1be8-7927-4e
  161. ff-b9f8-5a68e389e733]: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  162. 2013-08-05 11:43:49,668 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) SimpleInvestigator found VM[User|ca8d1be8-7927-4eff
  163. -b9f8-5a68e389e733]to be alive? null
  164. 2013-08-05 11:43:49,668 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) XenServerInvestigator found VM[User|ca8d1be8-7927-4
  165. eff-b9f8-5a68e389e733]to be alive? null
  166. 2013-08-05 11:43:49,668 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-104) testing if VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  167. is alive
  168. 2013-08-05 11:43:49,677 DEBUG [agent.transport.Request] (HA-Worker-3:work-105) Seq 1-204538739: Sending { Cmd , MgmtId: 90520732148205, via: 1
  169. , Ver: v1, Flags: 100011, [{"PingTestCommand":{"_routerIp":"169.254.1.243","_privateIp":"192.168.122.133","wait":20}}] }
  170. 2013-08-05 11:43:49,678 DEBUG [agent.transport.Request] (HA-Worker-4:work-104) Seq 1-204538740: Sending { Cmd , MgmtId: 90520732148205, via: 1
  171. , Ver: v1, Flags: 100011, [{"PingTestCommand":{"_routerIp":"169.254.1.243","_privateIp":"192.168.122.112","wait":20}}] }
  172. 2013-08-05 11:43:51,203 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 9
  173. 2013-08-05 11:43:52,703 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-9:null) SeqA 9-50081: Processing Seq 9-50081: { Cmd , Mgm
  174. tId: -1, via: 9, Ver: v1, Flags: 11, [{"ConsoleProxyLoadReportCommand":{"_proxyVmId":28,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] }
  175. 2013-08-05 11:43:52,706 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-9:null) SeqA 9-50081: Sending Seq 9-50081: { Ans: , MgmtI
  176. d: 90520732148205, via: 9, Ver: v1, Flags: 100010, [{"AgentControlAnswer":{"result":true,"wait":0}}] }
  177. 2013-08-05 11:43:54,243 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-106) Processing HAWork[106-HA-25-Running-Investigating]
  178. 2013-08-05 11:43:54,250 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-106) HA on VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]
  179. 2013-08-05 11:43:54,257 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-0:work-106) Unable to reach the agent for VM[User|c88924e9-a8c9-47
  180. 05-acc8-3237ffcf009d]: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  181. 2013-08-05 11:43:54,258 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-106) SimpleInvestigator found VM[User|c88924e9-a8c9-4705
  182. -acc8-3237ffcf009d]to be alive? null
  183. 2013-08-05 11:43:54,259 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-106) XenServerInvestigator found VM[User|c88924e9-a8c9-4
  184. 705-acc8-3237ffcf009d]to be alive? null
  185. 2013-08-05 11:43:54,259 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-0:work-106) testing if VM[User|c88924e9-a8c9-4705-acc8-3237ffcf009d]
  186. is alive
  187. 2013-08-05 11:43:54,266 DEBUG [agent.transport.Request] (HA-Worker-0:work-106) Seq 1-204538741: Sending { Cmd , MgmtId: 90520732148205, via: 1
  188. , Ver: v1, Flags: 100011, [{"PingTestCommand":{"_routerIp":"169.254.1.243","_privateIp":"192.168.122.170","wait":20}}] }
  189. 2013-08-05 11:43:54,315 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq 1-204538739: Processing: { Ans: , MgmtId: 9052073214
  190. 8205, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"PING 192.168.122.133 (192.168.122.133): 56 data bytes64 bytes from 192.
  191. 168.122.151: Destination Host UnreachableVr HL TOS Len ID Flg off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 3
  192. bc4 192.168.122.151 192.168.122.133 --- 192.168.122.133 ping statistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable t
  193. o ping the vm, exiting","wait":0}}] }
  194. 2013-08-05 11:43:54,316 DEBUG [agent.transport.Request] (HA-Worker-3:work-105) Seq 1-204538739: Received: { Ans: , MgmtId: 90520732148205, via
  195. : 1, Ver: v1, Flags: 10, { Answer } }
  196. 2013-08-05 11:43:54,316 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-3:work-105) Details from executing class com.cloud.agent.api.PingTest
  197. Command: PING 192.168.122.133 (192.168.122.133): 56 data bytes64 bytes from 192.168.122.151: Destination Host UnreachableVr HL TOS Len ID Fl
  198. g off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 3bc4 192.168.122.151 192.168.122.133 --- 192.168.122.133 ping s
  199. tatistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable to ping the vm, exiting
  200. 2013-08-05 11:43:54,316 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-3:work-105) VM[User|28a63a18-9969-4546-9463-f1c8828831b3] could not
  201. be pinged, returning that it is unknown
  202. 2013-08-05 11:43:54,316 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-3:work-105) Returning null since we're unable to determine state of
  203. VM[User|28a63a18-9969-4546-9463-f1c8828831b3]
  204. 2013-08-05 11:43:54,316 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) null found VM[User|28a63a18-9969-4546-9463-f1c88288
  205. 31b3]to be alive? null
  206. 2013-08-05 11:43:54,317 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-3:work-105) Not a System Vm, unable to determine state of
  207. VM[User|28a63a18-9969-4546-9463-f1c8828831b3] returning null
  208. 2013-08-05 11:43:54,317 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-3:work-105) Testing if VM[User|28a63a18-9969-4546-9463-f1c
  209. 8828831b3] is alive
  210. 2013-08-05 11:43:54,316 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-204538740: Processing: { Ans: , MgmtId: 9052073214
  211. 8205, via: 1, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"PING 192.168.122.112 (192.168.122.112): 56 data bytes64 bytes from 192.
  212. 168.122.151: Destination Host UnreachableVr HL TOS Len ID Flg off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 5
  213. 0c4 192.168.122.151 192.168.122.112 --- 192.168.122.112 ping statistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable t
  214. o ping the vm, exiting","wait":0}}] }
  215. 2013-08-05 11:43:54,317 DEBUG [agent.transport.Request] (HA-Worker-4:work-104) Seq 1-204538740: Received: { Ans: , MgmtId: 90520732148205, via
  216. : 1, Ver: v1, Flags: 10, { Answer } }
  217. 2013-08-05 11:43:54,317 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-4:work-104) Details from executing class com.cloud.agent.api.PingTest
  218. Command: PING 192.168.122.112 (192.168.122.112): 56 data bytes64 bytes from 192.168.122.151: Destination Host UnreachableVr HL TOS Len ID Fl
  219. g off TTL Pro cks Src Dst Data 4 5 00 5400 0000 0 0040 40 01 50c4 192.168.122.151 192.168.122.112 --- 192.168.122.112 ping s
  220. tatistics ---1 packets transmitted, 0 packets received, 100% packet lossUnable to ping the vm, exiting
  221. 2013-08-05 11:43:54,318 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-104) VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] could not
  222. be pinged, returning that it is unknown
  223. 2013-08-05 11:43:54,318 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-4:work-104) Returning null since we're unable to determine state of
  224. VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  225. 2013-08-05 11:43:54,318 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) null found VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389
  226. e733]to be alive? null
  227. 2013-08-05 11:43:54,318 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-4:work-104) Not a System Vm, unable to determine state of
  228. VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] returning null
  229. 2013-08-05 11:43:54,318 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-4:work-104) Testing if VM[User|ca8d1be8-7927-4eff-b9f8-5a6
  230. 8e389e733] is alive
  231. 2013-08-05 11:43:54,322 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-3:work-105) Unable to find a management nic, cannot ping t
  232. his system VM, unable to determine state of VM[User|28a63a18-9969-4546-9463-f1c8828831b3] returning null
  233. 2013-08-05 11:43:54,322 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) null found VM[User|28a63a18-9969-4546-9463-f1c88288
  234. 31b3]to be alive? null
  235. 2013-08-05 11:43:54,322 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) Fencing off VM that we don't know the state of
  236. 2013-08-05 11:43:54,323 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-3:work-105) Don't know how to fence non XenServer hosts KVM
  237. 2013-08-05 11:43:54,323 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) Fencer null returned null
  238. 2013-08-05 11:43:54,322 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-4:work-104) Unable to find a management nic, cannot ping t
  239. his system VM, unable to determine state of VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] returning null
  240. 2013-08-05 11:43:54,323 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) null found VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389
  241. e733]to be alive? null
  242. 2013-08-05 11:43:54,324 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) Fencing off VM that we don't know the state of
  243. 2013-08-05 11:43:54,324 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-4:work-104) Don't know how to fence non XenServer hosts KVM
  244. 2013-08-05 11:43:54,324 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) Fencer null returned null
  245. 2013-08-05 11:43:54,328 DEBUG [agent.transport.Request] (HA-Worker-3:work-105) Seq 1-204538742: Sending { Cmd , MgmtId: 90520732148205, via: 1
  246. , Ver: v1, Flags: 100011, [{"FenceCommand":{"vmName":"i-2-23-VM","hostGuid":"945a8003-3d66-335b-b49d-dc4e6e40f46e-LibvirtComputingResource","ho
  247. stIp":"192.168.122.32","inSeq":false,"wait":0}}] }
  248. 2013-08-05 11:43:54,329 DEBUG [agent.transport.Request] (HA-Worker-4:work-104) Seq 1-204538743: Sending { Cmd , MgmtId: 90520732148205, via: 1
  249. , Ver: v1, Flags: 100011, [{"FenceCommand":{"vmName":"i-2-19-VM","hostGuid":"945a8003-3d66-335b-b49d-dc4e6e40f46e-LibvirtComputingResource","ho
  250. stIp":"192.168.122.32","inSeq":false,"wait":0}}] }
  251. 2013-08-05 11:43:54,407 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 1-204538743: Processing: { Ans: , MgmtId: 905207321
  252. 48205, via: 1, Ver: v1, Flags: 10, [{"FenceAnswer":{"result":true,"wait":0}}] }
  253. 2013-08-05 11:43:54,407 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-204538742: Processing: { Ans: , MgmtId: 9052073214
  254. 8205, via: 1, Ver: v1, Flags: 10, [{"FenceAnswer":{"result":true,"wait":0}}] }
  255. 2013-08-05 11:43:54,407 DEBUG [agent.transport.Request] (HA-Worker-3:work-105) Seq 1-204538742: Received: { Ans: , MgmtId: 90520732148205, via
  256. : 1, Ver: v1, Flags: 10, { FenceAnswer } }
  257. 2013-08-05 11:43:54,408 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-105) Fencer KVMFenceBuilder returned true
  258. 2013-08-05 11:43:54,408 DEBUG [agent.transport.Request] (HA-Worker-4:work-104) Seq 1-204538743: Received: { Ans: , MgmtId: 90520732148205, via
  259. : 1, Ver: v1, Flags: 10, { FenceAnswer } }
  260. 2013-08-05 11:43:54,408 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-104) Fencer KVMFenceBuilder returned true
  261. 2013-08-05 11:43:54,416 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-104) VM state transitted from :Running to Stopping with ev
  262. ent: StopRequestedvm's original host id: 1 new host id: 7 host id before state transition: 7
  263. 2013-08-05 11:43:54,416 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-3:work-105) VM state transitted from :Running to Stopping with ev
  264. ent: StopRequestedvm's original host id: 1 new host id: 7 host id before state transition: 7
  265. 2013-08-05 11:43:54,420 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Unable to stop vm, agent unavailable: com.cloud.excep
  266. tion.AgentUnavailableException: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  267. 2013-08-05 11:43:54,421 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Unable to actually stop VM[User|ca8d1be8-7927-4eff-b9
  268. f8-5a68e389e733] but continue with release because it's a force stop
  269. 2013-08-05 11:43:54,420 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-105) Unable to stop vm, agent unavailable: com.cloud.excep
  270. tion.AgentUnavailableException: Resource [Host:7] is unreachable: Host 7: Host with specified id is not in the right state: Alert
  271. 2013-08-05 11:43:54,421 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-105) Unable to actually stop VM[User|28a63a18-9969-4546-94
  272. 63-f1c8828831b3] but continue with release because it's a force stop
  273. 2013-08-05 11:43:54,423 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-105) VM[User|28a63a18-9969-4546-9463-f1c8828831b3] is stop
  274. ped on the host. Proceeding to release resource held.
  275. 2013-08-05 11:43:54,424 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) VM[User|ca8d1be8-7927-4eff-b9f8-5a68e389e733] is stop
  276. ped on the host. Proceeding to release resource held.
  277. 2013-08-05 11:43:54,436 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-104) Changing active number of nics for network id=204 on -1
  278. 2013-08-05 11:43:54,436 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-3:work-105) Changing active number of nics for network id=204 on -1
  279. 2013-08-05 11:43:54,440 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-104) Asking VirtualRouter to release Nic[66-19-78f7d9c1-9668
  280. -486f-8351-323d2471b34b-192.168.122.112]
  281. 2013-08-05 11:43:54,440 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-104) Asking Ovs to release Nic[66-19-78f7d9c1-9668-486f-8351
  282. -323d2471b34b-192.168.122.112]
  283. 2013-08-05 11:43:54,441 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-104) Asking SecurityGroupProvider to release Nic[66-19-78f7d
  284. 9c1-9668-486f-8351-323d2471b34b-192.168.122.112]
  285. 2013-08-05 11:43:54,441 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-104) Asking VpcVirtualRouter to release Nic[66-19-78f7d9c1-9
  286. 668-486f-8351-323d2471b34b-192.168.122.112]
  287. 2013-08-05 11:43:54,441 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-4:work-104) Asking NiciraNvp to release Nic[66-19-78f7d9c1-9668-486
  288. f-8351-323d2471b34b-192.168.122.112]
  289. 2013-08-05 11:43:54,441 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-3:work-105) Asking VirtualRouter to release Nic[74-23-884e62bb-9cd6
  290. -4880-8232-13692d151df3-192.168.122.133]
  291. 2013-08-05 11:43:54,441 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-3:work-105) Asking Ovs to release Nic[74-23-884e62bb-9cd6-4880-8232
  292. -13692d151df3-192.168.122.133]
  293. 2013-08-05 11:43:54,442 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-3:work-105) Asking SecurityGroupProvider to release Nic[74-23-884e6
  294. 2bb-9cd6-4880-8232-13692d151df3-192.168.122.133]
  295. 2013-08-05 11:43:54,442 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-3:work-105) Asking VpcVirtualRouter to release Nic[74-23-884e62bb-9
  296. cd6-4880-8232-13692d151df3-192.168.122.133]
  297. 2013-08-05 11:43:54,442 DEBUG [cloud.network.NetworkManagerImpl] (HA-Worker-3:work-105) Asking NiciraNvp to release Nic[74-23-884e62bb-9cd6-488
  298. 0-8232-13692d151df3-192.168.122.133]
  299. 2013-08-05 11:43:54,443 DEBUG [network.element.NiciraNvpElement] (HA-Worker-3:work-105) Checking if NiciraNvpElement can handle service Connect
  300. ivity on network defaultGuestNetwork
  301. 2013-08-05 11:43:54,443 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-105) Successfully released network resources for the vm VM
  302. [User|28a63a18-9969-4546-9463-f1c8828831b3]
  303. 2013-08-05 11:43:54,443 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-105) Successfully released storage resources for the vm VM
  304. [User|28a63a18-9969-4546-9463-f1c8828831b3]
  305. 2013-08-05 11:43:54,441 DEBUG [network.element.NiciraNvpElement] (HA-Worker-4:work-104) Checking if NiciraNvpElement can handle service Connect
  306. ivity on network defaultGuestNetwork
  307. 2013-08-05 11:43:54,444 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Successfully released network resources for the vm VM
  308. [User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  309. 2013-08-05 11:43:54,444 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Successfully released storage resources for the vm VM
  310. [User|ca8d1be8-7927-4eff-b9f8-5a68e389e733]
  311. 2013-08-05 11:43:54,508 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-104) VM state transitted from :Stopping to Stopped with ev
  312. ent: OperationSucceededvm's original host id: 1 new host id: null host id before state transition: 7
  313. 2013-08-05 11:43:54,509 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-3:work-105) VM state transitted from :Stopping to Stopped with ev
  314. ent: OperationSucceededvm's original host id: 1 new host id: null host id before state transition: 7
  315. 2013-08-05 11:43:54,513 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-104) Hosts's actual total CPU: 3299 and CPU after applying
  316. overprovisioning: 3299
  317. 2013-08-05 11:43:54,514 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-104) release cpu from host: 7, old used: 1500,reserved: 0,
  318. actual total: 3299, total with overprovisioning: 3299; new used: 1000,reserved:500; movedfromreserved: false,moveToReserveredtrue
  319. 2013-08-05 11:43:54,514 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-104) release mem from host: 7, old used: 1610612736,reserv
  320. ed: 0, total: 4145119232; new used: 1073741824,reserved:536870912; movedfromreserved: false,moveToReserveredtrue
  321. 2013-08-05 11:43:54,516 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-3:work-105) Hosts's actual total CPU: 3299 and CPU after applying
  322. overprovisioning: 3299
  323. 2013-08-05 11:43:54,517 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-3:work-105) release cpu from host: 7, old used: 1000,reserved: 50
  324. 0, actual total: 3299, total with overprovisioning: 3299; new used: 500,reserved:1000; movedfromreserved: false,moveToReserveredtrue
  325. 2013-08-05 11:43:54,517 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-3:work-105) release mem from host: 7, old used: 1073741824,reserv
  326. ed: 536870912, total: 4145119232; new used: 536870912,reserved:1073741824; movedfromreserved: false,moveToReserveredtrue
  327. 2013-08-05 11:43:54,539 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-4:work-104) VM state transitted from :Stopped to Starting with ev
  328. ent: StartRequestedvm's original host id: 1 new host id: null host id before state transition: null
  329. 2013-08-05 11:43:54,539 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Successfully transitioned to start state for VM[User|
  330. ca8d1be8-7927-4eff-b9f8-5a68e389e733] reservation id = 9d90ccf4-4f8e-4517-960a-858a0dbd3eeb
  331. 2013-08-05 11:43:54,543 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Trying to deploy VM, vm has dcId: 1 and podId: 1
  332. 2013-08-05 11:43:54,543 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-104) Deploy avoids pods: null, clusters: null, hosts: null
  333. 2013-08-05 11:43:54,546 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-3:work-105) VM state transitted from :Stopped to Starting with ev
  334. ent: StartRequestedvm's original host id: 1 new host id: null host id before state transition: null
  335. 2013-08-05 11:43:54,546 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-105) Successfully transitioned to start state for VM[User|
  336. 28a63a18-9969-4546-9463-f1c8828831b3] reservation id = 9dd9a9fd-6910-4477-867d-7d8365e1db5b
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!