API tools faq
paste
Guest User

Untitled

a guest
May 23rd, 2014
196
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 35.12 KB | None | 0 0
raw download clone embed print report
  1. 2014-05-23 10:28:28,017 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-9cc9198a) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2. 2014-05-23 10:28:28,018 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-9cc9198a) Seq 3-785580946: Timed out on null
  3. 2014-05-23 10:28:28,018 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-9cc9198a) Seq 3-785580946: Cancelling.
  4. 2014-05-23 10:28:28,018 DEBUG [o.a.c.s.RemoteHostEndPoint] (StatsCollector-3:ctx-9cc9198a) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 785580946 to Host 3 timed out after 3600
  5. 2014-05-23 10:28:28,018 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-9cc9198a) Error trying to retrieve storage stats
  6. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 785580946 to Host 3 timed out after 3600
  7. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:116)
  8. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:550)
  9. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  10. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  11. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  12. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  13. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  14. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  15. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  16. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  17. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  18. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  19. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  20. at java.lang.Thread.run(Thread.java:744)
  21. 2014-05-23 10:28:33,773 WARN [c.c.u.n.Link] (AgentManager-Selector:null) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
  22. 2014-05-23 10:28:33,965 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) Seq 0-10724: Scheduling the first command { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":8,"speed":2400,"memory":67522400256,"dom0MinMemory":805306368,"poolSync":false,"_hostVmStateReport":{},"vms":{},"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS":"CentOS","Host.OS.Kernel.Version":"2.6.32-431.11.2.el6.x86_64","Host.OS.Version":"6.5"},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","iqn":"iqn.1994-05.com.redhat:2f10685539e","publicIpAddress":"192.168.100.6","publicNetmask":"255.255.255.0","publicMacAddress":"00:25:90:c9:6d:30","privateIpAddress":"192.168.100.6","privateMacAddress":"00:25:90:c9:6d:30","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.100.6","storageNetmask":"255.255.255.0","storageMacAddress":"00:25:90:c9:6d:30","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.100.1","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"e31690c1-49b6-47b9-bfeb-b884f5f64260","host":"192.168.100.6","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":1927565250560,"availableBytes":1654390530048},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","resourceName":"LibvirtComputingResource","wait":0}}] }
  23. 2014-05-23 10:28:33,966 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-5363:ctx-e5d22049) Seq 0-10724: Processing the first command { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":8,"speed":2400,"memory":67522400256,"dom0MinMemory":805306368,"poolSync":false,"_hostVmStateReport":{},"vms":{},"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS":"CentOS","Host.OS.Kernel.Version":"2.6.32-431.11.2.el6.x86_64","Host.OS.Version":"6.5"},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","iqn":"iqn.1994-05.com.redhat:2f10685539e","publicIpAddress":"192.168.100.6","publicNetmask":"255.255.255.0","publicMacAddress":"00:25:90:c9:6d:30","privateIpAddress":"192.168.100.6","privateMacAddress":"00:25:90:c9:6d:30","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.100.6","storageNetmask":"255.255.255.0","storageMacAddress":"00:25:90:c9:6d:30","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.100.1","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"e31690c1-49b6-47b9-bfeb-b884f5f64260","host":"192.168.100.6","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":1927565250560,"availableBytes":1654390530048},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","resourceName":"LibvirtComputingResource","wait":0}}] }
  24. 2014-05-23 10:28:33,969 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalDhcpManagerImpl
  25. 2014-05-23 10:28:33,969 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
  26. 2014-05-23 10:28:33,969 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
  27. 2014-05-23 10:28:33,969 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PaloAltoExternalFirewallElement
  28. 2014-05-23 10:28:33,969 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
  29. 2014-05-23 10:28:33,978 DEBUG [c.c.r.ResourceState] (AgentConnectTaskPool-5363:ctx-e5d22049) Resource state update: [id = 1; name = virthost1.redacted.com; old state = Enabled; event = InternalCreated; new state = Enabled]
  30. 2014-05-23 10:28:33,978 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5363:ctx-e5d22049) Transition:[Resource state = Enabled, Agent event = AgentConnected, Host id = 1, name = virthost1.redacted.com]
  31. 2014-05-23 10:28:33,980 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5363:ctx-e5d22049) Agent status update: [id = 1; name = virthost1.redacted.com; old status = Alert; event = AgentConnected; new status = Connecting; old update count = 11591; new update count = 11592]
  32. 2014-05-23 10:28:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) create ClusteredAgentAttache for 1
  33. 2014-05-23 10:28:33,980 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Connect to listener: XcpServerDiscoverer
  34. 2014-05-23 10:28:33,980 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-5363:ctx-e5d22049) Not XenServer so moving on.
  35. 2014-05-23 10:28:33,980 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Connect to listener: HypervServerDiscoverer
  36. 2014-05-23 10:28:33,980 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-5363:ctx-e5d22049) Not Hyper-V hypervisor, so moving on.
  37. 2014-05-23 10:28:33,980 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Connect to listener: ClusteredVirtualMachineManagerImpl
  38. 2014-05-23 10:28:33,980 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Received startup command from hypervisor host. host id: 1
  39. 2014-05-23 10:28:33,982 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Found 2 VMs for host 1
  40. 2014-05-23 10:28:33,983 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) VM v-2-VM: cs state = Starting and realState = Stopped
  41. 2014-05-23 10:28:33,983 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) VM v-2-VM: cs state = Starting and realState = Stopped
  42. 2014-05-23 10:28:33,983 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Ignoring VM in starting mode: v-2-VM
  43. 2014-05-23 10:28:33,983 DEBUG [c.c.h.HighAvailabilityManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) VM does not require investigation so I'm marking it as Stopped: VM[ConsoleProxy|v-2-VM]
  44. 2014-05-23 10:28:33,983 WARN [o.a.c.alerts] (AgentConnectTaskPool-5363:ctx-e5d22049) alertType:: 10 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: VM (name: v-2-VM, id: 2) stopped unexpectedly on host id:1, availability zone id:1, pod id:1
  45. 2014-05-23 10:28:33,985 DEBUG [c.c.h.HighAvailabilityManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) VM is not HA enabled so we're done.
  46. 2014-05-23 10:28:33,985 WARN [o.a.c.f.j.AsyncJobExecutionContext] (AgentConnectTaskPool-5363:ctx-e5d22049) Job is executed without a context, setup psudo job for the executing thread
  47. 2014-05-23 10:28:33,989 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Unable to transition the state but we're moving on because it's forced stop
  48. 2014-05-23 10:28:33,989 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Unable to cleanup VM: VM[ConsoleProxy|v-2-VM] ,since outstanding work item is not found
  49. 2014-05-23 10:28:33,989 ERROR [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Monitor ClusteredVirtualMachineManagerImpl says there is an error in the connect process for 1 due to Work item not found, We cannot stop VM[ConsoleProxy|v-2-VM] when it is in state Starting
  50. com.cloud.utils.exception.CloudRuntimeException: Work item not found, We cannot stop VM[ConsoleProxy|v-2-VM] when it is in state Starting
  51. at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1415)
  52. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1344)
  53. at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1312)
  54. at com.cloud.ha.HighAvailabilityManagerImpl.scheduleRestart(HighAvailabilityManagerImpl.java:346)
  55. at com.cloud.vm.VirtualMachineManagerImpl.compareState(VirtualMachineManagerImpl.java:2827)
  56. at com.cloud.vm.VirtualMachineManagerImpl.fullHostSync(VirtualMachineManagerImpl.java:2384)
  57. at com.cloud.vm.VirtualMachineManagerImpl.processConnect(VirtualMachineManagerImpl.java:3035)
  58. at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:495)
  59. at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:999)
  60. at com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:117)
  61. at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1082)
  62. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  63. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  64. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  65. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  66. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  67. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  68. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  69. at java.lang.Thread.run(Thread.java:744)
  70. 2014-05-23 10:28:33,989 INFO [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Host 1 is disconnecting with event AgentDisconnected
  71. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) The next status of agent 1is Alert, current status is Connecting
  72. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Deregistering link for 1 with state Alert
  73. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Remove Agent : 1
  74. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.ConnectedAgentAttache] (AgentConnectTaskPool-5363:ctx-e5d22049) Processing Disconnect.
  75. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
  76. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
  77. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
  78. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
  79. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
  80. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
  81. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
  82. 2014-05-23 10:28:33,990 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
  83. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl
  84. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener
  85. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
  86. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
  87. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  88. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
  89. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
  90. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
  91. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
  92. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
  93. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  94. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl
  95. 2014-05-23 10:28:33,991 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
  96. 2014-05-23 10:28:33,991 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Disconnected called on 1 with status Alert
  97. 2014-05-23 10:28:33,991 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5363:ctx-e5d22049) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = virthost1.redacted.com]
  98. 2014-05-23 10:28:33,993 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5363:ctx-e5d22049) Agent status update: [id = 1; name = virthost1.redacted.com; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 11592; new update count = 11593]
  99. 2014-05-23 10:28:33,993 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Notifying other nodes of to disconnect
  100. 2014-05-23 10:28:33,994 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Failed to handle host connection: com.cloud.utils.exception.CloudRuntimeException: Unable to connect 1
  101. 2014-05-23 10:28:33,995 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5363:ctx-e5d22049) Can not send command com.cloud.agent.api.ReadyCommand due to Host 1 is not up
  102. 2014-05-23 10:28:34,674 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5ff0b6da) Found 0 routers to update status.
  103. 2014-05-23 10:28:34,676 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5ff0b6da) Found 0 networks to update RvR status.
  104. 2014-05-23 10:28:39,606 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-d48b124d) Resetting hosts suitable for reconnect
  105. 2014-05-23 10:28:39,607 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-d48b124d) Completed resetting hosts suitable for reconnect
  106. 2014-05-23 10:28:39,607 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-d48b124d) Acquiring hosts for clusters already owned by this management server
  107. 2014-05-23 10:28:39,607 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-d48b124d) Completed acquiring hosts for clusters already owned by this management server
  108. 2014-05-23 10:28:39,607 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-d48b124d) Acquiring hosts for clusters not owned by any management server
  109. 2014-05-23 10:28:39,607 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-d48b124d) Completed acquiring hosts for clusters not owned by any management server
  110. 2014-05-23 10:28:43,993 WARN [c.c.u.n.Link] (AgentManager-Selector:null) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
  111. 2014-05-23 10:28:44,184 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) Seq 0-10726: Scheduling the first command { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":8,"speed":2400,"memory":67522400256,"dom0MinMemory":805306368,"poolSync":false,"_hostVmStateReport":{},"vms":{},"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS":"CentOS","Host.OS.Kernel.Version":"2.6.32-431.11.2.el6.x86_64","Host.OS.Version":"6.5"},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","iqn":"iqn.1994-05.com.redhat:2f10685539e","publicIpAddress":"192.168.100.6","publicNetmask":"255.255.255.0","publicMacAddress":"00:25:90:c9:6d:30","privateIpAddress":"192.168.100.6","privateMacAddress":"00:25:90:c9:6d:30","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.100.6","storageNetmask":"255.255.255.0","storageMacAddress":"00:25:90:c9:6d:30","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.100.1","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"e31690c1-49b6-47b9-bfeb-b884f5f64260","host":"192.168.100.6","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":1927565250560,"availableBytes":1654390530048},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","resourceName":"LibvirtComputingResource","wait":0}}] }
  112. 2014-05-23 10:28:44,186 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-5364:ctx-ac137393) Seq 0-10726: Processing the first command { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":8,"speed":2400,"memory":67522400256,"dom0MinMemory":805306368,"poolSync":false,"_hostVmStateReport":{},"vms":{},"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS":"CentOS","Host.OS.Kernel.Version":"2.6.32-431.11.2.el6.x86_64","Host.OS.Version":"6.5"},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","iqn":"iqn.1994-05.com.redhat:2f10685539e","publicIpAddress":"192.168.100.6","publicNetmask":"255.255.255.0","publicMacAddress":"00:25:90:c9:6d:30","privateIpAddress":"192.168.100.6","privateMacAddress":"00:25:90:c9:6d:30","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.100.6","storageNetmask":"255.255.255.0","storageMacAddress":"00:25:90:c9:6d:30","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.100.1","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"e31690c1-49b6-47b9-bfeb-b884f5f64260","host":"192.168.100.6","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":1927565250560,"availableBytes":1654390530048},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"de506c9b-50d1-36bc-8fc9-e8fb24a51ca6-LibvirtComputingResource","name":"virthost1.redacted.com","id":0,"version":"4.3.0","resourceName":"LibvirtComputingResource","wait":0}}] }
  113. 2014-05-23 10:28:44,189 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalDhcpManagerImpl
  114. 2014-05-23 10:28:44,189 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
  115. 2014-05-23 10:28:44,189 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
  116. 2014-05-23 10:28:44,189 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PaloAltoExternalFirewallElement
  117. 2014-05-23 10:28:44,189 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
  118. 2014-05-23 10:28:44,199 DEBUG [c.c.r.ResourceState] (AgentConnectTaskPool-5364:ctx-ac137393) Resource state update: [id = 1; name = virthost1.redacted.com; old state = Enabled; event = InternalCreated; new state = Enabled]
  119. 2014-05-23 10:28:44,199 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5364:ctx-ac137393) Transition:[Resource state = Enabled, Agent event = AgentConnected, Host id = 1, name = virthost1.redacted.com]
  120. 2014-05-23 10:28:44,200 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5364:ctx-ac137393) Agent status update: [id = 1; name = virthost1.redacted.com; old status = Alert; event = AgentConnected; new status = Connecting; old update count = 11593; new update count = 11594]
  121. 2014-05-23 10:28:44,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) create ClusteredAgentAttache for 1
  122. 2014-05-23 10:28:44,201 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Connect to listener: XcpServerDiscoverer
  123. 2014-05-23 10:28:44,201 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-5364:ctx-ac137393) Not XenServer so moving on.
  124. 2014-05-23 10:28:44,201 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Connect to listener: HypervServerDiscoverer
  125. 2014-05-23 10:28:44,201 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-5364:ctx-ac137393) Not Hyper-V hypervisor, so moving on.
  126. 2014-05-23 10:28:44,201 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Connect to listener: ClusteredVirtualMachineManagerImpl
  127. 2014-05-23 10:28:44,201 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Received startup command from hypervisor host. host id: 1
  128. 2014-05-23 10:28:44,203 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Found 2 VMs for host 1
  129. 2014-05-23 10:28:44,203 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) VM v-2-VM: cs state = Starting and realState = Stopped
  130. 2014-05-23 10:28:44,203 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) VM v-2-VM: cs state = Starting and realState = Stopped
  131. 2014-05-23 10:28:44,203 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Ignoring VM in starting mode: v-2-VM
  132. 2014-05-23 10:28:44,203 DEBUG [c.c.h.HighAvailabilityManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) VM does not require investigation so I'm marking it as Stopped: VM[ConsoleProxy|v-2-VM]
  133. 2014-05-23 10:28:44,203 WARN [o.a.c.alerts] (AgentConnectTaskPool-5364:ctx-ac137393) alertType:: 10 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: VM (name: v-2-VM, id: 2) stopped unexpectedly on host id:1, availability zone id:1, pod id:1
  134. 2014-05-23 10:28:44,205 DEBUG [c.c.h.HighAvailabilityManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) VM is not HA enabled so we're done.
  135. 2014-05-23 10:28:44,205 WARN [o.a.c.f.j.AsyncJobExecutionContext] (AgentConnectTaskPool-5364:ctx-ac137393) Job is executed without a context, setup psudo job for the executing thread
  136. 2014-05-23 10:28:44,209 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Unable to transition the state but we're moving on because it's forced stop
  137. 2014-05-23 10:28:44,209 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Unable to cleanup VM: VM[ConsoleProxy|v-2-VM] ,since outstanding work item is not found
  138. 2014-05-23 10:28:44,209 ERROR [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Monitor ClusteredVirtualMachineManagerImpl says there is an error in the connect process for 1 due to Work item not found, We cannot stop VM[ConsoleProxy|v-2-VM] when it is in state Starting
  139. com.cloud.utils.exception.CloudRuntimeException: Work item not found, We cannot stop VM[ConsoleProxy|v-2-VM] when it is in state Starting
  140. at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1415)
  141. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1344)
  142. at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1312)
  143. at com.cloud.ha.HighAvailabilityManagerImpl.scheduleRestart(HighAvailabilityManagerImpl.java:346)
  144. at com.cloud.vm.VirtualMachineManagerImpl.compareState(VirtualMachineManagerImpl.java:2827)
  145. at com.cloud.vm.VirtualMachineManagerImpl.fullHostSync(VirtualMachineManagerImpl.java:2384)
  146. at com.cloud.vm.VirtualMachineManagerImpl.processConnect(VirtualMachineManagerImpl.java:3035)
  147. at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:495)
  148. at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:999)
  149. at com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:117)
  150. at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1082)
  151. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  152. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  153. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  154. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  155. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  156. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  157. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  158. at java.lang.Thread.run(Thread.java:744)
  159. 2014-05-23 10:28:44,209 INFO [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Host 1 is disconnecting with event AgentDisconnected
  160. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) The next status of agent 1is Alert, current status is Connecting
  161. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Deregistering link for 1 with state Alert
  162. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Remove Agent : 1
  163. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.ConnectedAgentAttache] (AgentConnectTaskPool-5364:ctx-ac137393) Processing Disconnect.
  164. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
  165. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
  166. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
  167. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
  168. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
  169. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
  170. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
  171. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
  172. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl
  173. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener
  174. 2014-05-23 10:28:44,210 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
  175. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
  176. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  177. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
  178. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
  179. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
  180. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
  181. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
  182. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
  183. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl
  184. 2014-05-23 10:28:44,211 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
  185. 2014-05-23 10:28:44,211 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Disconnected called on 1 with status Alert
  186. 2014-05-23 10:28:44,211 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5364:ctx-ac137393) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = virthost1.redacted.com]
  187. 2014-05-23 10:28:44,213 DEBUG [c.c.h.Status] (AgentConnectTaskPool-5364:ctx-ac137393) Agent status update: [id = 1; name = virthost1.redacted.com; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 11594; new update count = 11595]
  188. 2014-05-23 10:28:44,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Notifying other nodes of to disconnect
  189. 2014-05-23 10:28:44,214 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Failed to handle host connection: com.cloud.utils.exception.CloudRuntimeException: Unable to connect 1
  190. 2014-05-23 10:28:44,214 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5364:ctx-ac137393) Can not send command com.cloud.agent.api.ReadyCommand due to Host 1 is not up
  191. 2014-05-23 10:28:54,213 WARN [c.c.u.n.Link] (AgentManager-Selector:null) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!