API tools faq
paste
Guest User

Untitled

a guest
May 1st, 2014
270
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 46.08 KB | None | 0 0
raw download clone embed print report
  1. 2014-05-01 15:11:37,230 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  2. 2014-05-01 15:11:37,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  3. 2014-05-01 15:11:37,945 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
  4. 2014-05-01 15:11:38,201 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982298196
  5. 2014-05-01 15:11:38,231 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982298196
  6. 2014-05-01 15:11:41,204 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982301197
  7. 2014-05-01 15:11:41,233 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982301197
  8. 2014-05-01 15:11:44,201 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982304197
  9. 2014-05-01 15:11:44,228 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982304197
  10. 2014-05-01 15:11:47,202 DEBUG [cloud.api.ApiServlet] (catalina-exec-17:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982307197
  11. 2014-05-01 15:11:47,229 DEBUG [cloud.api.ApiServlet] (catalina-exec-17:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982307197
  12. 2014-05-01 15:11:50,203 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982310197
  13. 2014-05-01 15:11:50,234 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982310197
  14. 2014-05-01 15:11:53,033 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) VmStatsCollector is running...
  15. 2014-05-01 15:11:53,216 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982313198
  16. 2014-05-01 15:11:53,241 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982313198
  17. 2014-05-01 15:11:53,786 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) StorageCollector is running...
  18. 2014-05-01 15:11:53,799 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-3:null) Seq 2-1652621321: Forwarding null to 279278805450809
  19. 2014-05-01 15:11:53,801 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 2-1652621321: Routing from 196683258161390
  20. 2014-05-01 15:11:53,801 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 2-1652621321: Link is closed
  21. 2014-05-01 15:11:53,801 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 2-1652621321: MgmtId 196683258161390: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  22. 2014-05-01 15:11:53,802 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) Seq 2--1: MgmtId 196683258161390: Req: Routing to peer
  23. 2014-05-01 15:11:53,803 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) Seq 2--1: MgmtId 196683258161390: Req: Cancel request received
  24. 2014-05-01 15:11:53,803 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-3:null) Seq 2-1652621321: Cancelling.
  25. 2014-05-01 15:11:53,803 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621321: Waiting some more time because this is the current command
  26. 2014-05-01 15:11:53,803 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621321: Waiting some more time because this is the current command
  27. 2014-05-01 15:11:53,804 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  28. 2014-05-01 15:11:53,812 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621321: Timed out on null
  29. 2014-05-01 15:11:53,817 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621321: Cancelling.
  30. 2014-05-01 15:11:53,817 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:2, com.cloud.exception.OperationTimedoutException: Commands 1652621321 to Host 2 timed out after 3600
  31. 2014-05-01 15:11:53,817 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
  32. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:2, com.cloud.exception.OperationTimedoutException: Commands 1652621321 to Host 2 timed out after 3600
  33. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:104)
  34. at com.cloud.server.StatsCollector$StorageCollector.run(StatsCollector.java:548)
  35. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  36. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  37. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  38. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  39. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  40. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  41. at java.lang.Thread.run(Thread.java:744)
  42. 2014-05-01 15:11:56,203 DEBUG [cloud.api.ApiServlet] (catalina-exec-9:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982316199
  43. 2014-05-01 15:11:56,230 DEBUG [cloud.api.ApiServlet] (catalina-exec-9:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982316199
  44. 2014-05-01 15:11:57,378 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) HostStatsCollector is running...
  45. 2014-05-01 15:11:58,050 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-2001993746: Received: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  46. 2014-05-01 15:11:59,205 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982319199
  47. 2014-05-01 15:11:59,232 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982319199
  48. 2014-05-01 15:12:02,206 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982322200
  49. 2014-05-01 15:12:02,233 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982322200
  50. 2014-05-01 15:12:05,205 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982325200
  51. 2014-05-01 15:12:05,233 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982325200
  52. 2014-05-01 15:12:07,231 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  53. 2014-05-01 15:12:07,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  54. 2014-05-01 15:12:07,945 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
  55. 2014-05-01 15:12:08,205 DEBUG [cloud.api.ApiServlet] (catalina-exec-5:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982328201
  56. 2014-05-01 15:12:08,235 DEBUG [cloud.api.ApiServlet] (catalina-exec-5:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982328201
  57. 2014-05-01 15:12:11,205 DEBUG [cloud.api.ApiServlet] (catalina-exec-11:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982331201
  58. 2014-05-01 15:12:11,233 DEBUG [cloud.api.ApiServlet] (catalina-exec-11:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982331201
  59. 2014-05-01 15:12:14,206 DEBUG [cloud.api.ApiServlet] (catalina-exec-23:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982334202
  60. 2014-05-01 15:12:14,233 DEBUG [cloud.api.ApiServlet] (catalina-exec-23:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982334202
  61. 2014-05-01 15:12:15,817 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-4:null) Ping from 1
  62. 2014-05-01 15:12:17,207 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982337202
  63. 2014-05-01 15:12:17,236 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982337202
  64. 2014-05-01 15:12:20,208 DEBUG [cloud.api.ApiServlet] (catalina-exec-3:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982340203
  65. 2014-05-01 15:12:20,235 DEBUG [cloud.api.ApiServlet] (catalina-exec-3:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982340203
  66. 2014-05-01 15:12:23,210 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982343204
  67. 2014-05-01 15:12:23,238 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982343204
  68. 2014-05-01 15:12:26,210 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982346204
  69. 2014-05-01 15:12:26,239 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982346204
  70. 2014-05-01 15:12:29,209 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982349205
  71. 2014-05-01 15:12:29,238 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982349205
  72. 2014-05-01 15:12:32,210 DEBUG [cloud.api.ApiServlet] (catalina-exec-13:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982352205
  73. 2014-05-01 15:12:32,237 DEBUG [cloud.api.ApiServlet] (catalina-exec-13:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982352205
  74. 2014-05-01 15:12:35,213 DEBUG [cloud.api.ApiServlet] (catalina-exec-16:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982355205
  75. 2014-05-01 15:12:35,241 DEBUG [cloud.api.ApiServlet] (catalina-exec-16:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982355205
  76. 2014-05-01 15:12:37,231 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  77. 2014-05-01 15:12:37,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  78. 2014-05-01 15:12:37,945 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
  79. 2014-05-01 15:12:38,211 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982358206
  80. 2014-05-01 15:12:38,238 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982358206
  81. 2014-05-01 15:12:41,211 DEBUG [cloud.api.ApiServlet] (catalina-exec-7:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982361207
  82. 2014-05-01 15:12:41,239 DEBUG [cloud.api.ApiServlet] (catalina-exec-7:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982361207
  83. 2014-05-01 15:12:43,007 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Resetting hosts suitable for reconnect
  84. 2014-05-01 15:12:43,010 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed resetting hosts suitable for reconnect
  85. 2014-05-01 15:12:43,011 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Acquiring hosts for clusters already owned by this management server
  86. 2014-05-01 15:12:43,012 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters already owned by this management server
  87. 2014-05-01 15:12:43,012 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Acquiring hosts for clusters not owned by any management server
  88. 2014-05-01 15:12:43,012 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters not owned by any management server
  89. 2014-05-01 15:12:44,212 DEBUG [cloud.api.ApiServlet] (catalina-exec-12:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982364207
  90. 2014-05-01 15:12:44,239 DEBUG [cloud.api.ApiServlet] (catalina-exec-12:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982364207
  91. 2014-05-01 15:12:47,212 DEBUG [cloud.api.ApiServlet] (catalina-exec-18:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982367207
  92. 2014-05-01 15:12:47,239 DEBUG [cloud.api.ApiServlet] (catalina-exec-18:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982367207
  93. 2014-05-01 15:12:50,213 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982370208
  94. 2014-05-01 15:12:50,238 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982370208
  95. 2014-05-01 15:12:53,039 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) VmStatsCollector is running...
  96. 2014-05-01 15:12:53,214 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982373209
  97. 2014-05-01 15:12:53,236 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982373209
  98. 2014-05-01 15:12:53,818 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) StorageCollector is running...
  99. 2014-05-01 15:12:53,831 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-3:null) Seq 2-1652621322: Forwarding null to 279278805450809
  100. 2014-05-01 15:12:53,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 2-1652621322: Routing from 196683258161390
  101. 2014-05-01 15:12:53,833 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 2-1652621322: Link is closed
  102. 2014-05-01 15:12:53,833 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 2-1652621322: MgmtId 196683258161390: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  103. 2014-05-01 15:12:53,833 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) Seq 2--1: MgmtId 196683258161390: Req: Routing to peer
  104. 2014-05-01 15:12:53,835 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) Seq 2--1: MgmtId 196683258161390: Req: Cancel request received
  105. 2014-05-01 15:12:53,835 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-6:null) Seq 2-1652621322: Cancelling.
  106. 2014-05-01 15:12:53,835 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621322: Waiting some more time because this is the current command
  107. 2014-05-01 15:12:53,835 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621322: Waiting some more time because this is the current command
  108. 2014-05-01 15:12:53,836 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  109. 2014-05-01 15:12:53,836 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621322: Timed out on null
  110. 2014-05-01 15:12:53,836 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621322: Cancelling.
  111. 2014-05-01 15:12:53,836 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:2, com.cloud.exception.OperationTimedoutException: Commands 1652621322 to Host 2 timed out after 3600
  112. 2014-05-01 15:12:53,836 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
  113. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:2, com.cloud.exception.OperationTimedoutException: Commands 1652621322 to Host 2 timed out after 3600
  114. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:104)
  115. at com.cloud.server.StatsCollector$StorageCollector.run(StatsCollector.java:548)
  116. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  117. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  118. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  119. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  120. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  121. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  122. at java.lang.Thread.run(Thread.java:744)
  123. 2014-05-01 15:12:56,215 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982376210
  124. 2014-05-01 15:12:56,244 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982376210
  125. 2014-05-01 15:12:58,051 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) HostStatsCollector is running...
  126. 2014-05-01 15:12:58,663 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-2001993747: Received: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  127. 2014-05-01 15:12:59,215 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982379211
  128. 2014-05-01 15:12:59,241 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982379211
  129. 2014-05-01 15:13:02,216 DEBUG [cloud.api.ApiServlet] (catalina-exec-17:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982382211
  130. 2014-05-01 15:13:02,241 DEBUG [cloud.api.ApiServlet] (catalina-exec-17:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982382211
  131. 2014-05-01 15:13:05,217 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982385212
  132. 2014-05-01 15:13:05,245 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982385212
  133. 2014-05-01 15:13:06,436 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-2001993742: Processing: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: Can't find volume:aac53f1c-e9c5-4e4b-841a-776b65e72809\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:166)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3670)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3539)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1254)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }
  134. 2014-05-01 15:13:06,437 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-2001993742: No more commands found
  135. 2014-05-01 15:13:06,437 DEBUG [agent.transport.Request] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Seq 1-2001993742: Received: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 110, { Answer, Answer } }
  136. 2014-05-01 15:13:06,441 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Failed to start instance VM[SecondaryStorageVm|s-1-VM]
  137. com.cloud.utils.exception.CloudRuntimeException: Unable to get answer that is of class com.cloud.agent.api.StartAnswer
  138. at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
  139. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:920)
  140. at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
  141. at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
  142. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
  143. at com.cloud.server.ManagementServerImpl.startSecondaryStorageVm(ManagementServerImpl.java:3026)
  144. at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
  145. at com.cloud.server.ManagementServerImpl.startSystemVM(ManagementServerImpl.java:3154)
  146. at org.apache.cloudstack.api.command.admin.systemvm.StartSystemVMCmd.execute(StartSystemVMCmd.java:106)
  147. at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
  148. at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
  149. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  150. at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  151. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  152. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  153. at java.lang.Thread.run(Thread.java:744)
  154. 2014-05-01 15:13:06,445 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1-VM] in Starting state
  155. 2014-05-01 15:13:06,448 DEBUG [agent.transport.Request] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Seq 1-2001993748: Sending { Cmd , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"s-1-VM","wait":0}}] }
  156. 2014-05-01 15:13:06,699 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 1-2001993748: Processing: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
  157. 2014-05-01 15:13:06,700 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-9:null) Seq 1-2001993748: No more commands found
  158. 2014-05-01 15:13:06,700 DEBUG [agent.transport.Request] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Seq 1-2001993748: Received: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 110, { StopAnswer } }
  159. 2014-05-01 15:13:06,725 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Asking VirtualRouter to release Nic[1-1-48606b40-8050-4c63-a501-c157b8a6f2d6-192.168.100.30]
  160. 2014-05-01 15:13:06,729 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Asking SecurityGroupProvider to release Nic[1-1-48606b40-8050-4c63-a501-c157b8a6f2d6-192.168.100.30]
  161. 2014-05-01 15:13:06,733 DEBUG [network.guru.ControlNetworkGuru] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Released nic: NicProfile[2-1-null-null-null
  162. 2014-05-01 15:13:06,743 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Releasing ip address for reservationId=48606b40-8050-4c63-a501-c157b8a6f2d6, instance=3
  163. 2014-05-01 15:13:06,744 DEBUG [network.guru.PodBasedNetworkGuru] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Released nic: NicProfile[3-1-null-null-null
  164. 2014-05-01 15:13:06,755 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Releasing ip address for reservationId=48606b40-8050-4c63-a501-c157b8a6f2d6, instance=4
  165. 2014-05-01 15:13:06,756 DEBUG [network.guru.PodBasedNetworkGuru] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Released nic: NicProfile[4-1-null-null-null
  166. 2014-05-01 15:13:06,759 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Successfully released network resources for the vm VM[SecondaryStorageVm|s-1-VM]
  167. 2014-05-01 15:13:06,759 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-1-VM] in Starting state
  168. 2014-05-01 15:13:06,766 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: 1
  169. 2014-05-01 15:13:06,775 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Hosts's actual total CPU: 19200 and CPU after applying overprovisioning: 57600
  170. 2014-05-01 15:13:06,775 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Hosts's actual total RAM: 67522400256 and RAM after applying overprovisioning: 135044800512
  171. 2014-05-01 15:13:06,775 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) release cpu from host: 1, old used: 500,reserved: 1000, actual total: 19200, total with overprovisioning: 57600; new used: 0,reserved:1000; movedfromreserved: false,moveToReserveredfalse
  172. 2014-05-01 15:13:06,775 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) release mem from host: 1, old used: 268435456,reserved: 1207959552, total: 135044800512; new used: 0,reserved:1207959552; movedfromreserved: false,moveToReserveredfalse
  173. 2014-05-01 15:13:06,779 WARN [storage.secondary.SecondaryStorageManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Exception while trying to start secondary storage vm
  174. com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
  175. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
  176. at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
  177. at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
  178. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
  179. at com.cloud.server.ManagementServerImpl.startSecondaryStorageVm(ManagementServerImpl.java:3026)
  180. at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
  181. at com.cloud.server.ManagementServerImpl.startSystemVM(ManagementServerImpl.java:3154)
  182. at org.apache.cloudstack.api.command.admin.systemvm.StartSystemVMCmd.execute(StartSystemVMCmd.java:106)
  183. at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
  184. at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
  185. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  186. at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  187. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  188. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  189. at java.lang.Thread.run(Thread.java:744)
  190. Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to get answer that is of class com.cloud.agent.api.StartAnswer
  191. at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80)
  192. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:920)
  193. ... 18 more
  194. 2014-05-01 15:13:06,789 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ]) Complete async job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Fail to start system vm
  195. 2014-05-01 15:13:07,231 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  196. 2014-05-01 15:13:07,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  197. 2014-05-01 15:13:07,945 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
  198. 2014-05-01 15:13:08,217 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982388213
  199. 2014-05-01 15:13:08,234 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null) Async job-363 = [ 3e0f18fa-4a3d-4157-8eee-76860e46cf5d ] completed
  200. 2014-05-01 15:13:08,241 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END=== 0:0:0:0:0:0:0:1 -- GET command=queryAsyncJobResult&jobId=3e0f18fa-4a3d-4157-8eee-76860e46cf5d&response=json&sessionkey=6InCgATbl8repjd1yNo4xi8slps%3D&_=1398982388213
  201. 2014-05-01 15:13:15,827 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null) Ping from 1
  202. 2014-05-01 15:13:37,229 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  203. 2014-05-01 15:13:37,843 DEBUG [cloud.network.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:null) External devices stats collector is running...
  204. 2014-05-01 15:13:37,938 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Found 0 running routers.
  205. 2014-05-01 15:13:37,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  206. 2014-05-01 15:13:37,944 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
  207. 2014-05-01 15:13:38,028 DEBUG [storage.snapshot.SnapshotSchedulerImpl] (SnapshotPollTask:null) Snapshot scheduler.poll is being called at 2014-05-01 22:13:38 GMT
  208. 2014-05-01 15:13:38,030 DEBUG [storage.snapshot.SnapshotSchedulerImpl] (SnapshotPollTask:null) Got 0 snapshots to be executed at 2014-05-01 22:13:38 GMT
  209. 2014-05-01 15:13:48,025 DEBUG [network.lb.LBHealthCheckManagerImpl] (LBHealthCheck-1:null) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
  210. 2014-05-01 15:13:48,045 DEBUG [network.lb.LBHealthCheckManagerImpl] (LBHealthCheck-1:null) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
  211. 2014-05-01 15:13:53,045 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) VmStatsCollector is running...
  212. 2014-05-01 15:13:53,838 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) StorageCollector is running...
  213. 2014-05-01 15:13:53,850 DEBUG [agent.manager.ClusteredAgentAttache] (StatsCollector-3:null) Seq 2-1652621323: Forwarding null to 279278805450809
  214. 2014-05-01 15:13:53,851 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 2-1652621323: Routing from 196683258161390
  215. 2014-05-01 15:13:53,851 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 2-1652621323: Link is closed
  216. 2014-05-01 15:13:53,852 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 2-1652621323: MgmtId 196683258161390: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  217. 2014-05-01 15:13:53,852 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) Seq 2--1: MgmtId 196683258161390: Req: Routing to peer
  218. 2014-05-01 15:13:53,854 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) Seq 2--1: MgmtId 196683258161390: Req: Cancel request received
  219. 2014-05-01 15:13:53,854 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-12:null) Seq 2-1652621323: Cancelling.
  220. 2014-05-01 15:13:53,854 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621323: Waiting some more time because this is the current command
  221. 2014-05-01 15:13:53,854 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621323: Waiting some more time because this is the current command
  222. 2014-05-01 15:13:53,854 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  223. 2014-05-01 15:13:53,855 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621323: Timed out on null
  224. 2014-05-01 15:13:53,855 DEBUG [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 2-1652621323: Cancelling.
  225. 2014-05-01 15:13:53,855 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:2, com.cloud.exception.OperationTimedoutException: Commands 1652621323 to Host 2 timed out after 3600
  226. 2014-05-01 15:13:53,855 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
  227. com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:2, com.cloud.exception.OperationTimedoutException: Commands 1652621323 to Host 2 timed out after 3600
  228. at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:104)
  229. at com.cloud.server.StatsCollector$StorageCollector.run(StatsCollector.java:548)
  230. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  231. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  232. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  233. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  234. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  235. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  236. at java.lang.Thread.run(Thread.java:744)
  237. 2014-05-01 15:13:58,663 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) HostStatsCollector is running...
  238. 2014-05-01 15:13:59,267 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-2001993749: Received: { Ans: , MgmtId: 196683258161390, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  239. 2014-05-01 15:14:07,229 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  240. 2014-05-01 15:14:07,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  241. 2014-05-01 15:14:07,944 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
  242. 2014-05-01 15:14:08,024 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:null) Checking if any host reservation can be released ...
  243. 2014-05-01 15:14:08,026 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Running Capacity Checker ...
  244. 2014-05-01 15:14:08,027 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) recalculating system capacity
  245. 2014-05-01 15:14:08,027 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing cpu/ram capacity update
  246. 2014-05-01 15:14:08,038 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 0 VMs on host 1
  247. 2014-05-01 15:14:08,052 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:null) Cannot release reservation, Found VM: VM[SecondaryStorageVm|s-1-VM] Stopped but reserved on host 1
  248. 2014-05-01 15:14:08,052 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:null) Done running HostReservationReleaseChecker ...
  249. 2014-05-01 15:14:08,060 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Found 25 VM, not running on host 1
  250. 2014-05-01 15:14:08,098 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Calibrate reserved cpu for host: 1 old reservedCpu:1000 new reservedCpu:1500
  251. 2014-05-01 15:14:08,098 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:null) Calibrate reserved memory for host: 1 old reservedMem:1207959552 new reservedMem:1476395008
  252. 2014-05-01 15:14:08,103 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done executing cpu/ram capacity update
  253. 2014-05-01 15:14:08,103 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing storage capacity update
  254. 2014-05-01 15:14:08,132 DEBUG [cloud.storage.StorageManagerImpl] (CapacityChecker:null) Successfully set Capacity - 3888952377344 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 1, PodId 1
  255. 2014-05-01 15:14:08,138 DEBUG [cloud.storage.StorageManagerImpl] (CapacityChecker:null) Successfully set Capacity - 1944475557888 for capacity type - 9 , DataCenterId - 1, HostOrPoolId - 2, PodId 1
  256. 2014-05-01 15:14:08,138 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done executing storage capacity update
  257. 2014-05-01 15:14:08,138 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing capacity updates for public ip and Vlans
  258. 2014-05-01 15:14:08,148 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done capacity updates for public ip and Vlans
  259. 2014-05-01 15:14:08,148 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Executing capacity updates for private ip
  260. 2014-05-01 15:14:08,159 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done executing capacity updates for private ip
  261. 2014-05-01 15:14:08,159 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done recalculating system capacity
  262. 2014-05-01 15:14:08,197 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Done running Capacity Checker ...
  263. 2014-05-01 15:14:13,007 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Resetting hosts suitable for reconnect
  264. 2014-05-01 15:14:13,010 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed resetting hosts suitable for reconnect
  265. 2014-05-01 15:14:13,011 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Acquiring hosts for clusters already owned by this management server
  266. 2014-05-01 15:14:13,012 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters already owned by this management server
  267. 2014-05-01 15:14:13,012 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Acquiring hosts for clusters not owned by any management server
  268. 2014-05-01 15:14:13,013 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters not owned by any management server
  269. 2014-05-01 15:14:15,813 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) Ping from 1
  270. 2014-05-01 15:14:16,568 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-71) Processing HAWork[71-HA-4-Starting-Investigating]
  271. 2014-05-01 15:14:16,574 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-71) HA on VM[DomainRouter|r-4-VM]
  272. 2014-05-01 15:14:16,574 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-71) VM VM[DomainRouter|r-4-VM] has been changed. Current State = Stopped Previous State = Starting last updated = 71 previous updated = 66
  273. 2014-05-01 15:14:16,574 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-71) Completed HAWork[71-HA-4-Starting-Investigating]
  274. 2014-05-01 15:14:37,229 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is not ready to launch console proxy yet
  275. 2014-05-01 15:14:37,942 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
  276. 2014-05-01 15:14:37,945 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!