API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Nov 13th, 2019
195
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 334.56 KB | None | 0 0
raw download clone embed print report
  1. 2019-11-13 10:53:02,104 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:2 usedCpu: 134600 reservedCpu: 0
  2. 2019-11-13 10:53:02,104 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:2 usedMem: 748398051328 reservedMem: 0
  3. 2019-11-13 10:53:02,388 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 122 VMs on host 3
  4. 2019-11-13 10:53:02,596 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-fb81a896) (logid:259c0c4e) Found 150 VPC networks to update Redundant State.
  5. 2019-11-13 10:53:02,678 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-fb81a896) (logid:259c0c4e) Found 0 networks to update RvR status.
  6. 2019-11-13 10:53:02,738 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-5:ctx-43caafa8) (logid:42a920b4) Skip router pair (r-2464-VM,r-2465-VM) due to not belong to this mgmt server
  7. 2019-11-13 10:53:02,859 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-7:ctx-3d674e63) (logid:ae6399a2) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  8. 2019-11-13 10:53:02,878 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-1:ctx-93c1a0ba) (logid:d485fd72) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  9. 2019-11-13 10:53:03,046 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 1 VM, not running on host 3
  10. 2019-11-13 10:53:03,065 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:3 usedCpuCore: 474 reservedCpuCore: 0
  11. 2019-11-13 10:53:03,065 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:3 usedCpu: 116924 reservedCpu: 0
  12. 2019-11-13 10:53:03,065 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:3 usedMem: 795642691584 reservedMem: 0
  13. 2019-11-13 10:53:03,254 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 97 VMs on host 9
  14. 2019-11-13 10:53:03,409 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 1 VM, not running on host 9
  15. 2019-11-13 10:53:03,417 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:9 usedCpuCore: 435 reservedCpuCore: 0
  16. 2019-11-13 10:53:03,418 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:9 usedCpu: 101100 reservedCpu: 0
  17. 2019-11-13 10:53:03,418 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:9 usedMem: 792421466112 reservedMem: 0
  18. 2019-11-13 10:53:03,492 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 5 VMs on host 10
  19. 2019-11-13 10:53:03,515 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 6 VM, not running on host 10
  20. 2019-11-13 10:53:03,534 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:10 usedCpuCore: 28 reservedCpuCore: 0
  21. 2019-11-13 10:53:03,534 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:10 usedCpu: 5600 reservedCpu: 0
  22. 2019-11-13 10:53:03,534 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:10 usedMem: 73047998464 reservedMem: 0
  23. 2019-11-13 10:53:03,631 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 59 VMs on host 13
  24. 2019-11-13 10:53:03,755 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 6 VM, not running on host 13
  25. 2019-11-13 10:53:03,772 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:13 usedCpuCore: 299 reservedCpuCore: 0
  26. 2019-11-13 10:53:03,773 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:13 usedCpu: 63100 reservedCpu: 0
  27. 2019-11-13 10:53:03,773 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:13 usedMem: 772029808640 reservedMem: 0
  28. 2019-11-13 10:53:03,880 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 112 VMs on host 15
  29. 2019-11-13 10:53:04,031 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 0 VM, not running on host 15
  30. 2019-11-13 10:53:04,037 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:15 usedCpuCore: 442 reservedCpuCore: 0
  31. 2019-11-13 10:53:04,037 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:15 usedCpu: 116112 reservedCpu: 0
  32. 2019-11-13 10:53:04,037 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:15 usedMem: 714546872320 reservedMem: 0
  33. 2019-11-13 10:53:04,093 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 56 VMs on host 19
  34. 2019-11-13 10:53:04,179 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 6 VM, not running on host 19
  35. 2019-11-13 10:53:04,196 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:19 usedCpuCore: 305 reservedCpuCore: 0
  36. 2019-11-13 10:53:04,196 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:19 usedCpu: 77400 reservedCpu: 0
  37. 2019-11-13 10:53:04,196 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:19 usedMem: 777441509376 reservedMem: 0
  38. 2019-11-13 10:53:04,264 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 26 VMs on host 74673
  39. 2019-11-13 10:53:04,305 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 0 VM, not running on host 74673
  40. 2019-11-13 10:53:04,310 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:74673 usedCpuCore: 170 reservedCpuCore: 0
  41. 2019-11-13 10:53:04,310 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:74673 usedCpu: 53800 reservedCpu: 0
  42. 2019-11-13 10:53:04,310 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:74673 usedMem: 434874875904 reservedMem: 0
  43. 2019-11-13 10:53:04,343 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 2 VMs on host 74677
  44. 2019-11-13 10:53:04,352 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found 0 VM, not running on host 74677
  45. 2019-11-13 10:53:04,364 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:74677 usedCpuCore: 16 reservedCpuCore: 0
  46. 2019-11-13 10:53:04,364 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate cpu capacity, host:74677 usedCpu: 0 reservedCpu: 0
  47. 2019-11-13 10:53:04,364 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) No need to calibrate memory capacity, host:74677 usedMem: 34359738368 reservedMem: 0
  48. 2019-11-13 10:53:04,370 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Done executing cpu/ram capacity update
  49. 2019-11-13 10:53:04,370 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Executing storage capacity update
  50. 2019-11-13 10:53:04,391 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_PROD01 of type PreSetup with overprovisioning factor 1
  51. 2019-11-13 10:53:04,393 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 10995103694848
  52. 2019-11-13 10:53:04,393 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_PROD01 id: 1 is 10995103694848
  53. 2019-11-13 10:53:04,399 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 10995103694848 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 1, PodId 1
  54. 2019-11-13 10:53:04,433 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_PROD02 of type PreSetup with overprovisioning factor 1
  55. 2019-11-13 10:53:04,433 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 10995103694848
  56. 2019-11-13 10:53:04,433 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_PROD02 id: 3 is 10995103694848
  57. 2019-11-13 10:53:04,437 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 10995103694848 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 3, PodId 1
  58. 2019-11-13 10:53:04,456 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_PROD03 of type PreSetup with overprovisioning factor 1
  59. 2019-11-13 10:53:04,457 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 10995103694848
  60. 2019-11-13 10:53:04,457 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_PROD03 id: 4 is 10995103694848
  61. 2019-11-13 10:53:04,459 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 10995103694848 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 4, PodId 1
  62. 2019-11-13 10:53:04,471 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_FAST_PROD01 of type PreSetup with overprovisioning factor 1
  63. 2019-11-13 10:53:04,471 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 5497545555968
  64. 2019-11-13 10:53:04,471 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_FAST_PROD01 id: 5 is 5497545555968
  65. 2019-11-13 10:53:04,473 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 5497545555968 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 5, PodId 1
  66. 2019-11-13 10:53:04,487 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_FAST_PROD02 of type PreSetup with overprovisioning factor 1
  67. 2019-11-13 10:53:04,487 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 3298522300416
  68. 2019-11-13 10:53:04,487 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_FAST_PROD02 id: 6 is 3298522300416
  69. 2019-11-13 10:53:04,492 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 3298522300416 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 6, PodId 1
  70. 2019-11-13 10:53:04,503 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_FAST_PROD03 of type PreSetup with overprovisioning factor 1
  71. 2019-11-13 10:53:04,503 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 2199010672640
  72. 2019-11-13 10:53:04,503 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_FAST_PROD03 id: 7 is 2199010672640
  73. 2019-11-13 10:53:04,506 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 2199010672640 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 7, PodId 1
  74. 2019-11-13 10:53:04,518 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Found storage pool SAN_PROD04 of type PreSetup with overprovisioning factor 1
  75. 2019-11-13 10:53:04,518 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity calculated is 1 * 3298522300416
  76. 2019-11-13 10:53:04,518 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Total over provisioned capacity of the pool SAN_PROD04 id: 8 is 3298522300416
  77. 2019-11-13 10:53:04,520 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Successfully set Capacity - 3298522300416 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 8, PodId 1
  78. 2019-11-13 10:53:04,520 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Done executing storage capacity update
  79. 2019-11-13 10:53:04,520 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Executing capacity updates for public ip and Vlans
  80. 2019-11-13 10:53:04,577 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Done capacity updates for public ip and Vlans
  81. 2019-11-13 10:53:04,577 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Executing capacity updates for private ip
  82. 2019-11-13 10:53:04,590 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Done executing capacity updates for private ip
  83. 2019-11-13 10:53:04,590 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Done recalculating system capacity
  84. 2019-11-13 10:53:04,731 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-faf3f218) (logid:6f91151d) Done running Capacity Checker ...
  85. 2019-11-13 10:53:09,643 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f545b6b0) (logid:0a7e76d5) Detected management node left, id:5, nodeIP:172.16.2.8
  86. 2019-11-13 10:53:09,643 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f545b6b0) (logid:0a7e76d5) Trying to connect to 172.16.2.8
  87. 2019-11-13 10:53:09,645 ERROR [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f545b6b0) (logid:0a7e76d5) Unable to ping management server at 172.16.2.8:9090 due to ConnectException
  88. java.net.ConnectException: Connection refused
  89. at sun.nio.ch.Net.connect0(Native Method)
  90. at sun.nio.ch.Net.connect(Net.java:454)
  91. at sun.nio.ch.Net.connect(Net.java:446)
  92. at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
  93. at com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1137)
  94. at com.cloud.cluster.ClusterManagerImpl.peerScan(ClusterManagerImpl.java:874)
  95. at com.cloud.cluster.ClusterManagerImpl.access$900(ClusterManagerImpl.java:69)
  96. at com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:566)
  97. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  98. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  99. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  100. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  101. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  102. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  103. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  104. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  105. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  106. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  107. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  108. at java.lang.Thread.run(Thread.java:748)
  109. 2019-11-13 10:53:09,646 WARN [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f545b6b0) (logid:0a7e76d5) Management node 5 is detected inactive by timestamp and also not pingable
  110. 2019-11-13 10:53:09,647 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Notify management server node left to listeners.
  111. 2019-11-13 10:53:09,647 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Leaving node, IP: 172.16.2.8, msid: 213677140831823
  112. 2019-11-13 10:53:09,648 INFO [c.c.a.m.ClusteredAgentManagerImpl] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Marking hosts as disconnected on Management server213677140831823
  113. 2019-11-13 10:53:09,677 DEBUG [o.a.c.o.d.OutOfBandManagementDaoImpl] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Expired out-of-band management ownership for hosts owned by management server id:213677140831823
  114. 2019-11-13 10:53:09,681 INFO [c.c.a.m.ClusteredAgentManagerImpl] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Deleting entries from op_host_transfer table for Management server 213677140831823
  115. 2019-11-13 10:53:09,691 INFO [c.c.u.d.Merovingian2] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Cleaning up locks for 213677140831823
  116. 2019-11-13 10:53:09,693 INFO [c.c.u.d.Merovingian2] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Released 0 locks for 213677140831823
  117. 2019-11-13 10:53:09,693 DEBUG [c.c.a.ClusterAlertAdapter] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Receive cluster alert, EventArgs: com.cloud.cluster.ClusterNodeLeftEventArgs
  118. 2019-11-13 10:53:09,693 DEBUG [c.c.a.ClusterAlertAdapter] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Handle cluster node left alert, leaving node: 172.16.2.8, msid: 213677140831823
  119. 2019-11-13 10:53:09,699 DEBUG [c.c.a.ClusterAlertAdapter] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Detected management server node 172.16.2.8 is down, send alert
  120. 2019-11-13 10:53:09,700 WARN [o.a.c.alerts] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) AlertType:: 14 | dataCenterId:: 0 | podId:: 0 | clusterId:: null | message:: Management server node 172.16.2.8 is down
  121. 2019-11-13 10:53:09,716 WARN [c.c.c.ClusterManagerImpl] (Cluster-Notification-1:ctx-75536a26) (logid:dc350960) Notifying management server leave event took 68 ms
  122. 2019-11-13 10:53:10,614 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-fc94f1a8) (logid:459d4150) Begin cleanup expired async-jobs
  123. 2019-11-13 10:53:10,624 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-fc94f1a8) (logid:459d4150) End cleanup expired async-jobs
  124. 2019-11-13 10:53:11,742 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-0f36431c) (logid:5dc17f8f) Zone 1 is not ready to launch secondary storage VM yet
  125. 2019-11-13 10:53:12,100 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-0583f453) (logid:73364700) Zone 1 is not ready to launch console proxy yet
  126. 2019-11-13 10:53:19,956 ERROR [c.c.a.AlertManagerImpl] (Email-Alerts-Sender-1:null) (logid:dc350960) Failed to send email alert javax.mail.MessagingException: Could not connect to SMTP host: smtp.gmail.com, port: 465, response: -1
  127. 2019-11-13 10:53:20,612 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-553bca12) (logid:f57ed1ea) Begin cleanup expired async-jobs
  128. 2019-11-13 10:53:20,623 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-553bca12) (logid:f57ed1ea) End cleanup expired async-jobs
  129. 2019-11-13 10:53:30,612 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-7a33ca72) (logid:e8340402) Begin cleanup expired async-jobs
  130. 2019-11-13 10:53:30,622 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-7a33ca72) (logid:e8340402) End cleanup expired async-jobs
  131. 2019-11-13 10:53:31,514 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-121acb6d) (logid:95bf8bb1) Found 117 routers to update status.
  132. 2019-11-13 10:53:31,514 DEBUG [c.c.a.ApiServlet] (qtp504527234-11:ctx-bf4fa7d2) (logid:bf46f24c) ===START=== 172.16.6.39 -- GET apikey=Iuj-MzZYb-O2-drEybbjpOvhANmcPnQa3a_-Y9m3xb1l0dZGpO63Z-Dti2IKdFw_M-0s9SEog6c6tcu5wymu2w&command=listVirtualMachines&response=json&signature=6GTPUREJsCbXCk%2BpCZy6k28rEwY%3D
  133. 2019-11-13 10:53:31,541 DEBUG [c.c.a.ApiServer] (qtp504527234-11:ctx-bf4fa7d2 ctx-e560674d) (logid:bf46f24c) CIDRs from which account 'Acct[57bda660-67fb-4720-a1ee-183d6016e61e-infra]' is allowed to perform API calls: 0.0.0.0/0,::/0
  134. 2019-11-13 10:53:31,651 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-11:ctx-bf4fa7d2 ctx-e560674d ctx-fa18f27b) (logid:bf46f24c) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  135. 2019-11-13 10:53:31,656 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-11:ctx-bf4fa7d2 ctx-e560674d ctx-fa18f27b) (logid:bf46f24c) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  136. 2019-11-13 10:53:31,911 DEBUG [c.c.a.ApiServlet] (qtp504527234-11:ctx-bf4fa7d2 ctx-e560674d ctx-fa18f27b) (logid:bf46f24c) ===END=== 172.16.6.39 -- GET apikey=Iuj-MzZYb-O2-drEybbjpOvhANmcPnQa3a_-Y9m3xb1l0dZGpO63Z-Dti2IKdFw_M-0s9SEog6c6tcu5wymu2w&command=listVirtualMachines&response=json&signature=6GTPUREJsCbXCk%2BpCZy6k28rEwY%3D
  137. 2019-11-13 10:53:31,936 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-ec7784a1) (logid:ba789d42) ===START=== 172.16.6.39 -- GET apikey=phwlEyvnaj_6pDIgP4x1Q7gY__S1puxxBLnJ6eOQ0jVEN-URH1e1O3YCS8bL6FgtnqZ75MGG2rVMah6E6wP_NQ&command=listVirtualMachines&response=json&signature=RLb%2FD4MZ64j3cxfgbfAqWmi0zI4%3D
  138. 2019-11-13 10:53:31,951 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-ec7784a1 ctx-d2e6d129) (logid:ba789d42) CIDRs from which account 'Acct[93e181f7-e3a9-4221-81ca-1c7564c4b552-admin-gis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  139. 2019-11-13 10:53:31,986 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-ec7784a1 ctx-d2e6d129 ctx-4b58d734) (logid:ba789d42) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  140. 2019-11-13 10:53:31,990 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-ec7784a1 ctx-d2e6d129 ctx-4b58d734) (logid:ba789d42) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  141. 2019-11-13 10:53:32,014 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-121acb6d) (logid:95bf8bb1) Found 150 VPC networks to update Redundant State.
  142. 2019-11-13 10:53:32,047 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-121acb6d) (logid:95bf8bb1) Found 0 networks to update RvR status.
  143. 2019-11-13 10:53:32,062 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-1:ctx-93c1a0ba) (logid:d485fd72) Skip router pair (r-2464-VM,r-2465-VM) due to not belong to this mgmt server
  144. 2019-11-13 10:53:32,168 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-ec7784a1 ctx-d2e6d129 ctx-4b58d734) (logid:ba789d42) ===END=== 172.16.6.39 -- GET apikey=phwlEyvnaj_6pDIgP4x1Q7gY__S1puxxBLnJ6eOQ0jVEN-URH1e1O3YCS8bL6FgtnqZ75MGG2rVMah6E6wP_NQ&command=listVirtualMachines&response=json&signature=RLb%2FD4MZ64j3cxfgbfAqWmi0zI4%3D
  145. 2019-11-13 10:53:32,179 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-69177fa6) (logid:860516dd) ===START=== 172.16.6.39 -- GET apikey=TzX6yP-0YuuDL9KeOabcYsVV-XL3dO8bBe6LerLbXuk-0HX-4_JFJxRl3MyOolddMILMN_L_li8VCuj02UsqYA&command=listVirtualMachines&response=json&signature=zmwagNWfKJ2BrLhV7dB4vsNhfe0%3D
  146. 2019-11-13 10:53:32,221 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-69177fa6 ctx-9d07f369) (logid:860516dd) CIDRs from which account 'Acct[99a93ba4-e2aa-462a-9473-d7ec3ef947db-admin-apontador]' is allowed to perform API calls: 0.0.0.0/0,::/0
  147. 2019-11-13 10:53:32,224 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-10:ctx-507220b2) (logid:581ce7f6) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  148. 2019-11-13 10:53:32,238 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-6:ctx-2ad603f7) (logid:cd7a7138) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  149. 2019-11-13 10:53:32,287 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-69177fa6 ctx-9d07f369 ctx-81f3d704) (logid:860516dd) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  150. 2019-11-13 10:53:32,297 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-69177fa6 ctx-9d07f369 ctx-81f3d704) (logid:860516dd) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  151. 2019-11-13 10:53:35,783 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-69177fa6 ctx-9d07f369 ctx-81f3d704) (logid:860516dd) ===END=== 172.16.6.39 -- GET apikey=TzX6yP-0YuuDL9KeOabcYsVV-XL3dO8bBe6LerLbXuk-0HX-4_JFJxRl3MyOolddMILMN_L_li8VCuj02UsqYA&command=listVirtualMachines&response=json&signature=zmwagNWfKJ2BrLhV7dB4vsNhfe0%3D
  152. 2019-11-13 10:53:35,808 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-086d53b2) (logid:9db1983f) ===START=== 172.16.6.39 -- GET apikey=1X2TrYPn9wWH2h7NhvNfuhgt_O5YgQ-4MzTZ_9CfWKvoC3fO-bgCVBupGWyXnsfM8w3CLhdWAvdmQZZsRJjVDQ&command=listVirtualMachines&response=json&signature=gnFfZrawiC6bSo1LjD1%2B8pz5Pw8%3D
  153. 2019-11-13 10:53:35,824 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-086d53b2 ctx-b0015a4e) (logid:9db1983f) CIDRs from which account 'Acct[2f3b55e6-0c5b-4c98-bd32-3dbe6110caa7-admin-unidadedenegocio]' is allowed to perform API calls: 0.0.0.0/0,::/0
  154. 2019-11-13 10:53:35,869 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-086d53b2 ctx-b0015a4e ctx-1f4d0ba4) (logid:9db1983f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  155. 2019-11-13 10:53:35,872 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-086d53b2 ctx-b0015a4e ctx-1f4d0ba4) (logid:9db1983f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  156. 2019-11-13 10:53:35,891 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-086d53b2 ctx-b0015a4e ctx-1f4d0ba4) (logid:9db1983f) ===END=== 172.16.6.39 -- GET apikey=1X2TrYPn9wWH2h7NhvNfuhgt_O5YgQ-4MzTZ_9CfWKvoC3fO-bgCVBupGWyXnsfM8w3CLhdWAvdmQZZsRJjVDQ&command=listVirtualMachines&response=json&signature=gnFfZrawiC6bSo1LjD1%2B8pz5Pw8%3D
  157. 2019-11-13 10:53:35,898 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-e7475406) (logid:3fc599ba) ===START=== 172.16.6.39 -- GET apikey=D851k2JpSGcUj2hv0cdlzohx5VNWHpZ100CSnDBHxLoLYMPosuKOWjnz24G1AYHyzA3jz73Y51S320C7gxLTyA&command=listVirtualMachines&response=json&signature=Lndcon41%2Fv5npjdR15QHFRh%2BnMg%3D
  158. 2019-11-13 10:53:35,915 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-e7475406 ctx-b0c449aa) (logid:3fc599ba) CIDRs from which account 'Acct[66258daa-3ace-4378-80a0-286bf8335457-admin-maplink-hmlg]' is allowed to perform API calls: 0.0.0.0/0,::/0
  159. 2019-11-13 10:53:35,943 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-e7475406 ctx-b0c449aa ctx-d214ac94) (logid:3fc599ba) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  160. 2019-11-13 10:53:35,947 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-e7475406 ctx-b0c449aa ctx-d214ac94) (logid:3fc599ba) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  161. 2019-11-13 10:53:36,700 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-e7475406 ctx-b0c449aa ctx-d214ac94) (logid:3fc599ba) ===END=== 172.16.6.39 -- GET apikey=D851k2JpSGcUj2hv0cdlzohx5VNWHpZ100CSnDBHxLoLYMPosuKOWjnz24G1AYHyzA3jz73Y51S320C7gxLTyA&command=listVirtualMachines&response=json&signature=Lndcon41%2Fv5npjdR15QHFRh%2BnMg%3D
  162. 2019-11-13 10:53:36,710 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-a5fa3320) (logid:0c8c8c88) ===START=== 172.16.6.39 -- GET apikey=mRlv1uPRoQbizxMZm2fN1PRKrzuqzwRcd-I-ult0FaefeHtiO-aF7U6PXD12fCBD57fmijoKvkoETqiBn6nZXA&command=listVirtualMachines&response=json&signature=ej%2Bs7kPyeqKAcahkWykP4LSnKZc%3D
  163. 2019-11-13 10:53:36,721 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-a5fa3320 ctx-f0944afd) (logid:0c8c8c88) CIDRs from which account 'Acct[df9f070c-2e4a-42ea-9b69-acfbf954b00d-admin-maplink-dev]' is allowed to perform API calls: 0.0.0.0/0,::/0
  164. 2019-11-13 10:53:36,747 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-a5fa3320 ctx-f0944afd ctx-1b228dd8) (logid:0c8c8c88) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  165. 2019-11-13 10:53:36,749 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-a5fa3320 ctx-f0944afd ctx-1b228dd8) (logid:0c8c8c88) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  166. 2019-11-13 10:53:36,836 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-a5fa3320 ctx-f0944afd ctx-1b228dd8) (logid:0c8c8c88) ===END=== 172.16.6.39 -- GET apikey=mRlv1uPRoQbizxMZm2fN1PRKrzuqzwRcd-I-ult0FaefeHtiO-aF7U6PXD12fCBD57fmijoKvkoETqiBn6nZXA&command=listVirtualMachines&response=json&signature=ej%2Bs7kPyeqKAcahkWykP4LSnKZc%3D
  167. 2019-11-13 10:53:36,843 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-3bcdae45) (logid:0c1a1cc3) ===START=== 172.16.6.39 -- GET apikey=9xG8Ekz2TzUVRgaxLa7gqgIukWRZzMwEScNh-Ufgb-iV-VHoUxZNf4tjTrWBFvy3B1F2fSOhooUI0ZnNqWMMQA&command=listVirtualMachines&response=json&signature=mbokVEPWP0ddlKcTDDlewjJLwVc%3D
  168. 2019-11-13 10:53:36,853 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-3bcdae45 ctx-301d3f6f) (logid:0c1a1cc3) CIDRs from which account 'Acct[d1566f21-219f-4149-a879-76fbedb170de-admin-maplink]' is allowed to perform API calls: 0.0.0.0/0,::/0
  169. 2019-11-13 10:53:36,872 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-3bcdae45 ctx-301d3f6f ctx-acc5d14e) (logid:0c1a1cc3) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  170. 2019-11-13 10:53:36,874 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-3bcdae45 ctx-301d3f6f ctx-acc5d14e) (logid:0c1a1cc3) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  171. 2019-11-13 10:53:40,748 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-27da621b) (logid:f31ff480) Begin cleanup expired async-jobs
  172. 2019-11-13 10:53:40,828 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-27da621b) (logid:f31ff480) End cleanup expired async-jobs
  173. 2019-11-13 10:53:40,888 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-3bcdae45 ctx-301d3f6f ctx-acc5d14e) (logid:0c1a1cc3) ===END=== 172.16.6.39 -- GET apikey=9xG8Ekz2TzUVRgaxLa7gqgIukWRZzMwEScNh-Ufgb-iV-VHoUxZNf4tjTrWBFvy3B1F2fSOhooUI0ZnNqWMMQA&command=listVirtualMachines&response=json&signature=mbokVEPWP0ddlKcTDDlewjJLwVc%3D
  174. 2019-11-13 10:53:40,914 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-107d1d58) (logid:b6ddef46) ===START=== 172.16.6.39 -- GET apikey=hE9BFOvFZntSwPoZ46tJK9ouSglRKuOcuMO-EqXL4dcVAuksuxLHdrIQJ4PHv7BD0hg6l6Ekc4Oxh8P_Dj2OjA&command=listVirtualMachines&response=json&signature=AbRruXle8snmuycRmReGz8QtK98%3D
  175. 2019-11-13 10:53:40,925 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-107d1d58 ctx-d09d8974) (logid:b6ddef46) CIDRs from which account 'Acct[fa7d13a9-7b1a-4d1f-9595-6dd06d170120-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
  176. 2019-11-13 10:53:40,946 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-107d1d58 ctx-d09d8974 ctx-ae7a776d) (logid:b6ddef46) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  177. 2019-11-13 10:53:40,948 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-107d1d58 ctx-d09d8974 ctx-ae7a776d) (logid:b6ddef46) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  178. 2019-11-13 10:53:40,965 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-107d1d58 ctx-d09d8974 ctx-ae7a776d) (logid:b6ddef46) ===END=== 172.16.6.39 -- GET apikey=hE9BFOvFZntSwPoZ46tJK9ouSglRKuOcuMO-EqXL4dcVAuksuxLHdrIQJ4PHv7BD0hg6l6Ekc4Oxh8P_Dj2OjA&command=listVirtualMachines&response=json&signature=AbRruXle8snmuycRmReGz8QtK98%3D
  179. 2019-11-13 10:53:40,974 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-86e56264) (logid:6a621f06) ===START=== 172.16.6.39 -- GET apikey=6T8ikMM472680Uk8IYh5-LYgWwdhd1FuD7K91FC_brcvRb4DATssTFhCVE8nd8wLYUfKJHUS1_icdELMKFc5yA&command=listVirtualMachines&response=json&signature=48K4Fy7HWab3syf%2F7hHwgVOUcRs%3D
  180. 2019-11-13 10:53:40,985 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-86e56264 ctx-01273aad) (logid:6a621f06) CIDRs from which account 'Acct[4a326a28-cef2-4f15-b8de-0f7cec7c1027-admin-optilogistic]' is allowed to perform API calls: 0.0.0.0/0,::/0
  181. 2019-11-13 10:53:41,005 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-86e56264 ctx-01273aad ctx-12b8c3e9) (logid:6a621f06) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  182. 2019-11-13 10:53:41,006 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-86e56264 ctx-01273aad ctx-12b8c3e9) (logid:6a621f06) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  183. 2019-11-13 10:53:41,515 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-86e56264 ctx-01273aad ctx-12b8c3e9) (logid:6a621f06) ===END=== 172.16.6.39 -- GET apikey=6T8ikMM472680Uk8IYh5-LYgWwdhd1FuD7K91FC_brcvRb4DATssTFhCVE8nd8wLYUfKJHUS1_icdELMKFc5yA&command=listVirtualMachines&response=json&signature=48K4Fy7HWab3syf%2F7hHwgVOUcRs%3D
  184. 2019-11-13 10:53:41,523 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-9e3f7694) (logid:6145d725) ===START=== 172.16.6.39 -- GET apikey=GDTBAQ0g2QY5MXb-cyp8Cwel2Ee6CPSrgry8kzo7rxo9dvfLtCNWNfI18qDLOxwdp8hpCTjaTX8-sDMYj22q3A&command=listVirtualMachines&response=json&signature=P8tjETWxf64PI35Nod2P1METQNI%3D
  185. 2019-11-13 10:53:41,534 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-9e3f7694 ctx-3e5514dd) (logid:6145d725) CIDRs from which account 'Acct[7ee02798-9ccc-4ba1-bb93-b0cec3e897c7-admin-axiodis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  186. 2019-11-13 10:53:41,551 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-9e3f7694 ctx-3e5514dd ctx-4fcdb218) (logid:6145d725) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  187. 2019-11-13 10:53:41,553 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-9e3f7694 ctx-3e5514dd ctx-4fcdb218) (logid:6145d725) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  188. 2019-11-13 10:53:41,609 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-9e3f7694 ctx-3e5514dd ctx-4fcdb218) (logid:6145d725) ===END=== 172.16.6.39 -- GET apikey=GDTBAQ0g2QY5MXb-cyp8Cwel2Ee6CPSrgry8kzo7rxo9dvfLtCNWNfI18qDLOxwdp8hpCTjaTX8-sDMYj22q3A&command=listVirtualMachines&response=json&signature=P8tjETWxf64PI35Nod2P1METQNI%3D
  189. 2019-11-13 10:53:41,617 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-d7d2f533) (logid:02b7050f) ===START=== 172.16.6.39 -- GET apikey=dOo2VogyqgcnC-y9jmnZlGZ04_xhgOk2j9bzDsvyU5gSDV_jjeCUEaf-AyLSoWrDGBje9BshbghOLcWxMvEidA&command=listVirtualMachines&response=json&signature=Mzbot%2BSM%2BC3zGGF5cU%2BhEZfHGqM%3D
  190. 2019-11-13 10:53:41,625 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-d7d2f533 ctx-f6004e87) (logid:02b7050f) CIDRs from which account 'Acct[babb2439-5bbc-4e50-90cf-ec3978ba6ee6-admin-apontador-hmlg]' is allowed to perform API calls: 0.0.0.0/0,::/0
  191. 2019-11-13 10:53:41,643 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-d7d2f533 ctx-f6004e87 ctx-46698b30) (logid:02b7050f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  192. 2019-11-13 10:53:41,644 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-d7d2f533 ctx-f6004e87 ctx-46698b30) (logid:02b7050f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  193. 2019-11-13 10:53:41,739 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f1034272) (logid:c20c55b0) Zone 1 is not ready to launch secondary storage VM yet
  194. 2019-11-13 10:53:41,999 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-d7d2f533 ctx-f6004e87 ctx-46698b30) (logid:02b7050f) ===END=== 172.16.6.39 -- GET apikey=dOo2VogyqgcnC-y9jmnZlGZ04_xhgOk2j9bzDsvyU5gSDV_jjeCUEaf-AyLSoWrDGBje9BshbghOLcWxMvEidA&command=listVirtualMachines&response=json&signature=Mzbot%2BSM%2BC3zGGF5cU%2BhEZfHGqM%3D
  195. 2019-11-13 10:53:42,007 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-9e309d2d) (logid:92c91d8e) ===START=== 172.16.6.39 -- GET apikey=uhGUyYK7Zl27dM_ZTX4d_KQNQx2hUWD9qqAS-SkzrVNDVlWtELCQgjVU29g2_HSwy98jVWZHSgrd7WeFTigMfg&command=listVirtualMachines&response=json&signature=V%2BQZ6VBtlZbgm%2B3FFa8oKQVyn6c%3D
  196. 2019-11-13 10:53:42,016 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-9e309d2d ctx-2005e687) (logid:92c91d8e) CIDRs from which account 'Acct[356e266d-a280-4548-9a94-e6dca9336148-admin-axiodis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  197. 2019-11-13 10:53:42,033 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-9e309d2d ctx-2005e687 ctx-9fdd8ce8) (logid:92c91d8e) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  198. 2019-11-13 10:53:42,035 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-9e309d2d ctx-2005e687 ctx-9fdd8ce8) (logid:92c91d8e) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  199. 2019-11-13 10:53:42,095 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-d2190c65) (logid:1d9d9035) Zone 1 is not ready to launch console proxy yet
  200. 2019-11-13 10:53:42,396 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-9e309d2d ctx-2005e687 ctx-9fdd8ce8) (logid:92c91d8e) ===END=== 172.16.6.39 -- GET apikey=uhGUyYK7Zl27dM_ZTX4d_KQNQx2hUWD9qqAS-SkzrVNDVlWtELCQgjVU29g2_HSwy98jVWZHSgrd7WeFTigMfg&command=listVirtualMachines&response=json&signature=V%2BQZ6VBtlZbgm%2B3FFa8oKQVyn6c%3D
  201. 2019-11-13 10:53:45,887 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-b5917c42) (logid:a366e5c4) AutoScaling Monitor is running...
  202. 2019-11-13 10:53:46,987 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Seq 2-2223652316014182402: Waiting some more time because this is the current command
  203. 2019-11-13 10:53:50,615 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9872987a) (logid:2678153f) Begin cleanup expired async-jobs
  204. 2019-11-13 10:53:50,625 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9872987a) (logid:2678153f) End cleanup expired async-jobs
  205. 2019-11-13 10:54:00,615 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d4ba14cd) (logid:d8726dfe) Begin cleanup expired async-jobs
  206. 2019-11-13 10:54:00,626 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-d4ba14cd) (logid:d8726dfe) End cleanup expired async-jobs
  207. 2019-11-13 10:54:01,372 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-42ef0da0) (logid:54fdbf5b) Found 117 routers to update status.
  208. 2019-11-13 10:54:01,571 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-42ef0da0) (logid:54fdbf5b) Found 150 VPC networks to update Redundant State.
  209. 2019-11-13 10:54:01,576 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-42ef0da0) (logid:54fdbf5b) Found 0 networks to update RvR status.
  210. 2019-11-13 10:54:01,612 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-10:ctx-507220b2) (logid:581ce7f6) Skip router pair (r-2464-VM,r-2465-VM) due to not belong to this mgmt server
  211. 2019-11-13 10:54:01,691 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-7:ctx-3d674e63) (logid:ae6399a2) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  212. 2019-11-13 10:54:01,692 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-1:ctx-93c1a0ba) (logid:d485fd72) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  213. 2019-11-13 10:54:05,833 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-8345782e) (logid:17fa3b67) Resetting hosts suitable for reconnect
  214. 2019-11-13 10:54:05,836 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-8345782e) (logid:17fa3b67) Completed resetting hosts suitable for reconnect
  215. 2019-11-13 10:54:05,836 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-8345782e) (logid:17fa3b67) Acquiring hosts for clusters already owned by this management server
  216. 2019-11-13 10:54:05,838 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-8345782e) (logid:17fa3b67) Completed acquiring hosts for clusters already owned by this management server
  217. 2019-11-13 10:54:05,838 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-8345782e) (logid:17fa3b67) Acquiring hosts for clusters not owned by any management server
  218. 2019-11-13 10:54:05,839 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-8345782e) (logid:17fa3b67) Completed acquiring hosts for clusters not owned by any management server
  219. 2019-11-13 10:54:10,613 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9e58d35b) (logid:d69af18a) Begin cleanup expired async-jobs
  220. 2019-11-13 10:54:10,624 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9e58d35b) (logid:d69af18a) End cleanup expired async-jobs
  221. 2019-11-13 10:54:11,739 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-bed28bf6) (logid:308fbe59) Zone 1 is not ready to launch secondary storage VM yet
  222. 2019-11-13 10:54:12,090 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-9e0ec3b8) (logid:c370fc3c) Zone 1 is not ready to launch console proxy yet
  223. 2019-11-13 10:54:20,613 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-fde65a2a) (logid:b9049965) Begin cleanup expired async-jobs
  224. 2019-11-13 10:54:20,623 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-fde65a2a) (logid:b9049965) Expunging completed job-78743
  225. 2019-11-13 10:54:20,632 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-fde65a2a) (logid:b9049965) End cleanup expired async-jobs
  226. 2019-11-13 10:54:30,611 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bec0d7be) (logid:a4a72bc5) Begin cleanup expired async-jobs
  227. 2019-11-13 10:54:30,624 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bec0d7be) (logid:a4a72bc5) End cleanup expired async-jobs
  228. 2019-11-13 10:54:31,359 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-19567efa) (logid:d529b672) Found 117 routers to update status.
  229. 2019-11-13 10:54:31,367 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-023f8f96) (logid:551da911) ===START=== 172.16.6.39 -- GET apikey=Iuj-MzZYb-O2-drEybbjpOvhANmcPnQa3a_-Y9m3xb1l0dZGpO63Z-Dti2IKdFw_M-0s9SEog6c6tcu5wymu2w&command=listVirtualMachines&response=json&signature=6GTPUREJsCbXCk%2BpCZy6k28rEwY%3D
  230. 2019-11-13 10:54:31,376 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-023f8f96 ctx-c3fd3100) (logid:551da911) CIDRs from which account 'Acct[57bda660-67fb-4720-a1ee-183d6016e61e-infra]' is allowed to perform API calls: 0.0.0.0/0,::/0
  231. 2019-11-13 10:54:31,391 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-023f8f96 ctx-c3fd3100 ctx-50204d52) (logid:551da911) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  232. 2019-11-13 10:54:31,395 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-023f8f96 ctx-c3fd3100 ctx-50204d52) (logid:551da911) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  233. 2019-11-13 10:54:31,503 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-023f8f96 ctx-c3fd3100 ctx-50204d52) (logid:551da911) ===END=== 172.16.6.39 -- GET apikey=Iuj-MzZYb-O2-drEybbjpOvhANmcPnQa3a_-Y9m3xb1l0dZGpO63Z-Dti2IKdFw_M-0s9SEog6c6tcu5wymu2w&command=listVirtualMachines&response=json&signature=6GTPUREJsCbXCk%2BpCZy6k28rEwY%3D
  234. 2019-11-13 10:54:31,510 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-71ab086c) (logid:122bb60a) ===START=== 172.16.6.39 -- GET apikey=phwlEyvnaj_6pDIgP4x1Q7gY__S1puxxBLnJ6eOQ0jVEN-URH1e1O3YCS8bL6FgtnqZ75MGG2rVMah6E6wP_NQ&command=listVirtualMachines&response=json&signature=RLb%2FD4MZ64j3cxfgbfAqWmi0zI4%3D
  235. 2019-11-13 10:54:31,518 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-71ab086c ctx-06ca8b8c) (logid:122bb60a) CIDRs from which account 'Acct[93e181f7-e3a9-4221-81ca-1c7564c4b552-admin-gis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  236. 2019-11-13 10:54:31,533 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-71ab086c ctx-06ca8b8c ctx-8c1d545f) (logid:122bb60a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  237. 2019-11-13 10:54:31,534 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-16:ctx-71ab086c ctx-06ca8b8c ctx-8c1d545f) (logid:122bb60a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  238. 2019-11-13 10:54:31,557 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-19567efa) (logid:d529b672) Found 150 VPC networks to update Redundant State.
  239. 2019-11-13 10:54:31,558 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-71ab086c ctx-06ca8b8c ctx-8c1d545f) (logid:122bb60a) ===END=== 172.16.6.39 -- GET apikey=phwlEyvnaj_6pDIgP4x1Q7gY__S1puxxBLnJ6eOQ0jVEN-URH1e1O3YCS8bL6FgtnqZ75MGG2rVMah6E6wP_NQ&command=listVirtualMachines&response=json&signature=RLb%2FD4MZ64j3cxfgbfAqWmi0zI4%3D
  240. 2019-11-13 10:54:31,565 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-19567efa) (logid:d529b672) Found 0 networks to update RvR status.
  241. 2019-11-13 10:54:31,573 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-87b4cab9) (logid:a2b6bf3a) ===START=== 172.16.6.39 -- GET apikey=TzX6yP-0YuuDL9KeOabcYsVV-XL3dO8bBe6LerLbXuk-0HX-4_JFJxRl3MyOolddMILMN_L_li8VCuj02UsqYA&command=listVirtualMachines&response=json&signature=zmwagNWfKJ2BrLhV7dB4vsNhfe0%3D
  242. 2019-11-13 10:54:31,603 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-9:ctx-9b95b854) (logid:e8939ebf) Skip router pair (r-2464-VM,r-2465-VM) due to not belong to this mgmt server
  243. 2019-11-13 10:54:31,612 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-87b4cab9 ctx-fa018416) (logid:a2b6bf3a) CIDRs from which account 'Acct[99a93ba4-e2aa-462a-9473-d7ec3ef947db-admin-apontador]' is allowed to perform API calls: 0.0.0.0/0,::/0
  244. 2019-11-13 10:54:31,672 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-9:ctx-9b95b854) (logid:e8939ebf) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  245. 2019-11-13 10:54:31,677 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-4:ctx-c1c13ec4) (logid:5ae904d2) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  246. 2019-11-13 10:54:31,684 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-87b4cab9 ctx-fa018416 ctx-95febeb9) (logid:a2b6bf3a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  247. 2019-11-13 10:54:31,697 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-87b4cab9 ctx-fa018416 ctx-95febeb9) (logid:a2b6bf3a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  248. 2019-11-13 10:54:34,212 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-87b4cab9 ctx-fa018416 ctx-95febeb9) (logid:a2b6bf3a) ===END=== 172.16.6.39 -- GET apikey=TzX6yP-0YuuDL9KeOabcYsVV-XL3dO8bBe6LerLbXuk-0HX-4_JFJxRl3MyOolddMILMN_L_li8VCuj02UsqYA&command=listVirtualMachines&response=json&signature=zmwagNWfKJ2BrLhV7dB4vsNhfe0%3D
  249. 2019-11-13 10:54:34,229 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-e697c562) (logid:f15f807c) ===START=== 172.16.6.39 -- GET apikey=1X2TrYPn9wWH2h7NhvNfuhgt_O5YgQ-4MzTZ_9CfWKvoC3fO-bgCVBupGWyXnsfM8w3CLhdWAvdmQZZsRJjVDQ&command=listVirtualMachines&response=json&signature=gnFfZrawiC6bSo1LjD1%2B8pz5Pw8%3D
  250. 2019-11-13 10:54:34,242 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-e697c562 ctx-0d901a1c) (logid:f15f807c) CIDRs from which account 'Acct[2f3b55e6-0c5b-4c98-bd32-3dbe6110caa7-admin-unidadedenegocio]' is allowed to perform API calls: 0.0.0.0/0,::/0
  251. 2019-11-13 10:54:34,261 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-e697c562 ctx-0d901a1c ctx-bd5f3ac6) (logid:f15f807c) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  252. 2019-11-13 10:54:34,262 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-e697c562 ctx-0d901a1c ctx-bd5f3ac6) (logid:f15f807c) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  253. 2019-11-13 10:54:34,280 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-e697c562 ctx-0d901a1c ctx-bd5f3ac6) (logid:f15f807c) ===END=== 172.16.6.39 -- GET apikey=1X2TrYPn9wWH2h7NhvNfuhgt_O5YgQ-4MzTZ_9CfWKvoC3fO-bgCVBupGWyXnsfM8w3CLhdWAvdmQZZsRJjVDQ&command=listVirtualMachines&response=json&signature=gnFfZrawiC6bSo1LjD1%2B8pz5Pw8%3D
  254. 2019-11-13 10:54:34,289 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-cb14ebab) (logid:e0964538) ===START=== 172.16.6.39 -- GET apikey=D851k2JpSGcUj2hv0cdlzohx5VNWHpZ100CSnDBHxLoLYMPosuKOWjnz24G1AYHyzA3jz73Y51S320C7gxLTyA&command=listVirtualMachines&response=json&signature=Lndcon41%2Fv5npjdR15QHFRh%2BnMg%3D
  255. 2019-11-13 10:54:34,300 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-cb14ebab ctx-bfe327b7) (logid:e0964538) CIDRs from which account 'Acct[66258daa-3ace-4378-80a0-286bf8335457-admin-maplink-hmlg]' is allowed to perform API calls: 0.0.0.0/0,::/0
  256. 2019-11-13 10:54:34,320 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-cb14ebab ctx-bfe327b7 ctx-329d5c2d) (logid:e0964538) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  257. 2019-11-13 10:54:34,321 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-cb14ebab ctx-bfe327b7 ctx-329d5c2d) (logid:e0964538) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  258. 2019-11-13 10:54:35,013 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-cb14ebab ctx-bfe327b7 ctx-329d5c2d) (logid:e0964538) ===END=== 172.16.6.39 -- GET apikey=D851k2JpSGcUj2hv0cdlzohx5VNWHpZ100CSnDBHxLoLYMPosuKOWjnz24G1AYHyzA3jz73Y51S320C7gxLTyA&command=listVirtualMachines&response=json&signature=Lndcon41%2Fv5npjdR15QHFRh%2BnMg%3D
  259. 2019-11-13 10:54:35,024 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-0649d220) (logid:c418694f) ===START=== 172.16.6.39 -- GET apikey=mRlv1uPRoQbizxMZm2fN1PRKrzuqzwRcd-I-ult0FaefeHtiO-aF7U6PXD12fCBD57fmijoKvkoETqiBn6nZXA&command=listVirtualMachines&response=json&signature=ej%2Bs7kPyeqKAcahkWykP4LSnKZc%3D
  260. 2019-11-13 10:54:35,034 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-0649d220 ctx-63e06cdb) (logid:c418694f) CIDRs from which account 'Acct[df9f070c-2e4a-42ea-9b69-acfbf954b00d-admin-maplink-dev]' is allowed to perform API calls: 0.0.0.0/0,::/0
  261. 2019-11-13 10:54:35,050 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-0649d220 ctx-63e06cdb ctx-1e487274) (logid:c418694f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  262. 2019-11-13 10:54:35,051 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-0649d220 ctx-63e06cdb ctx-1e487274) (logid:c418694f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  263. 2019-11-13 10:54:35,131 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-0649d220 ctx-63e06cdb ctx-1e487274) (logid:c418694f) ===END=== 172.16.6.39 -- GET apikey=mRlv1uPRoQbizxMZm2fN1PRKrzuqzwRcd-I-ult0FaefeHtiO-aF7U6PXD12fCBD57fmijoKvkoETqiBn6nZXA&command=listVirtualMachines&response=json&signature=ej%2Bs7kPyeqKAcahkWykP4LSnKZc%3D
  264. 2019-11-13 10:54:35,140 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-e853d4f0) (logid:c4ff197a) ===START=== 172.16.6.39 -- GET apikey=9xG8Ekz2TzUVRgaxLa7gqgIukWRZzMwEScNh-Ufgb-iV-VHoUxZNf4tjTrWBFvy3B1F2fSOhooUI0ZnNqWMMQA&command=listVirtualMachines&response=json&signature=mbokVEPWP0ddlKcTDDlewjJLwVc%3D
  265. 2019-11-13 10:54:35,154 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-e853d4f0 ctx-fbd00eb0) (logid:c4ff197a) CIDRs from which account 'Acct[d1566f21-219f-4149-a879-76fbedb170de-admin-maplink]' is allowed to perform API calls: 0.0.0.0/0,::/0
  266. 2019-11-13 10:54:35,174 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-e853d4f0 ctx-fbd00eb0 ctx-ed6c5f3a) (logid:c4ff197a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  267. 2019-11-13 10:54:35,175 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-e853d4f0 ctx-fbd00eb0 ctx-ed6c5f3a) (logid:c4ff197a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  268. 2019-11-13 10:54:38,866 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-e853d4f0 ctx-fbd00eb0 ctx-ed6c5f3a) (logid:c4ff197a) ===END=== 172.16.6.39 -- GET apikey=9xG8Ekz2TzUVRgaxLa7gqgIukWRZzMwEScNh-Ufgb-iV-VHoUxZNf4tjTrWBFvy3B1F2fSOhooUI0ZnNqWMMQA&command=listVirtualMachines&response=json&signature=mbokVEPWP0ddlKcTDDlewjJLwVc%3D
  269. 2019-11-13 10:54:38,905 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-dc220b3d) (logid:9891baed) ===START=== 172.16.6.39 -- GET apikey=hE9BFOvFZntSwPoZ46tJK9ouSglRKuOcuMO-EqXL4dcVAuksuxLHdrIQJ4PHv7BD0hg6l6Ekc4Oxh8P_Dj2OjA&command=listVirtualMachines&response=json&signature=AbRruXle8snmuycRmReGz8QtK98%3D
  270. 2019-11-13 10:54:38,922 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-dc220b3d ctx-a6b83b41) (logid:9891baed) CIDRs from which account 'Acct[fa7d13a9-7b1a-4d1f-9595-6dd06d170120-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
  271. 2019-11-13 10:54:38,945 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-dc220b3d ctx-a6b83b41 ctx-185f1be1) (logid:9891baed) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  272. 2019-11-13 10:54:38,946 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-dc220b3d ctx-a6b83b41 ctx-185f1be1) (logid:9891baed) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  273. 2019-11-13 10:54:38,962 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-dc220b3d ctx-a6b83b41 ctx-185f1be1) (logid:9891baed) ===END=== 172.16.6.39 -- GET apikey=hE9BFOvFZntSwPoZ46tJK9ouSglRKuOcuMO-EqXL4dcVAuksuxLHdrIQJ4PHv7BD0hg6l6Ekc4Oxh8P_Dj2OjA&command=listVirtualMachines&response=json&signature=AbRruXle8snmuycRmReGz8QtK98%3D
  274. 2019-11-13 10:54:38,972 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-635bffaf) (logid:92668c40) ===START=== 172.16.6.39 -- GET apikey=6T8ikMM472680Uk8IYh5-LYgWwdhd1FuD7K91FC_brcvRb4DATssTFhCVE8nd8wLYUfKJHUS1_icdELMKFc5yA&command=listVirtualMachines&response=json&signature=48K4Fy7HWab3syf%2F7hHwgVOUcRs%3D
  275. 2019-11-13 10:54:38,982 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-635bffaf ctx-b24dc8a4) (logid:92668c40) CIDRs from which account 'Acct[4a326a28-cef2-4f15-b8de-0f7cec7c1027-admin-optilogistic]' is allowed to perform API calls: 0.0.0.0/0,::/0
  276. 2019-11-13 10:54:38,998 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-635bffaf ctx-b24dc8a4 ctx-36696ee4) (logid:92668c40) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  277. 2019-11-13 10:54:39,000 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-635bffaf ctx-b24dc8a4 ctx-36696ee4) (logid:92668c40) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  278. 2019-11-13 10:54:39,593 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-635bffaf ctx-b24dc8a4 ctx-36696ee4) (logid:92668c40) ===END=== 172.16.6.39 -- GET apikey=6T8ikMM472680Uk8IYh5-LYgWwdhd1FuD7K91FC_brcvRb4DATssTFhCVE8nd8wLYUfKJHUS1_icdELMKFc5yA&command=listVirtualMachines&response=json&signature=48K4Fy7HWab3syf%2F7hHwgVOUcRs%3D
  279. 2019-11-13 10:54:39,595 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 9-1463388403918700545: Routing from 107973147312911
  280. 2019-11-13 10:54:39,604 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 9-1463388403918700545: Link is closed
  281. 2019-11-13 10:54:39,607 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 9-1463388403918700545: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  282. 2019-11-13 10:54:39,602 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 2-2223652316014182402: Routing from 107973147312911
  283. 2019-11-13 10:54:39,615 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 2-2223652316014182402: Link is closed
  284. 2019-11-13 10:54:39,615 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 2-2223652316014182402: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  285. 2019-11-13 10:54:39,615 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Connection from /172.16.2.5 closed but no cleanup was done.
  286. 2019-11-13 10:54:39,600 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 2-2223652316014182401: Routing from 107973147312911
  287. 2019-11-13 10:54:39,623 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 2-2223652316014182401: Link is closed
  288. 2019-11-13 10:54:39,623 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 2-2223652316014182401: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  289. 2019-11-13 10:54:39,610 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 2-2223652316014182403: Routing from 107973147312911
  290. 2019-11-13 10:54:39,636 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 2-2223652316014182403: Link is closed
  291. 2019-11-13 10:54:39,636 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 2-2223652316014182403: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  292. 2019-11-13 10:54:39,609 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-cef5f328) (logid:40b2a78a) ===START=== 172.16.6.39 -- GET apikey=GDTBAQ0g2QY5MXb-cyp8Cwel2Ee6CPSrgry8kzo7rxo9dvfLtCNWNfI18qDLOxwdp8hpCTjaTX8-sDMYj22q3A&command=listVirtualMachines&response=json&signature=P8tjETWxf64PI35Nod2P1METQNI%3D
  293. 2019-11-13 10:54:39,746 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-cef5f328 ctx-582bd159) (logid:40b2a78a) CIDRs from which account 'Acct[7ee02798-9ccc-4ba1-bb93-b0cec3e897c7-admin-axiodis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  294. 2019-11-13 10:54:39,884 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-cef5f328 ctx-582bd159 ctx-04394630) (logid:40b2a78a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  295. 2019-11-13 10:54:39,894 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-cef5f328 ctx-582bd159 ctx-04394630) (logid:40b2a78a) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  296. 2019-11-13 10:54:40,020 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) SSL: Handshake done with peer management server '107973147312911' on 172.16.2.7:8250
  297. 2019-11-13 10:54:40,020 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Connection to peer opened: 107973147312911, ip: 172.16.2.7
  298. 2019-11-13 10:54:40,021 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  299. 2019-11-13 10:54:40,022 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  300. 2019-11-13 10:54:40,025 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  301. 2019-11-13 10:54:40,025 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  302. 2019-11-13 10:54:40,066 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-cef5f328 ctx-582bd159 ctx-04394630) (logid:40b2a78a) ===END=== 172.16.6.39 -- GET apikey=GDTBAQ0g2QY5MXb-cyp8Cwel2Ee6CPSrgry8kzo7rxo9dvfLtCNWNfI18qDLOxwdp8hpCTjaTX8-sDMYj22q3A&command=listVirtualMachines&response=json&signature=P8tjETWxf64PI35Nod2P1METQNI%3D
  303. 2019-11-13 10:54:40,076 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-a0a4da32) (logid:54c4740e) ===START=== 172.16.6.39 -- GET apikey=dOo2VogyqgcnC-y9jmnZlGZ04_xhgOk2j9bzDsvyU5gSDV_jjeCUEaf-AyLSoWrDGBje9BshbghOLcWxMvEidA&command=listVirtualMachines&response=json&signature=Mzbot%2BSM%2BC3zGGF5cU%2BhEZfHGqM%3D
  304. 2019-11-13 10:54:40,084 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-a0a4da32 ctx-c034301e) (logid:54c4740e) CIDRs from which account 'Acct[babb2439-5bbc-4e50-90cf-ec3978ba6ee6-admin-apontador-hmlg]' is allowed to perform API calls: 0.0.0.0/0,::/0
  305. 2019-11-13 10:54:40,099 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-a0a4da32 ctx-c034301e ctx-d53ec714) (logid:54c4740e) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  306. 2019-11-13 10:54:40,101 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-a0a4da32 ctx-c034301e ctx-d53ec714) (logid:54c4740e) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  307. 2019-11-13 10:54:40,427 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-a0a4da32 ctx-c034301e ctx-d53ec714) (logid:54c4740e) ===END=== 172.16.6.39 -- GET apikey=dOo2VogyqgcnC-y9jmnZlGZ04_xhgOk2j9bzDsvyU5gSDV_jjeCUEaf-AyLSoWrDGBje9BshbghOLcWxMvEidA&command=listVirtualMachines&response=json&signature=Mzbot%2BSM%2BC3zGGF5cU%2BhEZfHGqM%3D
  308. 2019-11-13 10:54:40,435 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-7b67e9a9) (logid:7a55e157) ===START=== 172.16.6.39 -- GET apikey=uhGUyYK7Zl27dM_ZTX4d_KQNQx2hUWD9qqAS-SkzrVNDVlWtELCQgjVU29g2_HSwy98jVWZHSgrd7WeFTigMfg&command=listVirtualMachines&response=json&signature=V%2BQZ6VBtlZbgm%2B3FFa8oKQVyn6c%3D
  309. 2019-11-13 10:54:40,445 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-7b67e9a9 ctx-427d494a) (logid:7a55e157) CIDRs from which account 'Acct[356e266d-a280-4548-9a94-e6dca9336148-admin-axiodis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  310. 2019-11-13 10:54:40,460 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-7b67e9a9 ctx-427d494a ctx-44a1db4d) (logid:7a55e157) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  311. 2019-11-13 10:54:40,461 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-7b67e9a9 ctx-427d494a ctx-44a1db4d) (logid:7a55e157) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  312. 2019-11-13 10:54:40,609 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-3bb40133) (logid:78e8f80a) Begin cleanup expired async-jobs
  313. 2019-11-13 10:54:40,617 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-3bb40133) (logid:78e8f80a) End cleanup expired async-jobs
  314. 2019-11-13 10:54:40,804 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-7b67e9a9 ctx-427d494a ctx-44a1db4d) (logid:7a55e157) ===END=== 172.16.6.39 -- GET apikey=uhGUyYK7Zl27dM_ZTX4d_KQNQx2hUWD9qqAS-SkzrVNDVlWtELCQgjVU29g2_HSwy98jVWZHSgrd7WeFTigMfg&command=listVirtualMachines&response=json&signature=V%2BQZ6VBtlZbgm%2B3FFa8oKQVyn6c%3D
  315. 2019-11-13 10:54:41,737 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-715eaedf) (logid:cefb3549) Zone 1 is not ready to launch secondary storage VM yet
  316. 2019-11-13 10:54:42,083 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-9dcd5be7) (logid:826c64d6) Zone 1 is not ready to launch console proxy yet
  317. 2019-11-13 10:54:45,890 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-53b59e3f) (logid:b51e186f) AutoScaling Monitor is running...
  318. 2019-11-13 10:54:46,988 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Seq 2-2223652316014182402: Waiting some more time because this is the current command
  319. 2019-11-13 10:54:46,989 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  320. 2019-11-13 10:54:46,989 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Seq 2-2223652316014182402: Timed out on null
  321. 2019-11-13 10:54:46,990 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Seq 2-2223652316014182402: Cancelling.
  322. 2019-11-13 10:54:46,990 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Operation timed out: Commands 2223652316014182402 to Host 2 timed out after 120
  323. 2019-11-13 10:54:46,998 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Host 3 has switched to another management server, need to update agent map with a forwarding agent attache
  324. 2019-11-13 10:54:46,998 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) create forwarding ClusteredAgentAttache for 3
  325. 2019-11-13 10:54:47,002 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:ctx-5e396d28) (logid:f1c3ce8c) Seq 3-7917609619894042625: Forwarding null to 204920354261423
  326. 2019-11-13 10:54:50,610 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-f2c53131) (logid:fa31da0d) Begin cleanup expired async-jobs
  327. 2019-11-13 10:54:50,619 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-f2c53131) (logid:fa31da0d) End cleanup expired async-jobs
  328. 2019-11-13 10:55:00,612 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-aabc89c2) (logid:92d3bd34) Begin cleanup expired async-jobs
  329. 2019-11-13 10:55:00,621 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-aabc89c2) (logid:92d3bd34) End cleanup expired async-jobs
  330. 2019-11-13 10:55:01,362 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5dbe7ec6) (logid:4cd42245) Found 117 routers to update status.
  331. 2019-11-13 10:55:01,551 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5dbe7ec6) (logid:4cd42245) Found 150 VPC networks to update Redundant State.
  332. 2019-11-13 10:55:01,562 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5dbe7ec6) (logid:4cd42245) Found 0 networks to update RvR status.
  333. 2019-11-13 10:55:01,583 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-5:ctx-43caafa8) (logid:42a920b4) Skip router pair (r-2464-VM,r-2465-VM) due to not belong to this mgmt server
  334. 2019-11-13 10:55:01,659 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-6:ctx-2ad603f7) (logid:cd7a7138) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  335. 2019-11-13 10:55:01,666 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-8:ctx-8a740790) (logid:d8d962e2) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  336. 2019-11-13 10:55:10,612 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-35600460) (logid:89f6781a) Begin cleanup expired async-jobs
  337. 2019-11-13 10:55:10,623 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-35600460) (logid:89f6781a) End cleanup expired async-jobs
  338. 2019-11-13 10:55:11,737 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-b4bb1d4c) (logid:47bb551b) Zone 1 is not ready to launch secondary storage VM yet
  339. 2019-11-13 10:55:12,084 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4a0f168a) (logid:4f5cb435) Zone 1 is not ready to launch console proxy yet
  340. 2019-11-13 10:55:20,611 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9c4148f3) (logid:8b36eab4) Begin cleanup expired async-jobs
  341. 2019-11-13 10:55:20,620 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9c4148f3) (logid:8b36eab4) End cleanup expired async-jobs
  342. 2019-11-13 10:55:30,613 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6e6ba5bf) (logid:74477f24) Begin cleanup expired async-jobs
  343. 2019-11-13 10:55:30,623 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-6e6ba5bf) (logid:74477f24) End cleanup expired async-jobs
  344. 2019-11-13 10:55:31,363 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-46933b1c) (logid:9400098b) Found 117 routers to update status.
  345. 2019-11-13 10:55:31,368 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-a612c9fc) (logid:0c46690f) ===START=== 172.16.6.39 -- GET apikey=Iuj-MzZYb-O2-drEybbjpOvhANmcPnQa3a_-Y9m3xb1l0dZGpO63Z-Dti2IKdFw_M-0s9SEog6c6tcu5wymu2w&command=listVirtualMachines&response=json&signature=6GTPUREJsCbXCk%2BpCZy6k28rEwY%3D
  346. 2019-11-13 10:55:31,377 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-a612c9fc ctx-1d592c8f) (logid:0c46690f) CIDRs from which account 'Acct[57bda660-67fb-4720-a1ee-183d6016e61e-infra]' is allowed to perform API calls: 0.0.0.0/0,::/0
  347. 2019-11-13 10:55:31,394 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-a612c9fc ctx-1d592c8f ctx-892f13c2) (logid:0c46690f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  348. 2019-11-13 10:55:31,395 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-a612c9fc ctx-1d592c8f ctx-892f13c2) (logid:0c46690f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  349. 2019-11-13 10:55:31,516 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-a612c9fc ctx-1d592c8f ctx-892f13c2) (logid:0c46690f) ===END=== 172.16.6.39 -- GET apikey=Iuj-MzZYb-O2-drEybbjpOvhANmcPnQa3a_-Y9m3xb1l0dZGpO63Z-Dti2IKdFw_M-0s9SEog6c6tcu5wymu2w&command=listVirtualMachines&response=json&signature=6GTPUREJsCbXCk%2BpCZy6k28rEwY%3D
  350. 2019-11-13 10:55:31,524 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-6c4aa169) (logid:5209a701) ===START=== 172.16.6.39 -- GET apikey=phwlEyvnaj_6pDIgP4x1Q7gY__S1puxxBLnJ6eOQ0jVEN-URH1e1O3YCS8bL6FgtnqZ75MGG2rVMah6E6wP_NQ&command=listVirtualMachines&response=json&signature=RLb%2FD4MZ64j3cxfgbfAqWmi0zI4%3D
  351. 2019-11-13 10:55:31,542 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-6c4aa169 ctx-8ef29b91) (logid:5209a701) CIDRs from which account 'Acct[93e181f7-e3a9-4221-81ca-1c7564c4b552-admin-gis]' is allowed to perform API calls: 0.0.0.0/0,::/0
  352. 2019-11-13 10:55:31,558 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-6c4aa169 ctx-8ef29b91 ctx-06228caa) (logid:5209a701) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  353. 2019-11-13 10:55:31,559 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-6c4aa169 ctx-8ef29b91 ctx-06228caa) (logid:5209a701) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  354. 2019-11-13 10:55:31,581 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-46933b1c) (logid:9400098b) Found 150 VPC networks to update Redundant State.
  355. 2019-11-13 10:55:31,582 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-6c4aa169 ctx-8ef29b91 ctx-06228caa) (logid:5209a701) ===END=== 172.16.6.39 -- GET apikey=phwlEyvnaj_6pDIgP4x1Q7gY__S1puxxBLnJ6eOQ0jVEN-URH1e1O3YCS8bL6FgtnqZ75MGG2rVMah6E6wP_NQ&command=listVirtualMachines&response=json&signature=RLb%2FD4MZ64j3cxfgbfAqWmi0zI4%3D
  356. 2019-11-13 10:55:31,588 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-4ba02264) (logid:a219b496) ===START=== 172.16.6.39 -- GET apikey=TzX6yP-0YuuDL9KeOabcYsVV-XL3dO8bBe6LerLbXuk-0HX-4_JFJxRl3MyOolddMILMN_L_li8VCuj02UsqYA&command=listVirtualMachines&response=json&signature=zmwagNWfKJ2BrLhV7dB4vsNhfe0%3D
  357. 2019-11-13 10:55:31,592 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-46933b1c) (logid:9400098b) Found 0 networks to update RvR status.
  358. 2019-11-13 10:55:31,631 DEBUG [c.c.a.ApiServer] (qtp504527234-15:ctx-4ba02264 ctx-ef0d1149) (logid:a219b496) CIDRs from which account 'Acct[99a93ba4-e2aa-462a-9473-d7ec3ef947db-admin-apontador]' is allowed to perform API calls: 0.0.0.0/0,::/0
  359. 2019-11-13 10:55:31,632 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-9:ctx-9b95b854) (logid:e8939ebf) Skip router pair (r-2464-VM,r-2465-VM) due to not belong to this mgmt server
  360. 2019-11-13 10:55:31,670 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-1:ctx-93c1a0ba) (logid:d485fd72) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  361. 2019-11-13 10:55:31,675 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-7:ctx-3d674e63) (logid:ae6399a2) Skip router pair (r-2425-VM,r-2460-VM) due to not belong to this mgmt server
  362. 2019-11-13 10:55:31,682 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-4ba02264 ctx-ef0d1149 ctx-986552b2) (logid:a219b496) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  363. 2019-11-13 10:55:31,692 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-15:ctx-4ba02264 ctx-ef0d1149 ctx-986552b2) (logid:a219b496) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  364. 2019-11-13 10:55:34,125 DEBUG [c.c.a.ApiServlet] (qtp504527234-15:ctx-4ba02264 ctx-ef0d1149 ctx-986552b2) (logid:a219b496) ===END=== 172.16.6.39 -- GET apikey=TzX6yP-0YuuDL9KeOabcYsVV-XL3dO8bBe6LerLbXuk-0HX-4_JFJxRl3MyOolddMILMN_L_li8VCuj02UsqYA&command=listVirtualMachines&response=json&signature=zmwagNWfKJ2BrLhV7dB4vsNhfe0%3D
  365. 2019-11-13 10:55:34,151 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-c037e698) (logid:45d87d12) ===START=== 172.16.6.39 -- GET apikey=1X2TrYPn9wWH2h7NhvNfuhgt_O5YgQ-4MzTZ_9CfWKvoC3fO-bgCVBupGWyXnsfM8w3CLhdWAvdmQZZsRJjVDQ&command=listVirtualMachines&response=json&signature=gnFfZrawiC6bSo1LjD1%2B8pz5Pw8%3D
  366. 2019-11-13 10:55:34,171 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-c037e698 ctx-6dda40e9) (logid:45d87d12) CIDRs from which account 'Acct[2f3b55e6-0c5b-4c98-bd32-3dbe6110caa7-admin-unidadedenegocio]' is allowed to perform API calls: 0.0.0.0/0,::/0
  367. 2019-11-13 10:55:34,200 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-c037e698 ctx-6dda40e9 ctx-16537723) (logid:45d87d12) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  368. 2019-11-13 10:55:34,202 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-c037e698 ctx-6dda40e9 ctx-16537723) (logid:45d87d12) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  369. 2019-11-13 10:55:34,219 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-c037e698 ctx-6dda40e9 ctx-16537723) (logid:45d87d12) ===END=== 172.16.6.39 -- GET apikey=1X2TrYPn9wWH2h7NhvNfuhgt_O5YgQ-4MzTZ_9CfWKvoC3fO-bgCVBupGWyXnsfM8w3CLhdWAvdmQZZsRJjVDQ&command=listVirtualMachines&response=json&signature=gnFfZrawiC6bSo1LjD1%2B8pz5Pw8%3D
  370. 2019-11-13 10:55:34,227 DEBUG [c.c.a.ApiServlet] (qtp504527234-18:ctx-0903342e) (logid:71913d4f) ===START=== 172.16.6.39 -- GET apikey=D851k2JpSGcUj2hv0cdlzohx5VNWHpZ100CSnDBHxLoLYMPosuKOWjnz24G1AYHyzA3jz73Y51S320C7gxLTyA&command=listVirtualMachines&response=json&signature=Lndcon41%2Fv5npjdR15QHFRh%2BnMg%3D
  371. 2019-11-13 10:55:34,239 DEBUG [c.c.a.ApiServer] (qtp504527234-18:ctx-0903342e ctx-ce5aa353) (logid:71913d4f) CIDRs from which account 'Acct[66258daa-3ace-4378-80a0-286bf8335457-admin-maplink-hmlg]' is allowed to perform API calls: 0.0.0.0/0,::/0
  372. 2019-11-13 10:55:34,255 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-18:ctx-0903342e ctx-ce5aa353 ctx-48d1448b) (logid:71913d4f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  373. 2019-11-13 10:55:34,257 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-18:ctx-0903342e ctx-ce5aa353 ctx-48d1448b) (logid:71913d4f) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  374. 2019-11-13 10:55:34,881 DEBUG [c.c.a.ApiServlet] (qtp504527234-18:ctx-0903342e ctx-ce5aa353 ctx-48d1448b) (logid:71913d4f) ===END=== 172.16.6.39 -- GET apikey=D851k2JpSGcUj2hv0cdlzohx5VNWHpZ100CSnDBHxLoLYMPosuKOWjnz24G1AYHyzA3jz73Y51S320C7gxLTyA&command=listVirtualMachines&response=json&signature=Lndcon41%2Fv5npjdR15QHFRh%2BnMg%3D
  375. 2019-11-13 10:55:34,890 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-e977921e) (logid:62bfd1a1) ===START=== 172.16.6.39 -- GET apikey=mRlv1uPRoQbizxMZm2fN1PRKrzuqzwRcd-I-ult0FaefeHtiO-aF7U6PXD12fCBD57fmijoKvkoETqiBn6nZXA&command=listVirtualMachines&response=json&signature=ej%2Bs7kPyeqKAcahkWykP4LSnKZc%3D
  376. 2019-11-13 10:55:34,899 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-e977921e ctx-ee3697c5) (logid:62bfd1a1) CIDRs from which account 'Acct[df9f070c-2e4a-42ea-9b69-acfbf954b00d-admin-maplink-dev]' is allowed to perform API calls: 0.0.0.0/0,::/0
  377. 2019-11-13 10:55:34,915 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-e977921e ctx-ee3697c5 ctx-92a05779) (logid:62bfd1a1) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  378. 2019-11-13 10:55:34,917 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-17:ctx-e977921e ctx-ee3697c5 ctx-92a05779) (logid:62bfd1a1) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  379. 2019-11-13 10:55:34,996 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-e977921e ctx-ee3697c5 ctx-92a05779) (logid:62bfd1a1) ===END=== 172.16.6.39 -- GET apikey=mRlv1uPRoQbizxMZm2fN1PRKrzuqzwRcd-I-ult0FaefeHtiO-aF7U6PXD12fCBD57fmijoKvkoETqiBn6nZXA&command=listVirtualMachines&response=json&signature=ej%2Bs7kPyeqKAcahkWykP4LSnKZc%3D
  380. 2019-11-13 10:55:35,004 DEBUG [c.c.a.ApiServlet] (qtp504527234-18:ctx-c57bfe95) (logid:909a3c9d) ===START=== 172.16.6.39 -- GET apikey=9xG8Ekz2TzUVRgaxLa7gqgIukWRZzMwEScNh-Ufgb-iV-VHoUxZNf4tjTrWBFvy3B1F2fSOhooUI0ZnNqWMMQA&command=listVirtualMachines&response=json&signature=mbokVEPWP0ddlKcTDDlewjJLwVc%3D
  381. 2019-11-13 10:55:35,013 DEBUG [c.c.a.ApiServer] (qtp504527234-18:ctx-c57bfe95 ctx-1af49e03) (logid:909a3c9d) CIDRs from which account 'Acct[d1566f21-219f-4149-a879-76fbedb170de-admin-maplink]' is allowed to perform API calls: 0.0.0.0/0,::/0
  382. 2019-11-13 10:55:35,029 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-18:ctx-c57bfe95 ctx-1af49e03 ctx-02391549) (logid:909a3c9d) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  383. 2019-11-13 10:55:35,031 DEBUG [o.a.c.a.BaseCmd] (qtp504527234-18:ctx-c57bfe95 ctx-1af49e03 ctx-02391549) (logid:909a3c9d) Ignoring paremeter displayvm as the caller is not authorized to pass it in
  384. 2019-11-13 10:55:35,832 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ce894426) (logid:5f984231) Resetting hosts suitable for reconnect
  385. 2019-11-13 10:55:35,835 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ce894426) (logid:5f984231) Completed resetting hosts suitable for reconnect
  386. 2019-11-13 10:55:35,835 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ce894426) (logid:5f984231) Acquiring hosts for clusters already owned by this management server
  387. 2019-11-13 10:55:35,837 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ce894426) (logid:5f984231) Completed acquiring hosts for clusters already owned by this management server
  388. 2019-11-13 10:55:35,837 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ce894426) (logid:5f984231) Acquiring hosts for clusters not owned by any management server
  389. 2019-11-13 10:55:35,838 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ce894426) (logid:5f984231) Completed acquiring hosts for clusters not owned by any management server
  390. 2019-11-13 10:55:36,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  391. 2019-11-13 10:55:36,267 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  392. 2019-11-13 10:55:36,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  393. 2019-11-13 10:55:36,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  394. 2019-11-13 10:55:36,269 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 9-1463388403918700545: Cancelling.
  395. 2019-11-13 10:55:36,270 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700545: Waiting some more time because this is the current command
  396. 2019-11-13 10:55:36,270 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700545: Waiting some more time because this is the current command
  397. 2019-11-13 10:55:36,267 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 2-2223652316014182403: Cancelling.
  398. 2019-11-13 10:55:36,270 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 2-2223652316014182403: Waiting some more time because this is the current command
  399. 2019-11-13 10:55:36,270 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 2-2223652316014182403: Waiting some more time because this is the current command
  400. 2019-11-13 10:55:36,270 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 2-2223652316014182401: Cancelling.
  401. 2019-11-13 10:55:36,271 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182401: Waiting some more time because this is the current command
  402. 2019-11-13 10:55:36,271 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182401: Waiting some more time because this is the current command
  403. 2019-11-13 10:55:36,271 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 2-2223652316014182402: Cancelling.
  404. 2019-11-13 10:55:36,270 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  405. 2019-11-13 10:55:36,276 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700545: Timed out on null
  406. 2019-11-13 10:55:36,277 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700545: Cancelling.
  407. 2019-11-13 10:55:36,277 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 9
  408. com.cloud.exception.OperationTimedoutException: Commands 1463388403918700545 to Host 9 timed out after 3600
  409. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  410. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  411. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  412. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  413. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  414. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  415. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  416. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  417. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  418. at java.lang.reflect.Method.invoke(Method.java:498)
  419. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  420. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  421. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  422. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  423. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  424. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  425. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  426. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  427. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  428. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  429. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  430. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  431. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  432. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  433. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  434. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  435. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  436. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  437. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  438. at java.lang.Thread.run(Thread.java:748)
  439. 2019-11-13 10:55:36,277 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  440. 2019-11-13 10:55:36,277 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  441. 2019-11-13 10:55:36,271 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  442. 2019-11-13 10:55:36,279 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182401: Timed out on null
  443. 2019-11-13 10:55:36,280 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182401: Cancelling.
  444. 2019-11-13 10:55:36,280 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 2223652316014182401 to Host 2 timed out after 3600
  445. 2019-11-13 10:55:36,280 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 2 statistics.
  446. 2019-11-13 10:55:36,280 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 2
  447. 2019-11-13 10:55:36,270 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  448. 2019-11-13 10:55:36,281 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 2-2223652316014182403: Timed out on null
  449. 2019-11-13 10:55:36,281 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 2-2223652316014182403: Cancelling.
  450. 2019-11-13 10:55:36,281 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Operation timed out: Commands 2223652316014182403 to Host 2 timed out after 3600
  451. 2019-11-13 10:55:36,281 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Unable to obtain VM statistics.
  452. 2019-11-13 10:55:36,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Host 10 has switched to another management server, need to update agent map with a forwarding agent attache
  453. 2019-11-13 10:55:36,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) create forwarding ClusteredAgentAttache for 10
  454. 2019-11-13 10:55:36,296 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Forwarding null to 204920354261423
  455. 2019-11-13 10:55:36,297 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Error on connecting to management node: null try = 1
  456. 2019-11-13 10:55:36,297 INFO [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) IOException Broken pipe when sending data to peer 204920354261423, close peer connection and let it re-open
  457. 2019-11-13 10:55:36,605 INFO [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) SSL: Handshake done with peer management server '204920354261423' on 172.16.1.44:8250
  458. 2019-11-13 10:55:36,605 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Connection to peer opened: 204920354261423, ip: 172.16.1.44
  459. 2019-11-13 10:55:36,606 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Forwarding null to 204920354261423
  460. 2019-11-13 10:55:36,610 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042626: Forwarding null to 204920354261423
  461. 2019-11-13 10:55:36,847 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 3-7917609619894042627: Forwarding null to 204920354261423
  462. 2019-11-13 10:55:37,257 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 10-6636054050930425857: Routing from 107973147312911
  463. 2019-11-13 10:55:37,262 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 10-6636054050930425857: Link is closed
  464. 2019-11-13 10:55:37,262 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 10-6636054050930425857: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  465. 2019-11-13 10:55:37,262 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  466. 2019-11-13 10:55:37,260 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 3-7917609619894042627: Routing from 107973147312911
  467. 2019-11-13 10:55:37,264 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 3-7917609619894042627: Link is closed
  468. 2019-11-13 10:55:37,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 3-7917609619894042627: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  469. 2019-11-13 10:55:37,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  470. 2019-11-13 10:55:37,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  471. 2019-11-13 10:55:37,259 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 3-7917609619894042626: Routing from 107973147312911
  472. 2019-11-13 10:55:37,267 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 3-7917609619894042626: Link is closed
  473. 2019-11-13 10:55:37,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 3-7917609619894042626: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  474. 2019-11-13 10:55:37,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  475. 2019-11-13 10:55:37,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  476. 2019-11-13 10:55:37,270 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 3-7917609619894042627: Cancelling.
  477. 2019-11-13 10:55:37,270 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 3-7917609619894042627: Waiting some more time because this is the current command
  478. 2019-11-13 10:55:37,271 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 3-7917609619894042627: Waiting some more time because this is the current command
  479. 2019-11-13 10:55:37,271 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  480. 2019-11-13 10:55:37,271 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 3-7917609619894042627: Timed out on null
  481. 2019-11-13 10:55:37,271 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 3-7917609619894042627: Cancelling.
  482. 2019-11-13 10:55:37,271 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Operation timed out: Commands 7917609619894042627 to Host 3 timed out after 3600
  483. 2019-11-13 10:55:37,271 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Unable to obtain VM statistics.
  484. 2019-11-13 10:55:37,267 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 10-6636054050930425857: Cancelling.
  485. 2019-11-13 10:55:37,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  486. 2019-11-13 10:55:37,281 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 3-7917609619894042626: Cancelling.
  487. 2019-11-13 10:55:37,281 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042626: Waiting some more time because this is the current command
  488. 2019-11-13 10:55:37,281 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042626: Waiting some more time because this is the current command
  489. 2019-11-13 10:55:37,281 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  490. 2019-11-13 10:55:37,281 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042626: Timed out on null
  491. 2019-11-13 10:55:37,281 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042626: Cancelling.
  492. 2019-11-13 10:55:37,281 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 7917609619894042626 to Host 3 timed out after 3600
  493. 2019-11-13 10:55:37,281 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 3 statistics.
  494. 2019-11-13 10:55:37,282 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 3
  495. 2019-11-13 10:55:37,286 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Waiting some more time because this is the current command
  496. 2019-11-13 10:55:37,286 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Waiting some more time because this is the current command
  497. 2019-11-13 10:55:37,286 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  498. 2019-11-13 10:55:37,286 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Timed out on null
  499. 2019-11-13 10:55:37,286 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425857: Cancelling.
  500. 2019-11-13 10:55:37,286 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 10
  501. com.cloud.exception.OperationTimedoutException: Commands 6636054050930425857 to Host 10 timed out after 3600
  502. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  503. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  504. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  505. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  506. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  507. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  508. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  509. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  510. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  511. at java.lang.reflect.Method.invoke(Method.java:498)
  512. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  513. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  514. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  515. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  516. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  517. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  518. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  519. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  520. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  521. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  522. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  523. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  524. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  525. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  526. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  527. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  528. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  529. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  530. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  531. at java.lang.Thread.run(Thread.java:748)
  532. 2019-11-13 10:55:37,287 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  533. 2019-11-13 10:55:37,287 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  534. 2019-11-13 10:55:37,305 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182404: Forwarding null to 204920354261423
  535. 2019-11-13 10:55:37,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 2-2223652316014182404: Routing from 107973147312911
  536. 2019-11-13 10:55:37,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 2-2223652316014182404: Link is closed
  537. 2019-11-13 10:55:37,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 2-2223652316014182404: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  538. 2019-11-13 10:55:37,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  539. 2019-11-13 10:55:37,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  540. 2019-11-13 10:55:37,314 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 2-2223652316014182404: Cancelling.
  541. 2019-11-13 10:55:37,319 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182404: Waiting some more time because this is the current command
  542. 2019-11-13 10:55:37,319 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182404: Waiting some more time because this is the current command
  543. 2019-11-13 10:55:37,319 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  544. 2019-11-13 10:55:37,319 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182404: Timed out on null
  545. 2019-11-13 10:55:37,320 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182404: Cancelling.
  546. 2019-11-13 10:55:37,320 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 2
  547. com.cloud.exception.OperationTimedoutException: Commands 2223652316014182404 to Host 2 timed out after 3600
  548. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  549. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  550. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  551. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  552. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  553. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  554. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  555. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  556. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  557. at java.lang.reflect.Method.invoke(Method.java:498)
  558. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  559. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  560. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  561. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  562. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  563. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  564. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  565. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  566. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  567. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  568. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  569. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  570. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  571. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  572. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  573. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  574. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  575. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  576. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  577. at java.lang.Thread.run(Thread.java:748)
  578. 2019-11-13 10:55:37,320 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  579. 2019-11-13 10:55:37,320 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  580. 2019-11-13 10:55:37,325 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700546: Forwarding null to 204920354261423
  581. 2019-11-13 10:55:37,333 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 9-1463388403918700546: Routing from 107973147312911
  582. 2019-11-13 10:55:37,333 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 9-1463388403918700546: Link is closed
  583. 2019-11-13 10:55:37,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 9-1463388403918700546: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  584. 2019-11-13 10:55:37,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  585. 2019-11-13 10:55:37,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  586. 2019-11-13 10:55:37,339 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 9-1463388403918700546: Cancelling.
  587. 2019-11-13 10:55:37,339 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700546: Waiting some more time because this is the current command
  588. 2019-11-13 10:55:37,339 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700546: Waiting some more time because this is the current command
  589. 2019-11-13 10:55:37,339 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  590. 2019-11-13 10:55:37,339 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700546: Timed out on null
  591. 2019-11-13 10:55:37,340 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700546: Cancelling.
  592. 2019-11-13 10:55:37,340 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 1463388403918700546 to Host 9 timed out after 3600
  593. 2019-11-13 10:55:37,340 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 9 statistics.
  594. 2019-11-13 10:55:37,340 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 9
  595. 2019-11-13 10:55:37,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Host 74673 has switched to another management server, need to update agent map with a forwarding agent attache
  596. 2019-11-13 10:55:37,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) create forwarding ClusteredAgentAttache for 74673
  597. 2019-11-13 10:55:37,354 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082177: Forwarding null to 204920354261423
  598. 2019-11-13 10:55:37,355 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 74673-7260928499228082177: Routing from 107973147312911
  599. 2019-11-13 10:55:37,355 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 74673-7260928499228082177: Link is closed
  600. 2019-11-13 10:55:37,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 74673-7260928499228082177: MgmtId 107973147312911: Req: Resource [Host:74673] is unreachable: Host 74673: Link is closed
  601. 2019-11-13 10:55:37,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Routing to peer
  602. 2019-11-13 10:55:37,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Cancel request received
  603. 2019-11-13 10:55:37,360 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 74673-7260928499228082177: Cancelling.
  604. 2019-11-13 10:55:37,361 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082177: Waiting some more time because this is the current command
  605. 2019-11-13 10:55:37,361 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082177: Waiting some more time because this is the current command
  606. 2019-11-13 10:55:37,361 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  607. 2019-11-13 10:55:37,361 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082177: Timed out on null
  608. 2019-11-13 10:55:37,361 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082177: Cancelling.
  609. 2019-11-13 10:55:37,361 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 74673
  610. com.cloud.exception.OperationTimedoutException: Commands 7260928499228082177 to Host 74673 timed out after 3600
  611. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  612. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  613. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  614. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  615. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  616. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  617. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  618. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  619. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  620. at java.lang.reflect.Method.invoke(Method.java:498)
  621. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  622. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  623. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  624. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  625. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  626. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  627. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  628. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  629. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  630. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  631. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  632. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  633. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  634. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  635. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  636. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  637. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  638. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  639. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  640. at java.lang.Thread.run(Thread.java:748)
  641. 2019-11-13 10:55:37,362 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  642. 2019-11-13 10:55:37,362 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  643. 2019-11-13 10:55:37,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Host 19 has switched to another management server, need to update agent map with a forwarding agent attache
  644. 2019-11-13 10:55:37,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) create forwarding ClusteredAgentAttache for 19
  645. 2019-11-13 10:55:37,371 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425858: Forwarding null to 204920354261423
  646. 2019-11-13 10:55:37,372 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 10-6636054050930425858: Routing from 107973147312911
  647. 2019-11-13 10:55:37,372 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 10-6636054050930425858: Link is closed
  648. 2019-11-13 10:55:37,372 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 10-6636054050930425858: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  649. 2019-11-13 10:55:37,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  650. 2019-11-13 10:55:37,374 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  651. 2019-11-13 10:55:37,374 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 10-6636054050930425858: Cancelling.
  652. 2019-11-13 10:55:37,374 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425858: Waiting some more time because this is the current command
  653. 2019-11-13 10:55:37,374 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425858: Waiting some more time because this is the current command
  654. 2019-11-13 10:55:37,374 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  655. 2019-11-13 10:55:37,375 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425858: Timed out on null
  656. 2019-11-13 10:55:37,375 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425858: Cancelling.
  657. 2019-11-13 10:55:37,375 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 6636054050930425858 to Host 10 timed out after 3600
  658. 2019-11-13 10:55:37,375 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 10 statistics.
  659. 2019-11-13 10:55:37,375 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 10
  660. 2019-11-13 10:55:37,376 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649153: Forwarding null to 204920354261423
  661. 2019-11-13 10:55:37,378 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 19-7006475120281649153: Routing from 107973147312911
  662. 2019-11-13 10:55:37,380 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 19-7006475120281649153: Link is closed
  663. 2019-11-13 10:55:37,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 19-7006475120281649153: MgmtId 107973147312911: Req: Resource [Host:19] is unreachable: Host 19: Link is closed
  664. 2019-11-13 10:55:37,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Routing to peer
  665. 2019-11-13 10:55:37,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Cancel request received
  666. 2019-11-13 10:55:37,384 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 19-7006475120281649153: Cancelling.
  667. 2019-11-13 10:55:37,384 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649153: Waiting some more time because this is the current command
  668. 2019-11-13 10:55:37,384 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649153: Waiting some more time because this is the current command
  669. 2019-11-13 10:55:37,384 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  670. 2019-11-13 10:55:37,384 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649153: Timed out on null
  671. 2019-11-13 10:55:37,384 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649153: Cancelling.
  672. 2019-11-13 10:55:37,384 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 19
  673. com.cloud.exception.OperationTimedoutException: Commands 7006475120281649153 to Host 19 timed out after 3600
  674. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  675. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  676. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  677. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  678. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  679. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  680. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  681. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  682. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  683. at java.lang.reflect.Method.invoke(Method.java:498)
  684. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  685. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  686. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  687. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  688. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  689. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  690. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  691. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  692. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  693. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  694. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  695. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  696. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  697. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  698. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  699. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  700. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  701. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  702. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  703. at java.lang.Thread.run(Thread.java:748)
  704. 2019-11-13 10:55:37,384 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  705. 2019-11-13 10:55:37,384 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  706. 2019-11-13 10:55:37,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Host 13 has switched to another management server, need to update agent map with a forwarding agent attache
  707. 2019-11-13 10:55:37,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) create forwarding ClusteredAgentAttache for 13
  708. 2019-11-13 10:55:37,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Host 15 has switched to another management server, need to update agent map with a forwarding agent attache
  709. 2019-11-13 10:55:37,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) create forwarding ClusteredAgentAttache for 15
  710. 2019-11-13 10:55:37,394 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297409: Forwarding null to 204920354261423
  711. 2019-11-13 10:55:37,395 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661121: Forwarding null to 204920354261423
  712. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 13-1616510791249297409: Routing from 107973147312911
  713. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 15-5775866522102661121: Routing from 107973147312911
  714. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 15-5775866522102661121: Link is closed
  715. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 15-5775866522102661121: MgmtId 107973147312911: Req: Resource [Host:15] is unreachable: Host 15: Link is closed
  716. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 13-1616510791249297409: Link is closed
  717. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Routing to peer
  718. 2019-11-13 10:55:37,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 13-1616510791249297409: MgmtId 107973147312911: Req: Resource [Host:13] is unreachable: Host 13: Link is closed
  719. 2019-11-13 10:55:37,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Routing to peer
  720. 2019-11-13 10:55:37,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Cancel request received
  721. 2019-11-13 10:55:37,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Cancel request received
  722. 2019-11-13 10:55:37,405 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 15-5775866522102661121: Cancelling.
  723. 2019-11-13 10:55:37,405 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 13-1616510791249297409: Cancelling.
  724. 2019-11-13 10:55:37,408 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661121: Waiting some more time because this is the current command
  725. 2019-11-13 10:55:37,408 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661121: Waiting some more time because this is the current command
  726. 2019-11-13 10:55:37,408 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297409: Waiting some more time because this is the current command
  727. 2019-11-13 10:55:37,408 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  728. 2019-11-13 10:55:37,408 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661121: Timed out on null
  729. 2019-11-13 10:55:37,408 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661121: Cancelling.
  730. 2019-11-13 10:55:37,408 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 15
  731. com.cloud.exception.OperationTimedoutException: Commands 5775866522102661121 to Host 15 timed out after 3600
  732. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  733. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  734. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  735. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  736. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  737. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  738. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  739. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  740. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  741. at java.lang.reflect.Method.invoke(Method.java:498)
  742. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  743. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  744. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  745. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  746. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  747. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  748. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  749. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  750. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  751. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  752. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  753. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  754. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  755. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  756. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  757. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  758. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  759. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  760. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  761. at java.lang.Thread.run(Thread.java:748)
  762. 2019-11-13 10:55:37,409 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  763. 2019-11-13 10:55:37,409 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  764. 2019-11-13 10:55:37,408 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297409: Waiting some more time because this is the current command
  765. 2019-11-13 10:55:37,412 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  766. 2019-11-13 10:55:37,412 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297409: Timed out on null
  767. 2019-11-13 10:55:37,412 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297409: Cancelling.
  768. 2019-11-13 10:55:37,412 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 1616510791249297409 to Host 13 timed out after 3600
  769. 2019-11-13 10:55:37,412 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 13 statistics.
  770. 2019-11-13 10:55:37,412 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 13
  771. 2019-11-13 10:55:37,423 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042628: Forwarding null to 204920354261423
  772. 2019-11-13 10:55:37,425 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 3-7917609619894042628: Routing from 107973147312911
  773. 2019-11-13 10:55:37,425 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 3-7917609619894042628: Link is closed
  774. 2019-11-13 10:55:37,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 3-7917609619894042628: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  775. 2019-11-13 10:55:37,426 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  776. 2019-11-13 10:55:37,436 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661122: Forwarding null to 204920354261423
  777. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 15-5775866522102661122: Routing from 107973147312911
  778. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 15-5775866522102661122: Link is closed
  779. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 15-5775866522102661122: MgmtId 107973147312911: Req: Resource [Host:15] is unreachable: Host 15: Link is closed
  780. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Routing to peer
  781. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  782. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 3-7917609619894042628: Cancelling.
  783. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042628: Waiting some more time because this is the current command
  784. 2019-11-13 10:55:37,438 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042628: Waiting some more time because this is the current command
  785. 2019-11-13 10:55:37,438 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  786. 2019-11-13 10:55:37,439 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042628: Timed out on null
  787. 2019-11-13 10:55:37,439 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042628: Cancelling.
  788. 2019-11-13 10:55:37,439 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 3
  789. com.cloud.exception.OperationTimedoutException: Commands 7917609619894042628 to Host 3 timed out after 3600
  790. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  791. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  792. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  793. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  794. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  795. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  796. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  797. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  798. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  799. at java.lang.reflect.Method.invoke(Method.java:498)
  800. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  801. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  802. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  803. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  804. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  805. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  806. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  807. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  808. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  809. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  810. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  811. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  812. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  813. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  814. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  815. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  816. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  817. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  818. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  819. at java.lang.Thread.run(Thread.java:748)
  820. 2019-11-13 10:55:37,439 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  821. 2019-11-13 10:55:37,439 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  822. 2019-11-13 10:55:37,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Cancel request received
  823. 2019-11-13 10:55:37,443 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 15-5775866522102661122: Cancelling.
  824. 2019-11-13 10:55:37,445 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661122: Waiting some more time because this is the current command
  825. 2019-11-13 10:55:37,445 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661122: Waiting some more time because this is the current command
  826. 2019-11-13 10:55:37,445 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  827. 2019-11-13 10:55:37,445 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661122: Timed out on null
  828. 2019-11-13 10:55:37,445 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661122: Cancelling.
  829. 2019-11-13 10:55:37,445 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 5775866522102661122 to Host 15 timed out after 3600
  830. 2019-11-13 10:55:37,445 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 15 statistics.
  831. 2019-11-13 10:55:37,445 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 15
  832. 2019-11-13 10:55:37,457 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Host 74677 has switched to another management server, need to update agent map with a forwarding agent attache
  833. 2019-11-13 10:55:37,457 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) create forwarding ClusteredAgentAttache for 74677
  834. 2019-11-13 10:55:37,461 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351041: Forwarding null to 204920354261423
  835. 2019-11-13 10:55:37,464 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 74677-2875266887099351041: Routing from 107973147312911
  836. 2019-11-13 10:55:37,464 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 74677-2875266887099351041: Link is closed
  837. 2019-11-13 10:55:37,464 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 74677-2875266887099351041: MgmtId 107973147312911: Req: Resource [Host:74677] is unreachable: Host 74677: Link is closed
  838. 2019-11-13 10:55:37,464 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Routing to peer
  839. 2019-11-13 10:55:37,468 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649154: Forwarding null to 204920354261423
  840. 2019-11-13 10:55:37,473 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 19-7006475120281649154: Routing from 107973147312911
  841. 2019-11-13 10:55:37,473 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 19-7006475120281649154: Link is closed
  842. 2019-11-13 10:55:37,473 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 19-7006475120281649154: MgmtId 107973147312911: Req: Resource [Host:19] is unreachable: Host 19: Link is closed
  843. 2019-11-13 10:55:37,473 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Routing to peer
  844. 2019-11-13 10:55:37,474 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Cancel request received
  845. 2019-11-13 10:55:37,477 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 74677-2875266887099351041: Cancelling.
  846. 2019-11-13 10:55:37,478 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Cancel request received
  847. 2019-11-13 10:55:37,478 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 19-7006475120281649154: Cancelling.
  848. 2019-11-13 10:55:37,478 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649154: Waiting some more time because this is the current command
  849. 2019-11-13 10:55:37,478 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649154: Waiting some more time because this is the current command
  850. 2019-11-13 10:55:37,478 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  851. 2019-11-13 10:55:37,478 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649154: Timed out on null
  852. 2019-11-13 10:55:37,478 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649154: Cancelling.
  853. 2019-11-13 10:55:37,478 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 7006475120281649154 to Host 19 timed out after 3600
  854. 2019-11-13 10:55:37,478 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 19 statistics.
  855. 2019-11-13 10:55:37,478 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 19
  856. 2019-11-13 10:55:37,482 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351041: Waiting some more time because this is the current command
  857. 2019-11-13 10:55:37,482 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351041: Waiting some more time because this is the current command
  858. 2019-11-13 10:55:37,482 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  859. 2019-11-13 10:55:37,482 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351041: Timed out on null
  860. 2019-11-13 10:55:37,483 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351041: Cancelling.
  861. 2019-11-13 10:55:37,483 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 74677
  862. com.cloud.exception.OperationTimedoutException: Commands 2875266887099351041 to Host 74677 timed out after 3600
  863. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  864. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  865. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  866. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  867. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  868. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  869. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  870. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  871. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  872. at java.lang.reflect.Method.invoke(Method.java:498)
  873. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  874. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  875. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  876. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  877. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  878. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  879. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  880. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  881. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  882. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  883. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  884. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  885. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  886. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  887. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  888. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  889. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  890. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  891. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  892. at java.lang.Thread.run(Thread.java:748)
  893. 2019-11-13 10:55:37,483 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  894. 2019-11-13 10:55:37,483 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  895. 2019-11-13 10:55:37,494 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297410: Forwarding null to 204920354261423
  896. 2019-11-13 10:55:37,497 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 13-1616510791249297410: Routing from 107973147312911
  897. 2019-11-13 10:55:37,497 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 13-1616510791249297410: Link is closed
  898. 2019-11-13 10:55:37,497 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 13-1616510791249297410: MgmtId 107973147312911: Req: Resource [Host:13] is unreachable: Host 13: Link is closed
  899. 2019-11-13 10:55:37,497 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Routing to peer
  900. 2019-11-13 10:55:37,499 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Cancel request received
  901. 2019-11-13 10:55:37,499 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 13-1616510791249297410: Cancelling.
  902. 2019-11-13 10:55:37,499 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297410: Waiting some more time because this is the current command
  903. 2019-11-13 10:55:37,499 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297410: Waiting some more time because this is the current command
  904. 2019-11-13 10:55:37,499 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  905. 2019-11-13 10:55:37,500 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297410: Timed out on null
  906. 2019-11-13 10:55:37,500 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297410: Cancelling.
  907. 2019-11-13 10:55:37,500 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[1|PreSetup] via 13
  908. com.cloud.exception.OperationTimedoutException: Commands 1616510791249297410 to Host 13 timed out after 3600
  909. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  910. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  911. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  912. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  913. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  914. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  915. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  916. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  917. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  918. at java.lang.reflect.Method.invoke(Method.java:498)
  919. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  920. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  921. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  922. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  923. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  924. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  925. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  926. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  927. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  928. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  929. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  930. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  931. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  932. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  933. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  934. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  935. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  936. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  937. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  938. at java.lang.Thread.run(Thread.java:748)
  939. 2019-11-13 10:55:37,501 INFO [c.c.s.StatsCollector] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to reach Pool[1|PreSetup]
  940. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to send command to the pool
  941. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1059)
  942. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  943. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  944. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  945. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  946. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  947. at java.lang.reflect.Method.invoke(Method.java:498)
  948. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  949. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  950. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  951. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  952. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  953. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  954. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  955. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  956. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  957. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  958. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  959. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  960. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  961. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  962. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  963. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  964. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  965. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  966. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  967. at java.lang.Thread.run(Thread.java:748)
  968. 2019-11-13 10:55:37,506 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  969. 2019-11-13 10:55:37,506 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  970. 2019-11-13 10:55:37,511 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082178: Forwarding null to 204920354261423
  971. 2019-11-13 10:55:37,513 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 74673-7260928499228082178: Routing from 107973147312911
  972. 2019-11-13 10:55:37,513 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 74673-7260928499228082178: Link is closed
  973. 2019-11-13 10:55:37,513 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 74673-7260928499228082178: MgmtId 107973147312911: Req: Resource [Host:74673] is unreachable: Host 74673: Link is closed
  974. 2019-11-13 10:55:37,514 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Routing to peer
  975. 2019-11-13 10:55:37,516 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082179: Forwarding null to 204920354261423
  976. 2019-11-13 10:55:37,518 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Cancel request received
  977. 2019-11-13 10:55:37,518 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 74673-7260928499228082178: Cancelling.
  978. 2019-11-13 10:55:37,519 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 74673-7260928499228082179: Routing from 107973147312911
  979. 2019-11-13 10:55:37,519 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 74673-7260928499228082179: Link is closed
  980. 2019-11-13 10:55:37,519 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 74673-7260928499228082179: MgmtId 107973147312911: Req: Resource [Host:74673] is unreachable: Host 74673: Link is closed
  981. 2019-11-13 10:55:37,519 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Routing to peer
  982. 2019-11-13 10:55:37,523 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082178: Waiting some more time because this is the current command
  983. 2019-11-13 10:55:37,523 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082178: Waiting some more time because this is the current command
  984. 2019-11-13 10:55:37,523 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  985. 2019-11-13 10:55:37,523 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Cancel request received
  986. 2019-11-13 10:55:37,523 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082178: Timed out on null
  987. 2019-11-13 10:55:37,523 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 74673-7260928499228082179: Cancelling.
  988. 2019-11-13 10:55:37,523 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082178: Cancelling.
  989. 2019-11-13 10:55:37,524 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 7260928499228082178 to Host 74673 timed out after 3600
  990. 2019-11-13 10:55:37,524 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 74673 statistics.
  991. 2019-11-13 10:55:37,524 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 74673
  992. 2019-11-13 10:55:37,524 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082179: Waiting some more time because this is the current command
  993. 2019-11-13 10:55:37,524 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082179: Waiting some more time because this is the current command
  994. 2019-11-13 10:55:37,524 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  995. 2019-11-13 10:55:37,524 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082179: Timed out on null
  996. 2019-11-13 10:55:37,524 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082179: Cancelling.
  997. 2019-11-13 10:55:37,525 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 74673
  998. com.cloud.exception.OperationTimedoutException: Commands 7260928499228082179 to Host 74673 timed out after 3600
  999. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1000. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1001. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1002. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1003. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1004. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1005. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1006. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1007. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1008. at java.lang.reflect.Method.invoke(Method.java:498)
  1009. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1010. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1011. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1012. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1013. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1014. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1015. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1016. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1017. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1018. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1019. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1020. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1021. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1022. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1023. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1024. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1025. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1026. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1027. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1028. at java.lang.Thread.run(Thread.java:748)
  1029. 2019-11-13 10:55:37,525 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1030. 2019-11-13 10:55:37,525 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1031. 2019-11-13 10:55:37,537 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351042: Forwarding null to 204920354261423
  1032. 2019-11-13 10:55:37,545 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 74677-2875266887099351042: Routing from 107973147312911
  1033. 2019-11-13 10:55:37,545 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 74677-2875266887099351042: Link is closed
  1034. 2019-11-13 10:55:37,545 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 74677-2875266887099351042: MgmtId 107973147312911: Req: Resource [Host:74677] is unreachable: Host 74677: Link is closed
  1035. 2019-11-13 10:55:37,545 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Routing to peer
  1036. 2019-11-13 10:55:37,547 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Cancel request received
  1037. 2019-11-13 10:55:37,547 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 74677-2875266887099351042: Cancelling.
  1038. 2019-11-13 10:55:37,548 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351042: Waiting some more time because this is the current command
  1039. 2019-11-13 10:55:37,548 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351042: Waiting some more time because this is the current command
  1040. 2019-11-13 10:55:37,548 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1041. 2019-11-13 10:55:37,549 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351042: Timed out on null
  1042. 2019-11-13 10:55:37,549 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351042: Cancelling.
  1043. 2019-11-13 10:55:37,549 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 74677
  1044. com.cloud.exception.OperationTimedoutException: Commands 2875266887099351042 to Host 74677 timed out after 3600
  1045. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1046. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1047. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1048. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1049. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1050. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1051. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1052. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1053. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1054. at java.lang.reflect.Method.invoke(Method.java:498)
  1055. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1056. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1057. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1058. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1059. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1060. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1061. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1062. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1063. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1064. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1065. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1066. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1067. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1068. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1069. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1070. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1071. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1072. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1073. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1074. at java.lang.Thread.run(Thread.java:748)
  1075. 2019-11-13 10:55:37,549 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1076. 2019-11-13 10:55:37,549 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1077. 2019-11-13 10:55:37,557 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351043: Forwarding null to 204920354261423
  1078. 2019-11-13 10:55:37,560 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 74677-2875266887099351043: Routing from 107973147312911
  1079. 2019-11-13 10:55:37,560 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 74677-2875266887099351043: Link is closed
  1080. 2019-11-13 10:55:37,560 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 74677-2875266887099351043: MgmtId 107973147312911: Req: Resource [Host:74677] is unreachable: Host 74677: Link is closed
  1081. 2019-11-13 10:55:37,560 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Routing to peer
  1082. 2019-11-13 10:55:37,562 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Cancel request received
  1083. 2019-11-13 10:55:37,562 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 74677-2875266887099351043: Cancelling.
  1084. 2019-11-13 10:55:37,562 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351043: Waiting some more time because this is the current command
  1085. 2019-11-13 10:55:37,562 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351043: Waiting some more time because this is the current command
  1086. 2019-11-13 10:55:37,562 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1087. 2019-11-13 10:55:37,562 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351043: Timed out on null
  1088. 2019-11-13 10:55:37,562 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351043: Cancelling.
  1089. 2019-11-13 10:55:37,562 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 2875266887099351043 to Host 74677 timed out after 3600
  1090. 2019-11-13 10:55:37,563 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain host 74677 statistics.
  1091. 2019-11-13 10:55:37,563 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Received invalid host stats for host: 74677
  1092. 2019-11-13 10:55:37,568 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425859: Forwarding null to 204920354261423
  1093. 2019-11-13 10:55:37,569 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 10-6636054050930425859: Routing from 107973147312911
  1094. 2019-11-13 10:55:37,570 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 10-6636054050930425859: Link is closed
  1095. 2019-11-13 10:55:37,570 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 10-6636054050930425859: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  1096. 2019-11-13 10:55:37,570 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  1097. 2019-11-13 10:55:37,571 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  1098. 2019-11-13 10:55:37,571 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 10-6636054050930425859: Cancelling.
  1099. 2019-11-13 10:55:37,572 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425859: Waiting some more time because this is the current command
  1100. 2019-11-13 10:55:37,572 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425859: Waiting some more time because this is the current command
  1101. 2019-11-13 10:55:37,572 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1102. 2019-11-13 10:55:37,572 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425859: Timed out on null
  1103. 2019-11-13 10:55:37,572 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425859: Cancelling.
  1104. 2019-11-13 10:55:37,572 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 10
  1105. com.cloud.exception.OperationTimedoutException: Commands 6636054050930425859 to Host 10 timed out after 3600
  1106. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1107. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1108. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1109. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1110. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1111. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1112. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1113. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1114. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1115. at java.lang.reflect.Method.invoke(Method.java:498)
  1116. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1117. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1118. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1119. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1120. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1121. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1122. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1123. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1124. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1125. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1126. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1127. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1128. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1129. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1130. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1131. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1132. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1133. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1134. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1135. at java.lang.Thread.run(Thread.java:748)
  1136. 2019-11-13 10:55:37,573 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1137. 2019-11-13 10:55:37,573 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1138. 2019-11-13 10:55:37,580 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182405: Forwarding null to 204920354261423
  1139. 2019-11-13 10:55:37,582 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 2-2223652316014182405: Routing from 107973147312911
  1140. 2019-11-13 10:55:37,582 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 2-2223652316014182405: Link is closed
  1141. 2019-11-13 10:55:37,583 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 2-2223652316014182405: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  1142. 2019-11-13 10:55:37,583 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  1143. 2019-11-13 10:55:37,584 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  1144. 2019-11-13 10:55:37,584 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 2-2223652316014182405: Cancelling.
  1145. 2019-11-13 10:55:37,584 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182405: Waiting some more time because this is the current command
  1146. 2019-11-13 10:55:37,584 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182405: Waiting some more time because this is the current command
  1147. 2019-11-13 10:55:37,585 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1148. 2019-11-13 10:55:37,585 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182405: Timed out on null
  1149. 2019-11-13 10:55:37,585 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 2-2223652316014182405: Cancelling.
  1150. 2019-11-13 10:55:37,585 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 2223652316014182405 to Host 2 timed out after 3600
  1151. 2019-11-13 10:55:37,585 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server03
  1152. 2019-11-13 10:55:37,592 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700547: Forwarding null to 204920354261423
  1153. 2019-11-13 10:55:37,595 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042629: Forwarding null to 204920354261423
  1154. 2019-11-13 10:55:37,595 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 9-1463388403918700547: Routing from 107973147312911
  1155. 2019-11-13 10:55:37,595 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 9-1463388403918700547: Link is closed
  1156. 2019-11-13 10:55:37,595 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 9-1463388403918700547: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  1157. 2019-11-13 10:55:37,596 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  1158. 2019-11-13 10:55:37,602 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 3-7917609619894042629: Routing from 107973147312911
  1159. 2019-11-13 10:55:37,602 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 3-7917609619894042629: Link is closed
  1160. 2019-11-13 10:55:37,602 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 3-7917609619894042629: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  1161. 2019-11-13 10:55:37,602 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  1162. 2019-11-13 10:55:37,603 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  1163. 2019-11-13 10:55:37,603 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 9-1463388403918700547: Cancelling.
  1164. 2019-11-13 10:55:37,604 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  1165. 2019-11-13 10:55:37,604 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 3-7917609619894042629: Cancelling.
  1166. 2019-11-13 10:55:37,604 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042629: Waiting some more time because this is the current command
  1167. 2019-11-13 10:55:37,605 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042629: Waiting some more time because this is the current command
  1168. 2019-11-13 10:55:37,605 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1169. 2019-11-13 10:55:37,605 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042629: Timed out on null
  1170. 2019-11-13 10:55:37,605 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 3-7917609619894042629: Cancelling.
  1171. 2019-11-13 10:55:37,605 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 7917609619894042629 to Host 3 timed out after 3600
  1172. 2019-11-13 10:55:37,605 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server06
  1173. 2019-11-13 10:55:37,608 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700547: Waiting some more time because this is the current command
  1174. 2019-11-13 10:55:37,609 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700547: Waiting some more time because this is the current command
  1175. 2019-11-13 10:55:37,609 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1176. 2019-11-13 10:55:37,609 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700547: Timed out on null
  1177. 2019-11-13 10:55:37,609 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700547: Cancelling.
  1178. 2019-11-13 10:55:37,609 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 9
  1179. com.cloud.exception.OperationTimedoutException: Commands 1463388403918700547 to Host 9 timed out after 3600
  1180. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1181. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1182. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1183. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1184. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1185. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1186. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1187. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1188. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1189. at java.lang.reflect.Method.invoke(Method.java:498)
  1190. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1191. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1192. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1193. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1194. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1195. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1196. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1197. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1198. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1199. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1200. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1201. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1202. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1203. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1204. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1205. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1206. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1207. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1208. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1209. at java.lang.Thread.run(Thread.java:748)
  1210. 2019-11-13 10:55:37,609 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1211. 2019-11-13 10:55:37,609 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1212. 2019-11-13 10:55:37,617 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700548: Forwarding null to 204920354261423
  1213. 2019-11-13 10:55:37,620 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042630: Forwarding null to 204920354261423
  1214. 2019-11-13 10:55:37,620 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 9-1463388403918700548: Routing from 107973147312911
  1215. 2019-11-13 10:55:37,620 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 9-1463388403918700548: Link is closed
  1216. 2019-11-13 10:55:37,620 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 9-1463388403918700548: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  1217. 2019-11-13 10:55:37,620 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  1218. 2019-11-13 10:55:37,624 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 3-7917609619894042630: Routing from 107973147312911
  1219. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 3-7917609619894042630: Link is closed
  1220. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 3-7917609619894042630: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  1221. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  1222. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 9-1463388403918700548: Cancelling.
  1223. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700548: Waiting some more time because this is the current command
  1224. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700548: Waiting some more time because this is the current command
  1225. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  1226. 2019-11-13 10:55:37,625 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1227. 2019-11-13 10:55:37,625 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700548: Timed out on null
  1228. 2019-11-13 10:55:37,625 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 9-1463388403918700548: Cancelling.
  1229. 2019-11-13 10:55:37,625 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 1463388403918700548 to Host 9 timed out after 3600
  1230. 2019-11-13 10:55:37,625 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server10
  1231. 2019-11-13 10:55:37,626 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  1232. 2019-11-13 10:55:37,626 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 3-7917609619894042630: Cancelling.
  1233. 2019-11-13 10:55:37,626 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042630: Waiting some more time because this is the current command
  1234. 2019-11-13 10:55:37,626 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042630: Waiting some more time because this is the current command
  1235. 2019-11-13 10:55:37,627 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1236. 2019-11-13 10:55:37,627 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042630: Timed out on null
  1237. 2019-11-13 10:55:37,627 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042630: Cancelling.
  1238. 2019-11-13 10:55:37,627 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 3
  1239. com.cloud.exception.OperationTimedoutException: Commands 7917609619894042630 to Host 3 timed out after 3600
  1240. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1241. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1242. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1243. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1244. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1245. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1246. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1247. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1248. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1249. at java.lang.reflect.Method.invoke(Method.java:498)
  1250. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1251. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1252. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1253. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1254. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1255. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1256. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1257. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1258. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1259. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1260. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1261. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1262. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1263. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1264. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1265. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1266. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1267. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1268. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1269. at java.lang.Thread.run(Thread.java:748)
  1270. 2019-11-13 10:55:37,627 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1271. 2019-11-13 10:55:37,627 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1272. 2019-11-13 10:55:37,637 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661123: Forwarding null to 204920354261423
  1273. 2019-11-13 10:55:37,639 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 15-5775866522102661123: Routing from 107973147312911
  1274. 2019-11-13 10:55:37,639 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 15-5775866522102661123: Link is closed
  1275. 2019-11-13 10:55:37,639 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 15-5775866522102661123: MgmtId 107973147312911: Req: Resource [Host:15] is unreachable: Host 15: Link is closed
  1276. 2019-11-13 10:55:37,639 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Routing to peer
  1277. 2019-11-13 10:55:37,641 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Cancel request received
  1278. 2019-11-13 10:55:37,641 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 15-5775866522102661123: Cancelling.
  1279. 2019-11-13 10:55:37,641 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661123: Waiting some more time because this is the current command
  1280. 2019-11-13 10:55:37,641 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661123: Waiting some more time because this is the current command
  1281. 2019-11-13 10:55:37,641 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1282. 2019-11-13 10:55:37,642 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661123: Timed out on null
  1283. 2019-11-13 10:55:37,642 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661123: Cancelling.
  1284. 2019-11-13 10:55:37,642 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 15
  1285. com.cloud.exception.OperationTimedoutException: Commands 5775866522102661123 to Host 15 timed out after 3600
  1286. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1287. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1288. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1289. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1290. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1291. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1292. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1293. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1294. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1295. at java.lang.reflect.Method.invoke(Method.java:498)
  1296. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1297. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1298. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1299. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1300. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1301. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1302. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1303. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1304. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1305. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1306. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1307. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1308. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1309. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1310. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1311. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1312. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1313. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1314. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1315. at java.lang.Thread.run(Thread.java:748)
  1316. 2019-11-13 10:55:37,642 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1317. 2019-11-13 10:55:37,642 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1318. 2019-11-13 10:55:37,649 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425860: Forwarding null to 204920354261423
  1319. 2019-11-13 10:55:37,657 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297411: Forwarding null to 204920354261423
  1320. 2019-11-13 10:55:37,659 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 10-6636054050930425860: Routing from 107973147312911
  1321. 2019-11-13 10:55:37,659 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 10-6636054050930425860: Link is closed
  1322. 2019-11-13 10:55:37,659 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 10-6636054050930425860: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  1323. 2019-11-13 10:55:37,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  1324. 2019-11-13 10:55:37,660 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 13-1616510791249297411: Routing from 107973147312911
  1325. 2019-11-13 10:55:37,660 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 13-1616510791249297411: Link is closed
  1326. 2019-11-13 10:55:37,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 13-1616510791249297411: MgmtId 107973147312911: Req: Resource [Host:13] is unreachable: Host 13: Link is closed
  1327. 2019-11-13 10:55:37,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Routing to peer
  1328. 2019-11-13 10:55:37,662 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  1329. 2019-11-13 10:55:37,662 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 10-6636054050930425860: Cancelling.
  1330. 2019-11-13 10:55:37,662 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425860: Waiting some more time because this is the current command
  1331. 2019-11-13 10:55:37,662 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425860: Waiting some more time because this is the current command
  1332. 2019-11-13 10:55:37,662 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1333. 2019-11-13 10:55:37,662 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425860: Timed out on null
  1334. 2019-11-13 10:55:37,662 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 10-6636054050930425860: Cancelling.
  1335. 2019-11-13 10:55:37,662 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 6636054050930425860 to Host 10 timed out after 3600
  1336. 2019-11-13 10:55:37,663 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server09
  1337. 2019-11-13 10:55:37,662 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Cancel request received
  1338. 2019-11-13 10:55:37,666 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 13-1616510791249297411: Cancelling.
  1339. 2019-11-13 10:55:37,667 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297411: Waiting some more time because this is the current command
  1340. 2019-11-13 10:55:37,667 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297411: Waiting some more time because this is the current command
  1341. 2019-11-13 10:55:37,667 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1342. 2019-11-13 10:55:37,667 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297411: Timed out on null
  1343. 2019-11-13 10:55:37,667 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297411: Cancelling.
  1344. 2019-11-13 10:55:37,667 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 13
  1345. com.cloud.exception.OperationTimedoutException: Commands 1616510791249297411 to Host 13 timed out after 3600
  1346. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1347. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1348. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1349. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1350. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1351. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1352. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1353. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1354. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1355. at java.lang.reflect.Method.invoke(Method.java:498)
  1356. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1357. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1358. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1359. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1360. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1361. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1362. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1363. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1364. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1365. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1366. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1367. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1368. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1369. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1370. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1371. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1372. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1373. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1374. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1375. at java.lang.Thread.run(Thread.java:748)
  1376. 2019-11-13 10:55:37,667 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1377. 2019-11-13 10:55:37,667 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1378. 2019-11-13 10:55:37,676 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297412: Forwarding null to 204920354261423
  1379. 2019-11-13 10:55:37,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 13-1616510791249297412: Routing from 107973147312911
  1380. 2019-11-13 10:55:37,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 13-1616510791249297412: Link is closed
  1381. 2019-11-13 10:55:37,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 13-1616510791249297412: MgmtId 107973147312911: Req: Resource [Host:13] is unreachable: Host 13: Link is closed
  1382. 2019-11-13 10:55:37,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Routing to peer
  1383. 2019-11-13 10:55:37,682 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Cancel request received
  1384. 2019-11-13 10:55:37,683 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 13-1616510791249297412: Cancelling.
  1385. 2019-11-13 10:55:37,683 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297412: Waiting some more time because this is the current command
  1386. 2019-11-13 10:55:37,683 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297412: Waiting some more time because this is the current command
  1387. 2019-11-13 10:55:37,683 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1388. 2019-11-13 10:55:37,683 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297412: Timed out on null
  1389. 2019-11-13 10:55:37,683 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 13-1616510791249297412: Cancelling.
  1390. 2019-11-13 10:55:37,683 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 1616510791249297412 to Host 13 timed out after 3600
  1391. 2019-11-13 10:55:37,683 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server08
  1392. 2019-11-13 10:55:37,700 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661124: Forwarding null to 204920354261423
  1393. 2019-11-13 10:55:37,702 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 15-5775866522102661124: Routing from 107973147312911
  1394. 2019-11-13 10:55:37,702 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 15-5775866522102661124: Link is closed
  1395. 2019-11-13 10:55:37,702 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 15-5775866522102661124: MgmtId 107973147312911: Req: Resource [Host:15] is unreachable: Host 15: Link is closed
  1396. 2019-11-13 10:55:37,702 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Routing to peer
  1397. 2019-11-13 10:55:37,706 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Cancel request received
  1398. 2019-11-13 10:55:37,706 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 15-5775866522102661124: Cancelling.
  1399. 2019-11-13 10:55:37,707 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661124: Waiting some more time because this is the current command
  1400. 2019-11-13 10:55:37,707 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661124: Waiting some more time because this is the current command
  1401. 2019-11-13 10:55:37,707 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1402. 2019-11-13 10:55:37,707 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661124: Timed out on null
  1403. 2019-11-13 10:55:37,707 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 15-5775866522102661124: Cancelling.
  1404. 2019-11-13 10:55:37,707 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 5775866522102661124 to Host 15 timed out after 3600
  1405. 2019-11-13 10:55:37,707 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server04
  1406. 2019-11-13 10:55:37,711 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182406: Forwarding null to 204920354261423
  1407. 2019-11-13 10:55:37,714 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 2-2223652316014182406: Routing from 107973147312911
  1408. 2019-11-13 10:55:37,714 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 2-2223652316014182406: Link is closed
  1409. 2019-11-13 10:55:37,714 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 2-2223652316014182406: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  1410. 2019-11-13 10:55:37,714 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  1411. 2019-11-13 10:55:37,716 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  1412. 2019-11-13 10:55:37,716 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 2-2223652316014182406: Cancelling.
  1413. 2019-11-13 10:55:37,723 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649155: Forwarding null to 204920354261423
  1414. 2019-11-13 10:55:37,725 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 19-7006475120281649155: Routing from 107973147312911
  1415. 2019-11-13 10:55:37,725 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 19-7006475120281649155: Link is closed
  1416. 2019-11-13 10:55:37,725 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 19-7006475120281649155: MgmtId 107973147312911: Req: Resource [Host:19] is unreachable: Host 19: Link is closed
  1417. 2019-11-13 10:55:37,726 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Routing to peer
  1418. 2019-11-13 10:55:37,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Cancel request received
  1419. 2019-11-13 10:55:37,727 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 19-7006475120281649155: Cancelling.
  1420. 2019-11-13 10:55:37,727 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649155: Waiting some more time because this is the current command
  1421. 2019-11-13 10:55:37,727 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649155: Waiting some more time because this is the current command
  1422. 2019-11-13 10:55:37,727 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1423. 2019-11-13 10:55:37,727 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649155: Timed out on null
  1424. 2019-11-13 10:55:37,727 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 19-7006475120281649155: Cancelling.
  1425. 2019-11-13 10:55:37,728 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 7006475120281649155 to Host 19 timed out after 3600
  1426. 2019-11-13 10:55:37,728 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server01
  1427. 2019-11-13 10:55:37,731 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182406: Waiting some more time because this is the current command
  1428. 2019-11-13 10:55:37,731 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182406: Waiting some more time because this is the current command
  1429. 2019-11-13 10:55:37,731 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1430. 2019-11-13 10:55:37,731 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182406: Timed out on null
  1431. 2019-11-13 10:55:37,732 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182406: Cancelling.
  1432. 2019-11-13 10:55:37,732 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 2
  1433. com.cloud.exception.OperationTimedoutException: Commands 2223652316014182406 to Host 2 timed out after 3600
  1434. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1435. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1436. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1437. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1438. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1439. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1440. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1441. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1442. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1443. at java.lang.reflect.Method.invoke(Method.java:498)
  1444. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1445. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1446. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1447. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1448. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1449. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1450. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1451. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1452. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1453. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1454. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1455. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1456. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1457. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1458. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1459. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1460. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1461. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1462. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1463. at java.lang.Thread.run(Thread.java:748)
  1464. 2019-11-13 10:55:37,732 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1465. 2019-11-13 10:55:37,732 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1466. 2019-11-13 10:55:37,746 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082180: Forwarding null to 204920354261423
  1467. 2019-11-13 10:55:37,748 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 74673-7260928499228082180: Routing from 107973147312911
  1468. 2019-11-13 10:55:37,748 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 74673-7260928499228082180: Link is closed
  1469. 2019-11-13 10:55:37,748 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 74673-7260928499228082180: MgmtId 107973147312911: Req: Resource [Host:74673] is unreachable: Host 74673: Link is closed
  1470. 2019-11-13 10:55:37,748 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Routing to peer
  1471. 2019-11-13 10:55:37,749 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649156: Forwarding null to 204920354261423
  1472. 2019-11-13 10:55:37,751 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Cancel request received
  1473. 2019-11-13 10:55:37,751 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 74673-7260928499228082180: Cancelling.
  1474. 2019-11-13 10:55:37,752 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 19-7006475120281649156: Routing from 107973147312911
  1475. 2019-11-13 10:55:37,752 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 19-7006475120281649156: Link is closed
  1476. 2019-11-13 10:55:37,752 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 19-7006475120281649156: MgmtId 107973147312911: Req: Resource [Host:19] is unreachable: Host 19: Link is closed
  1477. 2019-11-13 10:55:37,752 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Routing to peer
  1478. 2019-11-13 10:55:37,755 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082180: Waiting some more time because this is the current command
  1479. 2019-11-13 10:55:37,756 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082180: Waiting some more time because this is the current command
  1480. 2019-11-13 10:55:37,756 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1481. 2019-11-13 10:55:37,756 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082180: Timed out on null
  1482. 2019-11-13 10:55:37,756 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74673-7260928499228082180: Cancelling.
  1483. 2019-11-13 10:55:37,756 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 7260928499228082180 to Host 74673 timed out after 3600
  1484. 2019-11-13 10:55:37,756 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server07
  1485. 2019-11-13 10:55:37,764 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Cancel request received
  1486. 2019-11-13 10:55:37,764 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 19-7006475120281649156: Cancelling.
  1487. 2019-11-13 10:55:37,766 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649156: Waiting some more time because this is the current command
  1488. 2019-11-13 10:55:37,766 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649156: Waiting some more time because this is the current command
  1489. 2019-11-13 10:55:37,766 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1490. 2019-11-13 10:55:37,766 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649156: Timed out on null
  1491. 2019-11-13 10:55:37,766 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649156: Cancelling.
  1492. 2019-11-13 10:55:37,766 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[3|PreSetup] via 19
  1493. com.cloud.exception.OperationTimedoutException: Commands 7006475120281649156 to Host 19 timed out after 3600
  1494. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1495. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1496. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1497. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1498. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1499. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1500. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1501. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1502. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1503. at java.lang.reflect.Method.invoke(Method.java:498)
  1504. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1505. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1506. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1507. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1508. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1509. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1510. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1511. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1512. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1513. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1514. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1515. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1516. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1517. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1518. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1519. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1520. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1521. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1522. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1523. at java.lang.Thread.run(Thread.java:748)
  1524. 2019-11-13 10:55:37,767 INFO [c.c.s.StatsCollector] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to reach Pool[3|PreSetup]
  1525. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to send command to the pool
  1526. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1059)
  1527. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1528. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1529. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1530. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1531. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1532. at java.lang.reflect.Method.invoke(Method.java:498)
  1533. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1534. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1535. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1536. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1537. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1538. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1539. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1540. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1541. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1542. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1543. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1544. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1545. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1546. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1547. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1548. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1549. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1550. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1551. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1552. at java.lang.Thread.run(Thread.java:748)
  1553. 2019-11-13 10:55:37,792 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1554. 2019-11-13 10:55:37,792 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1555. 2019-11-13 10:55:37,795 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351044: Forwarding null to 204920354261423
  1556. 2019-11-13 10:55:37,804 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 74677-2875266887099351044: Routing from 107973147312911
  1557. 2019-11-13 10:55:37,804 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 74677-2875266887099351044: Link is closed
  1558. 2019-11-13 10:55:37,804 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 74677-2875266887099351044: MgmtId 107973147312911: Req: Resource [Host:74677] is unreachable: Host 74677: Link is closed
  1559. 2019-11-13 10:55:37,804 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Routing to peer
  1560. 2019-11-13 10:55:37,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Cancel request received
  1561. 2019-11-13 10:55:37,806 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 74677-2875266887099351044: Cancelling.
  1562. 2019-11-13 10:55:37,806 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351044: Waiting some more time because this is the current command
  1563. 2019-11-13 10:55:37,806 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351044: Waiting some more time because this is the current command
  1564. 2019-11-13 10:55:37,807 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1565. 2019-11-13 10:55:37,807 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351044: Timed out on null
  1566. 2019-11-13 10:55:37,807 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Seq 74677-2875266887099351044: Cancelling.
  1567. 2019-11-13 10:55:37,807 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Operation timed out: Commands 2875266887099351044 to Host 74677 timed out after 3600
  1568. 2019-11-13 10:55:37,807 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-02ad4bbf) (logid:2b6e450b) Unable to obtain GPU stats for host lbs-server05
  1569. 2019-11-13 10:55:37,809 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649157: Forwarding null to 204920354261423
  1570. 2019-11-13 10:55:37,817 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 19-7006475120281649157: Routing from 107973147312911
  1571. 2019-11-13 10:55:37,817 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 19-7006475120281649157: Link is closed
  1572. 2019-11-13 10:55:37,817 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 19-7006475120281649157: MgmtId 107973147312911: Req: Resource [Host:19] is unreachable: Host 19: Link is closed
  1573. 2019-11-13 10:55:37,818 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Routing to peer
  1574. 2019-11-13 10:55:37,820 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Cancel request received
  1575. 2019-11-13 10:55:37,820 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 19-7006475120281649157: Cancelling.
  1576. 2019-11-13 10:55:37,820 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649157: Waiting some more time because this is the current command
  1577. 2019-11-13 10:55:37,820 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649157: Waiting some more time because this is the current command
  1578. 2019-11-13 10:55:37,820 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1579. 2019-11-13 10:55:37,820 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649157: Timed out on null
  1580. 2019-11-13 10:55:37,820 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649157: Cancelling.
  1581. 2019-11-13 10:55:37,820 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 19
  1582. com.cloud.exception.OperationTimedoutException: Commands 7006475120281649157 to Host 19 timed out after 3600
  1583. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1584. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1585. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1586. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1587. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1588. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1589. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1590. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1591. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1592. at java.lang.reflect.Method.invoke(Method.java:498)
  1593. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1594. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1595. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1596. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1597. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1598. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1599. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1600. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1601. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1602. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1603. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1604. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1605. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1606. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1607. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1608. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1609. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1610. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1611. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1612. at java.lang.Thread.run(Thread.java:748)
  1613. 2019-11-13 10:55:37,821 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1614. 2019-11-13 10:55:37,821 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1615. 2019-11-13 10:55:37,836 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661125: Forwarding null to 204920354261423
  1616. 2019-11-13 10:55:37,838 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 15-5775866522102661125: Routing from 107973147312911
  1617. 2019-11-13 10:55:37,838 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 15-5775866522102661125: Link is closed
  1618. 2019-11-13 10:55:37,838 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 15-5775866522102661125: MgmtId 107973147312911: Req: Resource [Host:15] is unreachable: Host 15: Link is closed
  1619. 2019-11-13 10:55:37,838 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Routing to peer
  1620. 2019-11-13 10:55:37,842 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Cancel request received
  1621. 2019-11-13 10:55:37,843 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 15-5775866522102661125: Cancelling.
  1622. 2019-11-13 10:55:37,843 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661125: Waiting some more time because this is the current command
  1623. 2019-11-13 10:55:37,843 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661125: Waiting some more time because this is the current command
  1624. 2019-11-13 10:55:37,843 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1625. 2019-11-13 10:55:37,843 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661125: Timed out on null
  1626. 2019-11-13 10:55:37,843 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661125: Cancelling.
  1627. 2019-11-13 10:55:37,843 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 15
  1628. com.cloud.exception.OperationTimedoutException: Commands 5775866522102661125 to Host 15 timed out after 3600
  1629. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1630. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1631. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1632. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1633. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1634. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1635. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1636. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1637. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1638. at java.lang.reflect.Method.invoke(Method.java:498)
  1639. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1640. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1641. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1642. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1643. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1644. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1645. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1646. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1647. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1648. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1649. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1650. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1651. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1652. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1653. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1654. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1655. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1656. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1657. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1658. at java.lang.Thread.run(Thread.java:748)
  1659. 2019-11-13 10:55:37,844 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1660. 2019-11-13 10:55:37,844 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1661. 2019-11-13 10:55:37,852 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351045: Forwarding null to 204920354261423
  1662. 2019-11-13 10:55:37,854 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 74677-2875266887099351045: Routing from 107973147312911
  1663. 2019-11-13 10:55:37,854 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 74677-2875266887099351045: Link is closed
  1664. 2019-11-13 10:55:37,854 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 74677-2875266887099351045: MgmtId 107973147312911: Req: Resource [Host:74677] is unreachable: Host 74677: Link is closed
  1665. 2019-11-13 10:55:37,854 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Routing to peer
  1666. 2019-11-13 10:55:37,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Cancel request received
  1667. 2019-11-13 10:55:37,856 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 74677-2875266887099351045: Cancelling.
  1668. 2019-11-13 10:55:37,856 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351045: Waiting some more time because this is the current command
  1669. 2019-11-13 10:55:37,856 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351045: Waiting some more time because this is the current command
  1670. 2019-11-13 10:55:37,856 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1671. 2019-11-13 10:55:37,856 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351045: Timed out on null
  1672. 2019-11-13 10:55:37,856 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351045: Cancelling.
  1673. 2019-11-13 10:55:37,856 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 74677
  1674. com.cloud.exception.OperationTimedoutException: Commands 2875266887099351045 to Host 74677 timed out after 3600
  1675. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1676. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1677. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1678. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1679. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1680. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1681. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1682. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1683. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1684. at java.lang.reflect.Method.invoke(Method.java:498)
  1685. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1686. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1687. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1688. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1689. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1690. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1691. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1692. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1693. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1694. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1695. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1696. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1697. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1698. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1699. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1700. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1701. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1702. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1703. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1704. at java.lang.Thread.run(Thread.java:748)
  1705. 2019-11-13 10:55:37,857 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1706. 2019-11-13 10:55:37,857 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1707. 2019-11-13 10:55:37,865 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182407: Forwarding null to 204920354261423
  1708. 2019-11-13 10:55:37,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 2-2223652316014182407: Routing from 107973147312911
  1709. 2019-11-13 10:55:37,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 2-2223652316014182407: Link is closed
  1710. 2019-11-13 10:55:37,867 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 2-2223652316014182407: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  1711. 2019-11-13 10:55:37,867 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  1712. 2019-11-13 10:55:37,868 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  1713. 2019-11-13 10:55:37,868 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 2-2223652316014182407: Cancelling.
  1714. 2019-11-13 10:55:37,869 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182407: Waiting some more time because this is the current command
  1715. 2019-11-13 10:55:37,869 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182407: Waiting some more time because this is the current command
  1716. 2019-11-13 10:55:37,869 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1717. 2019-11-13 10:55:37,869 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182407: Timed out on null
  1718. 2019-11-13 10:55:37,869 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182407: Cancelling.
  1719. 2019-11-13 10:55:37,869 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 2
  1720. com.cloud.exception.OperationTimedoutException: Commands 2223652316014182407 to Host 2 timed out after 3600
  1721. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1722. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1723. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1724. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1725. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1726. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1727. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1728. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1729. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1730. at java.lang.reflect.Method.invoke(Method.java:498)
  1731. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1732. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1733. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1734. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1735. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1736. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1737. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1738. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1739. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1740. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1741. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1742. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1743. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1744. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1745. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1746. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1747. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1748. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1749. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1750. at java.lang.Thread.run(Thread.java:748)
  1751. 2019-11-13 10:55:37,869 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1752. 2019-11-13 10:55:37,869 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1753. 2019-11-13 10:55:37,873 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 9-1463388403918700549: Forwarding null to 204920354261423
  1754. 2019-11-13 10:55:37,875 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 9-1463388403918700549: Routing from 107973147312911
  1755. 2019-11-13 10:55:37,875 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 9-1463388403918700549: Link is closed
  1756. 2019-11-13 10:55:37,875 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 9-1463388403918700549: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  1757. 2019-11-13 10:55:37,875 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  1758. 2019-11-13 10:55:37,876 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  1759. 2019-11-13 10:55:37,876 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 9-1463388403918700549: Cancelling.
  1760. 2019-11-13 10:55:37,877 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 9-1463388403918700549: Waiting some more time because this is the current command
  1761. 2019-11-13 10:55:37,877 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 9-1463388403918700549: Waiting some more time because this is the current command
  1762. 2019-11-13 10:55:37,877 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1763. 2019-11-13 10:55:37,877 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 9-1463388403918700549: Timed out on null
  1764. 2019-11-13 10:55:37,877 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 9-1463388403918700549: Cancelling.
  1765. 2019-11-13 10:55:37,877 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Operation timed out: Commands 1463388403918700549 to Host 9 timed out after 3600
  1766. 2019-11-13 10:55:37,877 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Unable to obtain VM statistics.
  1767. 2019-11-13 10:55:37,880 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082181: Forwarding null to 204920354261423
  1768. 2019-11-13 10:55:37,886 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 74673-7260928499228082181: Routing from 107973147312911
  1769. 2019-11-13 10:55:37,886 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 74673-7260928499228082181: Link is closed
  1770. 2019-11-13 10:55:37,886 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 74673-7260928499228082181: MgmtId 107973147312911: Req: Resource [Host:74673] is unreachable: Host 74673: Link is closed
  1771. 2019-11-13 10:55:37,886 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Routing to peer
  1772. 2019-11-13 10:55:37,888 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Cancel request received
  1773. 2019-11-13 10:55:37,888 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 74673-7260928499228082181: Cancelling.
  1774. 2019-11-13 10:55:37,889 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082181: Waiting some more time because this is the current command
  1775. 2019-11-13 10:55:37,889 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082181: Waiting some more time because this is the current command
  1776. 2019-11-13 10:55:37,889 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1777. 2019-11-13 10:55:37,889 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082181: Timed out on null
  1778. 2019-11-13 10:55:37,889 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082181: Cancelling.
  1779. 2019-11-13 10:55:37,889 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 74673
  1780. com.cloud.exception.OperationTimedoutException: Commands 7260928499228082181 to Host 74673 timed out after 3600
  1781. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1782. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1783. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1784. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1785. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1786. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1787. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1788. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1789. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1790. at java.lang.reflect.Method.invoke(Method.java:498)
  1791. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1792. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1793. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1794. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1795. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1796. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1797. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1798. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1799. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1800. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1801. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1802. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1803. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1804. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1805. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1806. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1807. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1808. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1809. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1810. at java.lang.Thread.run(Thread.java:748)
  1811. 2019-11-13 10:55:37,889 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1812. 2019-11-13 10:55:37,889 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1813. 2019-11-13 10:55:37,899 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297413: Forwarding null to 204920354261423
  1814. 2019-11-13 10:55:37,904 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 13-1616510791249297413: Routing from 107973147312911
  1815. 2019-11-13 10:55:37,904 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 13-1616510791249297413: Link is closed
  1816. 2019-11-13 10:55:37,904 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 13-1616510791249297413: MgmtId 107973147312911: Req: Resource [Host:13] is unreachable: Host 13: Link is closed
  1817. 2019-11-13 10:55:37,904 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Routing to peer
  1818. 2019-11-13 10:55:37,906 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Cancel request received
  1819. 2019-11-13 10:55:37,906 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 13-1616510791249297413: Cancelling.
  1820. 2019-11-13 10:55:37,907 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297413: Waiting some more time because this is the current command
  1821. 2019-11-13 10:55:37,907 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297413: Waiting some more time because this is the current command
  1822. 2019-11-13 10:55:37,907 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1823. 2019-11-13 10:55:37,907 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297413: Timed out on null
  1824. 2019-11-13 10:55:37,907 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297413: Cancelling.
  1825. 2019-11-13 10:55:37,907 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 13
  1826. com.cloud.exception.OperationTimedoutException: Commands 1616510791249297413 to Host 13 timed out after 3600
  1827. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1828. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1829. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1830. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1831. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1832. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1833. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1834. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1835. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1836. at java.lang.reflect.Method.invoke(Method.java:498)
  1837. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1838. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1839. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1840. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1841. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1842. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1843. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1844. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1845. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1846. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1847. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1848. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1849. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1850. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1851. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1852. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1853. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1854. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1855. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1856. at java.lang.Thread.run(Thread.java:748)
  1857. 2019-11-13 10:55:37,908 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1858. 2019-11-13 10:55:37,908 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1859. 2019-11-13 10:55:37,915 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 10-6636054050930425861: Forwarding null to 204920354261423
  1860. 2019-11-13 10:55:37,918 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 10-6636054050930425861: Routing from 107973147312911
  1861. 2019-11-13 10:55:37,918 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 10-6636054050930425861: Link is closed
  1862. 2019-11-13 10:55:37,918 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 10-6636054050930425861: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  1863. 2019-11-13 10:55:37,918 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  1864. 2019-11-13 10:55:37,919 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  1865. 2019-11-13 10:55:37,919 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 10-6636054050930425861: Cancelling.
  1866. 2019-11-13 10:55:37,919 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 10-6636054050930425861: Waiting some more time because this is the current command
  1867. 2019-11-13 10:55:37,919 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 10-6636054050930425861: Waiting some more time because this is the current command
  1868. 2019-11-13 10:55:37,919 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1869. 2019-11-13 10:55:37,920 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 10-6636054050930425861: Timed out on null
  1870. 2019-11-13 10:55:37,920 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Seq 10-6636054050930425861: Cancelling.
  1871. 2019-11-13 10:55:37,920 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Operation timed out: Commands 6636054050930425861 to Host 10 timed out after 3600
  1872. 2019-11-13 10:55:37,920 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-93396182) (logid:992c38a0) Unable to obtain VM statistics.
  1873. 2019-11-13 10:55:37,921 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042631: Forwarding null to 204920354261423
  1874. 2019-11-13 10:55:37,925 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 3-7917609619894042631: Routing from 107973147312911
  1875. 2019-11-13 10:55:37,925 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 3-7917609619894042631: Link is closed
  1876. 2019-11-13 10:55:37,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 3-7917609619894042631: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  1877. 2019-11-13 10:55:37,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  1878. 2019-11-13 10:55:37,930 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  1879. 2019-11-13 10:55:37,930 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 3-7917609619894042631: Cancelling.
  1880. 2019-11-13 10:55:37,933 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042631: Waiting some more time because this is the current command
  1881. 2019-11-13 10:55:37,933 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042631: Waiting some more time because this is the current command
  1882. 2019-11-13 10:55:37,933 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1883. 2019-11-13 10:55:37,934 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042631: Timed out on null
  1884. 2019-11-13 10:55:37,934 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042631: Cancelling.
  1885. 2019-11-13 10:55:37,934 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 3
  1886. com.cloud.exception.OperationTimedoutException: Commands 7917609619894042631 to Host 3 timed out after 3600
  1887. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1888. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1889. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1890. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1891. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1892. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1893. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1894. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1895. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1896. at java.lang.reflect.Method.invoke(Method.java:498)
  1897. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1898. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1899. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1900. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1901. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1902. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1903. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1904. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1905. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1906. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1907. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1908. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1909. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1910. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1911. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1912. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1913. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1914. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1915. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1916. at java.lang.Thread.run(Thread.java:748)
  1917. 2019-11-13 10:55:37,934 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1918. 2019-11-13 10:55:37,934 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1919. 2019-11-13 10:55:37,943 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425862: Forwarding null to 204920354261423
  1920. 2019-11-13 10:55:37,946 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 10-6636054050930425862: Routing from 107973147312911
  1921. 2019-11-13 10:55:37,946 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 10-6636054050930425862: Link is closed
  1922. 2019-11-13 10:55:37,946 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 10-6636054050930425862: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  1923. 2019-11-13 10:55:37,946 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  1924. 2019-11-13 10:55:37,947 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  1925. 2019-11-13 10:55:37,948 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 10-6636054050930425862: Cancelling.
  1926. 2019-11-13 10:55:37,948 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425862: Waiting some more time because this is the current command
  1927. 2019-11-13 10:55:37,948 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425862: Waiting some more time because this is the current command
  1928. 2019-11-13 10:55:37,948 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1929. 2019-11-13 10:55:37,948 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425862: Timed out on null
  1930. 2019-11-13 10:55:37,948 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425862: Cancelling.
  1931. 2019-11-13 10:55:37,948 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 10
  1932. com.cloud.exception.OperationTimedoutException: Commands 6636054050930425862 to Host 10 timed out after 3600
  1933. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1934. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1935. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1936. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1937. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1938. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1939. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1940. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1941. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1942. at java.lang.reflect.Method.invoke(Method.java:498)
  1943. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1944. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1945. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1946. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1947. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1948. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1949. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1950. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1951. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1952. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1953. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  1954. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  1955. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  1956. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  1957. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  1958. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  1959. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  1960. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  1961. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  1962. at java.lang.Thread.run(Thread.java:748)
  1963. 2019-11-13 10:55:37,949 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  1964. 2019-11-13 10:55:37,949 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  1965. 2019-11-13 10:55:37,957 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700550: Forwarding null to 204920354261423
  1966. 2019-11-13 10:55:37,961 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 9-1463388403918700550: Routing from 107973147312911
  1967. 2019-11-13 10:55:37,961 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 9-1463388403918700550: Link is closed
  1968. 2019-11-13 10:55:37,961 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 9-1463388403918700550: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  1969. 2019-11-13 10:55:37,961 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  1970. 2019-11-13 10:55:37,963 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  1971. 2019-11-13 10:55:37,963 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 9-1463388403918700550: Cancelling.
  1972. 2019-11-13 10:55:37,970 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700550: Waiting some more time because this is the current command
  1973. 2019-11-13 10:55:37,970 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700550: Waiting some more time because this is the current command
  1974. 2019-11-13 10:55:37,970 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  1975. 2019-11-13 10:55:37,970 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700550: Timed out on null
  1976. 2019-11-13 10:55:37,971 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700550: Cancelling.
  1977. 2019-11-13 10:55:37,971 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[4|PreSetup] via 9
  1978. com.cloud.exception.OperationTimedoutException: Commands 1463388403918700550 to Host 9 timed out after 3600
  1979. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  1980. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  1981. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  1982. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  1983. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  1984. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  1985. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1986. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  1987. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  1988. at java.lang.reflect.Method.invoke(Method.java:498)
  1989. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  1990. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  1991. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  1992. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  1993. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  1994. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  1995. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  1996. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  1997. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  1998. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  1999. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2000. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2001. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2002. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2003. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2004. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2005. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2006. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2007. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2008. at java.lang.Thread.run(Thread.java:748)
  2009. 2019-11-13 10:55:37,971 INFO [c.c.s.StatsCollector] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to reach Pool[4|PreSetup]
  2010. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to send command to the pool
  2011. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1059)
  2012. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2013. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2014. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2015. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2016. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2017. at java.lang.reflect.Method.invoke(Method.java:498)
  2018. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2019. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2020. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2021. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2022. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2023. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2024. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2025. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2026. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2027. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2028. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2029. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2030. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2031. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2032. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2033. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2034. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2035. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2036. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2037. at java.lang.Thread.run(Thread.java:748)
  2038. 2019-11-13 10:55:37,988 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2039. 2019-11-13 10:55:37,988 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2040. 2019-11-13 10:55:37,996 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661126: Forwarding null to 204920354261423
  2041. 2019-11-13 10:55:37,998 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 15-5775866522102661126: Routing from 107973147312911
  2042. 2019-11-13 10:55:37,998 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 15-5775866522102661126: Link is closed
  2043. 2019-11-13 10:55:37,998 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 15-5775866522102661126: MgmtId 107973147312911: Req: Resource [Host:15] is unreachable: Host 15: Link is closed
  2044. 2019-11-13 10:55:37,998 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Routing to peer
  2045. 2019-11-13 10:55:38,000 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 15--1: MgmtId 107973147312911: Req: Cancel request received
  2046. 2019-11-13 10:55:38,000 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 15-5775866522102661126: Cancelling.
  2047. 2019-11-13 10:55:38,000 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661126: Waiting some more time because this is the current command
  2048. 2019-11-13 10:55:38,000 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661126: Waiting some more time because this is the current command
  2049. 2019-11-13 10:55:38,000 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2050. 2019-11-13 10:55:38,000 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661126: Timed out on null
  2051. 2019-11-13 10:55:38,000 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 15-5775866522102661126: Cancelling.
  2052. 2019-11-13 10:55:38,000 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 15
  2053. com.cloud.exception.OperationTimedoutException: Commands 5775866522102661126 to Host 15 timed out after 3600
  2054. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2055. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2056. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2057. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2058. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2059. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2060. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2061. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2062. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2063. at java.lang.reflect.Method.invoke(Method.java:498)
  2064. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2065. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2066. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2067. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2068. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2069. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2070. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2071. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2072. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2073. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2074. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2075. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2076. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2077. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2078. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2079. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2080. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2081. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2082. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2083. at java.lang.Thread.run(Thread.java:748)
  2084. 2019-11-13 10:55:38,001 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2085. 2019-11-13 10:55:38,001 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2086. 2019-11-13 10:55:38,008 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297414: Forwarding null to 204920354261423
  2087. 2019-11-13 10:55:38,010 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 13-1616510791249297414: Routing from 107973147312911
  2088. 2019-11-13 10:55:38,010 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 13-1616510791249297414: Link is closed
  2089. 2019-11-13 10:55:38,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 13-1616510791249297414: MgmtId 107973147312911: Req: Resource [Host:13] is unreachable: Host 13: Link is closed
  2090. 2019-11-13 10:55:38,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Routing to peer
  2091. 2019-11-13 10:55:38,013 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 13--1: MgmtId 107973147312911: Req: Cancel request received
  2092. 2019-11-13 10:55:38,013 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 13-1616510791249297414: Cancelling.
  2093. 2019-11-13 10:55:38,016 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297414: Waiting some more time because this is the current command
  2094. 2019-11-13 10:55:38,016 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297414: Waiting some more time because this is the current command
  2095. 2019-11-13 10:55:38,016 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2096. 2019-11-13 10:55:38,016 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297414: Timed out on null
  2097. 2019-11-13 10:55:38,016 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 13-1616510791249297414: Cancelling.
  2098. 2019-11-13 10:55:38,016 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 13
  2099. com.cloud.exception.OperationTimedoutException: Commands 1616510791249297414 to Host 13 timed out after 3600
  2100. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2101. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2102. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2103. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2104. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2105. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2106. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2107. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2108. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2109. at java.lang.reflect.Method.invoke(Method.java:498)
  2110. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2111. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2112. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2113. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2114. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2115. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2116. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2117. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2118. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2119. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2120. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2121. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2122. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2123. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2124. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2125. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2126. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2127. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2128. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2129. at java.lang.Thread.run(Thread.java:748)
  2130. 2019-11-13 10:55:38,016 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2131. 2019-11-13 10:55:38,017 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2132. 2019-11-13 10:55:38,025 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700551: Forwarding null to 204920354261423
  2133. 2019-11-13 10:55:38,026 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 9-1463388403918700551: Routing from 107973147312911
  2134. 2019-11-13 10:55:38,026 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 9-1463388403918700551: Link is closed
  2135. 2019-11-13 10:55:38,027 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 9-1463388403918700551: MgmtId 107973147312911: Req: Resource [Host:9] is unreachable: Host 9: Link is closed
  2136. 2019-11-13 10:55:38,027 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Routing to peer
  2137. 2019-11-13 10:55:38,030 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Seq 9--1: MgmtId 107973147312911: Req: Cancel request received
  2138. 2019-11-13 10:55:38,030 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 9-1463388403918700551: Cancelling.
  2139. 2019-11-13 10:55:38,030 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700551: Waiting some more time because this is the current command
  2140. 2019-11-13 10:55:38,030 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700551: Waiting some more time because this is the current command
  2141. 2019-11-13 10:55:38,030 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2142. 2019-11-13 10:55:38,030 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700551: Timed out on null
  2143. 2019-11-13 10:55:38,031 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 9-1463388403918700551: Cancelling.
  2144. 2019-11-13 10:55:38,031 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 9
  2145. com.cloud.exception.OperationTimedoutException: Commands 1463388403918700551 to Host 9 timed out after 3600
  2146. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2147. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2148. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2149. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2150. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2151. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2152. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2153. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2154. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2155. at java.lang.reflect.Method.invoke(Method.java:498)
  2156. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2157. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2158. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2159. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2160. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2161. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2162. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2163. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2164. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2165. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2166. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2167. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2168. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2169. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2170. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2171. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2172. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2173. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2174. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2175. at java.lang.Thread.run(Thread.java:748)
  2176. 2019-11-13 10:55:38,031 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2177. 2019-11-13 10:55:38,031 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2178. 2019-11-13 10:55:38,038 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425863: Forwarding null to 204920354261423
  2179. 2019-11-13 10:55:38,040 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 10-6636054050930425863: Routing from 107973147312911
  2180. 2019-11-13 10:55:38,040 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 10-6636054050930425863: Link is closed
  2181. 2019-11-13 10:55:38,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 10-6636054050930425863: MgmtId 107973147312911: Req: Resource [Host:10] is unreachable: Host 10: Link is closed
  2182. 2019-11-13 10:55:38,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Routing to peer
  2183. 2019-11-13 10:55:38,042 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Seq 10--1: MgmtId 107973147312911: Req: Cancel request received
  2184. 2019-11-13 10:55:38,042 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-7:null) (logid:) Seq 10-6636054050930425863: Cancelling.
  2185. 2019-11-13 10:55:38,042 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425863: Waiting some more time because this is the current command
  2186. 2019-11-13 10:55:38,042 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425863: Waiting some more time because this is the current command
  2187. 2019-11-13 10:55:38,042 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2188. 2019-11-13 10:55:38,042 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425863: Timed out on null
  2189. 2019-11-13 10:55:38,042 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 10-6636054050930425863: Cancelling.
  2190. 2019-11-13 10:55:38,042 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 10
  2191. com.cloud.exception.OperationTimedoutException: Commands 6636054050930425863 to Host 10 timed out after 3600
  2192. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2193. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2194. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2195. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2196. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2197. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2198. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2199. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2200. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2201. at java.lang.reflect.Method.invoke(Method.java:498)
  2202. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2203. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2204. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2205. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2206. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2207. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2208. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2209. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2210. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2211. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2212. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2213. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2214. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2215. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2216. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2217. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2218. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2219. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2220. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2221. at java.lang.Thread.run(Thread.java:748)
  2222. 2019-11-13 10:55:38,043 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2223. 2019-11-13 10:55:38,043 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2224. 2019-11-13 10:55:38,056 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351046: Forwarding null to 204920354261423
  2225. 2019-11-13 10:55:38,057 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 74677-2875266887099351046: Routing from 107973147312911
  2226. 2019-11-13 10:55:38,057 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:null) (logid:) Seq 74677-2875266887099351046: Link is closed
  2227. 2019-11-13 10:55:38,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 74677-2875266887099351046: MgmtId 107973147312911: Req: Resource [Host:74677] is unreachable: Host 74677: Link is closed
  2228. 2019-11-13 10:55:38,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Routing to peer
  2229. 2019-11-13 10:55:38,059 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 74677--1: MgmtId 107973147312911: Req: Cancel request received
  2230. 2019-11-13 10:55:38,059 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 74677-2875266887099351046: Cancelling.
  2231. 2019-11-13 10:55:38,059 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351046: Waiting some more time because this is the current command
  2232. 2019-11-13 10:55:38,059 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351046: Waiting some more time because this is the current command
  2233. 2019-11-13 10:55:38,059 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2234. 2019-11-13 10:55:38,059 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351046: Timed out on null
  2235. 2019-11-13 10:55:38,059 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74677-2875266887099351046: Cancelling.
  2236. 2019-11-13 10:55:38,059 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 74677
  2237. com.cloud.exception.OperationTimedoutException: Commands 2875266887099351046 to Host 74677 timed out after 3600
  2238. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2239. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2240. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2241. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2242. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2243. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2244. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2245. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2246. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2247. at java.lang.reflect.Method.invoke(Method.java:498)
  2248. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2249. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2250. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2251. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2252. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2253. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2254. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2255. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2256. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2257. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2258. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2259. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2260. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2261. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2262. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2263. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2264. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2265. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2266. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2267. at java.lang.Thread.run(Thread.java:748)
  2268. 2019-11-13 10:55:38,060 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2269. 2019-11-13 10:55:38,060 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2270. 2019-11-13 10:55:38,069 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082182: Forwarding null to 204920354261423
  2271. 2019-11-13 10:55:38,071 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 74673-7260928499228082182: Routing from 107973147312911
  2272. 2019-11-13 10:55:38,071 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 74673-7260928499228082182: Link is closed
  2273. 2019-11-13 10:55:38,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 74673-7260928499228082182: MgmtId 107973147312911: Req: Resource [Host:74673] is unreachable: Host 74673: Link is closed
  2274. 2019-11-13 10:55:38,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Routing to peer
  2275. 2019-11-13 10:55:38,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Seq 74673--1: MgmtId 107973147312911: Req: Cancel request received
  2276. 2019-11-13 10:55:38,076 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-11:null) (logid:) Seq 74673-7260928499228082182: Cancelling.
  2277. 2019-11-13 10:55:38,077 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082182: Waiting some more time because this is the current command
  2278. 2019-11-13 10:55:38,077 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082182: Waiting some more time because this is the current command
  2279. 2019-11-13 10:55:38,077 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2280. 2019-11-13 10:55:38,077 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082182: Timed out on null
  2281. 2019-11-13 10:55:38,078 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 74673-7260928499228082182: Cancelling.
  2282. 2019-11-13 10:55:38,078 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 74673
  2283. com.cloud.exception.OperationTimedoutException: Commands 7260928499228082182 to Host 74673 timed out after 3600
  2284. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2285. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2286. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2287. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2288. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2289. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2290. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2291. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2292. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2293. at java.lang.reflect.Method.invoke(Method.java:498)
  2294. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2295. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2296. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2297. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2298. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2299. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2300. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2301. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2302. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2303. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2304. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2305. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2306. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2307. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2308. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2309. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2310. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2311. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2312. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2313. at java.lang.Thread.run(Thread.java:748)
  2314. 2019-11-13 10:55:38,078 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2315. 2019-11-13 10:55:38,078 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2316. 2019-11-13 10:55:38,091 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182408: Forwarding null to 204920354261423
  2317. 2019-11-13 10:55:38,093 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 2-2223652316014182408: Routing from 107973147312911
  2318. 2019-11-13 10:55:38,093 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 2-2223652316014182408: Link is closed
  2319. 2019-11-13 10:55:38,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 2-2223652316014182408: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  2320. 2019-11-13 10:55:38,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  2321. 2019-11-13 10:55:38,094 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  2322. 2019-11-13 10:55:38,095 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-13:null) (logid:) Seq 2-2223652316014182408: Cancelling.
  2323. 2019-11-13 10:55:38,095 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182408: Waiting some more time because this is the current command
  2324. 2019-11-13 10:55:38,095 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182408: Waiting some more time because this is the current command
  2325. 2019-11-13 10:55:38,095 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2326. 2019-11-13 10:55:38,095 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182408: Timed out on null
  2327. 2019-11-13 10:55:38,095 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182408: Cancelling.
  2328. 2019-11-13 10:55:38,095 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 2
  2329. com.cloud.exception.OperationTimedoutException: Commands 2223652316014182408 to Host 2 timed out after 3600
  2330. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2331. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2332. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2333. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2334. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2335. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2336. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2337. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2338. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2339. at java.lang.reflect.Method.invoke(Method.java:498)
  2340. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2341. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2342. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2343. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2344. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2345. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2346. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2347. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2348. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2349. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2350. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2351. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2352. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2353. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2354. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2355. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2356. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2357. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2358. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2359. at java.lang.Thread.run(Thread.java:748)
  2360. 2019-11-13 10:55:38,096 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2361. 2019-11-13 10:55:38,096 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2362. 2019-11-13 10:55:38,103 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042632: Forwarding null to 204920354261423
  2363. 2019-11-13 10:55:38,104 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 3-7917609619894042632: Routing from 107973147312911
  2364. 2019-11-13 10:55:38,104 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:null) (logid:) Seq 3-7917609619894042632: Link is closed
  2365. 2019-11-13 10:55:38,104 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 3-7917609619894042632: MgmtId 107973147312911: Req: Resource [Host:3] is unreachable: Host 3: Link is closed
  2366. 2019-11-13 10:55:38,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Routing to peer
  2367. 2019-11-13 10:55:38,106 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Seq 3--1: MgmtId 107973147312911: Req: Cancel request received
  2368. 2019-11-13 10:55:38,106 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-6:null) (logid:) Seq 3-7917609619894042632: Cancelling.
  2369. 2019-11-13 10:55:38,106 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042632: Waiting some more time because this is the current command
  2370. 2019-11-13 10:55:38,106 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042632: Waiting some more time because this is the current command
  2371. 2019-11-13 10:55:38,106 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2372. 2019-11-13 10:55:38,106 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042632: Timed out on null
  2373. 2019-11-13 10:55:38,106 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 3-7917609619894042632: Cancelling.
  2374. 2019-11-13 10:55:38,106 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 3
  2375. com.cloud.exception.OperationTimedoutException: Commands 7917609619894042632 to Host 3 timed out after 3600
  2376. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2377. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2378. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2379. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2380. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2381. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2382. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2383. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2384. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2385. at java.lang.reflect.Method.invoke(Method.java:498)
  2386. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2387. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2388. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2389. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2390. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2391. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2392. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2393. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2394. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2395. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2396. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2397. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2398. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2399. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2400. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2401. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2402. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2403. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2404. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2405. at java.lang.Thread.run(Thread.java:748)
  2406. 2019-11-13 10:55:38,107 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2407. 2019-11-13 10:55:38,107 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2408. 2019-11-13 10:55:38,122 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649158: Forwarding null to 204920354261423
  2409. 2019-11-13 10:55:38,124 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 19-7006475120281649158: Routing from 107973147312911
  2410. 2019-11-13 10:55:38,124 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-1:null) (logid:) Seq 19-7006475120281649158: Link is closed
  2411. 2019-11-13 10:55:38,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 19-7006475120281649158: MgmtId 107973147312911: Req: Resource [Host:19] is unreachable: Host 19: Link is closed
  2412. 2019-11-13 10:55:38,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Routing to peer
  2413. 2019-11-13 10:55:38,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Seq 19--1: MgmtId 107973147312911: Req: Cancel request received
  2414. 2019-11-13 10:55:38,125 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 19-7006475120281649158: Cancelling.
  2415. 2019-11-13 10:55:38,125 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649158: Waiting some more time because this is the current command
  2416. 2019-11-13 10:55:38,125 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649158: Waiting some more time because this is the current command
  2417. 2019-11-13 10:55:38,126 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2418. 2019-11-13 10:55:38,126 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649158: Timed out on null
  2419. 2019-11-13 10:55:38,126 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 19-7006475120281649158: Cancelling.
  2420. 2019-11-13 10:55:38,126 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[5|PreSetup] via 19
  2421. com.cloud.exception.OperationTimedoutException: Commands 7006475120281649158 to Host 19 timed out after 3600
  2422. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2423. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2424. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2425. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
  2426. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2427. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2428. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2429. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2430. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2431. at java.lang.reflect.Method.invoke(Method.java:498)
  2432. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2433. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2434. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2435. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2436. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2437. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2438. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2439. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2440. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2441. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2442. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2443. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2444. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2445. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2446. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2447. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2448. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2449. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2450. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2451. at java.lang.Thread.run(Thread.java:748)
  2452. 2019-11-13 10:55:38,126 INFO [c.c.s.StatsCollector] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to reach Pool[5|PreSetup]
  2453. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:5] is unreachable: Unable to send command to the pool
  2454. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1059)
  2455. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:418)
  2456. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:432)
  2457. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2458. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2459. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2460. at java.lang.reflect.Method.invoke(Method.java:498)
  2461. at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
  2462. at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
  2463. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  2464. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
  2465. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
  2466. at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  2467. at com.sun.proxy.$Proxy97.sendToPool(Unknown Source)
  2468. at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:1004)
  2469. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  2470. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  2471. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  2472. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  2473. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  2474. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  2475. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  2476. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  2477. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  2478. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2479. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2480. at java.lang.Thread.run(Thread.java:748)
  2481. 2019-11-13 10:55:38,147 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
  2482. 2019-11-13 10:55:38,147 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) We are returning the default host to execute commands because the command is not of Copy type.
  2483. 2019-11-13 10:55:38,156 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182409: Forwarding null to 204920354261423
  2484. 2019-11-13 10:55:38,159 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 2-2223652316014182409: Routing from 107973147312911
  2485. 2019-11-13 10:55:38,159 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:null) (logid:) Seq 2-2223652316014182409: Link is closed
  2486. 2019-11-13 10:55:38,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 2-2223652316014182409: MgmtId 107973147312911: Req: Resource [Host:2] is unreachable: Host 2: Link is closed
  2487. 2019-11-13 10:55:38,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Routing to peer
  2488. 2019-11-13 10:55:38,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 2--1: MgmtId 107973147312911: Req: Cancel request received
  2489. 2019-11-13 10:55:38,161 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 2-2223652316014182409: Cancelling.
  2490. 2019-11-13 10:55:38,161 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182409: Waiting some more time because this is the current command
  2491. 2019-11-13 10:55:38,161 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182409: Waiting some more time because this is the current command
  2492. 2019-11-13 10:55:38,161 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
  2493. 2019-11-13 10:55:38,161 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182409: Timed out on null
  2494. 2019-11-13 10:55:38,161 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Seq 2-2223652316014182409: Cancelling.
  2495. 2019-11-13 10:55:38,161 DEBUG [c.c.s.StorageManagerImpl] (StatsCollector-3:ctx-df80aa18) (logid:f197c75e) Unable to send storage pool command to Pool[6|PreSetup] via 2
  2496. com.cloud.exception.OperationTimedoutException: Commands 2223652316014182409 to Host 2 timed out after 3600
  2497. at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:440)
  2498. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:462)
  2499. at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:368)
  2500. at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1049)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!