API tools faq
paste
Guest User

Untitled

a guest
Mar 28th, 2014
46
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 158.31 KB | None | 0 0
raw download clone embed print report
  1. 2014-03-28 23:26:59,436 INFO [c.c.u.c.ComponentContext] (main:null) Starting com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStack_15a541ee
  2. 2014-03-28 23:26:59,437 INFO [c.c.u.c.ComponentContext] (main:null) Starting com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_1a56bc94
  3. 2014-03-28 23:26:59,438 INFO [c.c.u.c.ComponentContext] (main:null) Starting com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_6ce3a144
  4. 2014-03-28 23:26:59,438 INFO [c.c.u.c.ComponentContext] (main:null) Starting com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloudStack_98143bb9
  5. 2014-03-28 23:27:00,443 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-8baa8652) HostStatsCollector is running...
  6. 2014-03-28 23:27:00,446 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-3bde4a18) VmStatsCollector is running...
  7. 2014-03-28 23:27:00,449 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-fdba6c32) StorageCollector is running...
  8. 2014-03-28 23:27:00,485 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-14:ctx-1332c742) Seq 1-16056331: Executing request
  9. 2014-03-28 23:27:00,492 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-fdba6c32) There is no secondary storage VM for secondary storage host nfs://192.168.1.216/secondary
  10. 2014-03-28 23:27:00,505 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-15:ctx-e23a89d6) Seq 1-16056332: Executing request
  11. 2014-03-28 23:27:00,638 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-14:ctx-1332c742) Seq 1-16056331: Response Received:
  12. 2014-03-28 23:27:00,639 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8baa8652) Seq 1-16056331: Received: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  13. 2014-03-28 23:27:00,656 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-16:ctx-1b44ee34) Seq 2-1891631113: Executing request
  14. 2014-03-28 23:27:00,675 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-15:ctx-e23a89d6) Seq 1-16056332: Response Received:
  15. 2014-03-28 23:27:00,675 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-fdba6c32) Seq 1-16056332: Received: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
  16. 2014-03-28 23:27:00,912 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-16:ctx-1b44ee34) Seq 2-1891631113: Response Received:
  17. 2014-03-28 23:27:00,913 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8baa8652) Seq 2-1891631113: Received: { Ans: , MgmtId: 126274761388196, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  18. 2014-03-28 23:27:00,929 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-17:ctx-d015538a) Seq 3-376307721: Executing request
  19. 2014-03-28 23:27:01,142 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-17:ctx-d015538a) Seq 3-376307721: Response Received:
  20. 2014-03-28 23:27:01,143 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8baa8652) Seq 3-376307721: Received: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  21. 2014-03-28 23:27:01,154 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-9:ctx-461b34d7) Seq 4-642908170: Executing request
  22. 2014-03-28 23:27:01,341 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-9:ctx-461b34d7) Seq 4-642908170: Response Received:
  23. 2014-03-28 23:27:01,344 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8baa8652) Seq 4-642908170: Received: { Ans: , MgmtId: 126274761388196, via: 4, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
  24. 2014-03-28 23:27:15,370 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aae9d191) Checking if any host reservation can be released ...
  25. 2014-03-28 23:27:15,376 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Running Capacity Checker ...
  26. 2014-03-28 23:27:15,376 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) recalculating system capacity
  27. 2014-03-28 23:27:15,376 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Executing cpu/ram capacity update
  28. 2014-03-28 23:27:15,389 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aae9d191) Cannot release reservation, Found VM: VM[SecondaryStorageVm|s-13-VM] Stopped but reserved on host 1
  29. 2014-03-28 23:27:15,393 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 0 VMs on host 1
  30. 2014-03-28 23:27:15,398 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 1 VM, not running on host 1
  31. 2014-03-28 23:27:15,402 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate cpu capacity, host:1 usedCpu: 0 reservedCpu: 500
  32. 2014-03-28 23:27:15,403 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate memory capacity, host:1 usedMem: 0 reservedMem: 536870912
  33. 2014-03-28 23:27:15,403 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aae9d191) Host has no VMs associated, releasing the planner reservation for host 2
  34. 2014-03-28 23:27:15,413 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 0 VMs on host 2
  35. 2014-03-28 23:27:15,420 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 0 VM, not running on host 2
  36. 2014-03-28 23:27:15,427 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-bc08a7d6) Found 0 routers to update status.
  37. 2014-03-28 23:27:15,430 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-bc08a7d6) Found 0 networks to update RvR status.
  38. 2014-03-28 23:27:15,439 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate cpu capacity, host:2 usedCpu: 0 reservedCpu: 0
  39. 2014-03-28 23:27:15,439 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate memory capacity, host:2 usedMem: 0 reservedMem: 0
  40. 2014-03-28 23:27:15,441 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aae9d191) Host has no VMs associated, releasing the planner reservation for host 3
  41. 2014-03-28 23:27:15,453 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 0 VMs on host 3
  42. 2014-03-28 23:27:15,457 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aae9d191) Cannot release reservation, Found VM: VM[ConsoleProxy|v-2-VM] Stopped but reserved on host 4
  43. 2014-03-28 23:27:15,459 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-aae9d191) Done running HostReservationReleaseChecker ...
  44. 2014-03-28 23:27:15,459 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 0 VM, not running on host 3
  45. 2014-03-28 23:27:15,462 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate cpu capacity, host:3 usedCpu: 0 reservedCpu: 0
  46. 2014-03-28 23:27:15,462 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate memory capacity, host:3 usedMem: 0 reservedMem: 0
  47. 2014-03-28 23:27:15,466 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 0 VMs on host 4
  48. 2014-03-28 23:27:15,471 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) Found 1 VM, not running on host 4
  49. 2014-03-28 23:27:15,474 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate cpu capacity, host:4 usedCpu: 0 reservedCpu: 500
  50. 2014-03-28 23:27:15,474 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-2935d624) No need to calibrate memory capacity, host:4 usedMem: 0 reservedMem: 1073741824
  51. 2014-03-28 23:27:15,474 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Done executing cpu/ram capacity update
  52. 2014-03-28 23:27:15,474 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Executing storage capacity update
  53. 2014-03-28 23:27:15,488 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-2935d624) Successfully set Capacity - 59812872192 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 3, PodId 1
  54. 2014-03-28 23:27:15,488 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Done executing storage capacity update
  55. 2014-03-28 23:27:15,488 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Executing capacity updates for public ip and Vlans
  56. 2014-03-28 23:27:15,494 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Done capacity updates for public ip and Vlans
  57. 2014-03-28 23:27:15,494 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Executing capacity updates for private ip
  58. 2014-03-28 23:27:15,507 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Done executing capacity updates for private ip
  59. 2014-03-28 23:27:15,507 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Done recalculating system capacity
  60. 2014-03-28 23:27:15,538 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-2935d624) Done running Capacity Checker ...
  61. 2014-03-28 23:27:33,584 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-befa82ae) Zone 1 is ready to launch secondary storage VM
  62. 2014-03-28 23:27:33,605 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-c6b337fc) Zone 1 is ready to launch console proxy
  63. 2014-03-28 23:27:33,605 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-c6b337fc) Expand console proxy standby capacity for zone master
  64. 2014-03-28 23:27:33,613 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-c6b337fc) Found a stopped console proxy, bring it up to running pool. proxy vm id : 2
  65. 2014-03-28 23:27:33,622 INFO [c.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-befa82ae) No running secondary storage vms found in datacenter id=1, starting one
  66. 2014-03-28 23:27:33,634 INFO [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-befa82ae) Found a stopped secondary storage vm, bring it up to running pool. secStorageVm vm id : 13
  67. 2014-03-28 23:27:33,639 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: 4 new host id: null host id before state transition: null
  68. 2014-03-28 23:27:33,639 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Successfully transitioned to start state for VM[ConsoleProxy|v-2-VM] reservation id = bd17156e-47d8-4ab4-8eaa-d092a3448357
  69. 2014-03-28 23:27:33,644 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Trying to deploy VM, vm has dcId: 1 and podId: 1
  70. 2014-03-28 23:27:33,644 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Deploy avoids pods: null, clusters: null, hosts: null
  71. 2014-03-28 23:27:33,654 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: 1 new host id: null host id before state transition: null
  72. 2014-03-28 23:27:33,654 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully transitioned to start state for VM[SecondaryStorageVm|s-13-VM] reservation id = d7a59a72-8ffb-4fb7-b330-deac03f36ccd
  73. 2014-03-28 23:27:33,654 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Deploy avoids pods: null, clusters: null, hosts: null
  74. 2014-03-28 23:27:33,656 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Trying to deploy VM, vm has dcId: 1 and podId: 1
  75. 2014-03-28 23:27:33,656 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Deploy avoids pods: null, clusters: null, hosts: null
  76. 2014-03-28 23:27:33,656 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@32f56fb
  77. 2014-03-28 23:27:33,657 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 1073741824
  78. 2014-03-28 23:27:33,657 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Is ROOT volume READY (pool already allocated)?: No
  79. 2014-03-28 23:27:33,658 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) This VM has last host_id specified, trying to choose the same host: 4
  80. 2014-03-28 23:27:33,664 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Deploy avoids pods: null, clusters: null, hosts: null
  81. 2014-03-28 23:27:33,666 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@32f56fb
  82. 2014-03-28 23:27:33,666 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 536870912
  83. 2014-03-28 23:27:33,666 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Is ROOT volume READY (pool already allocated)?: No
  84. 2014-03-28 23:27:33,666 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) This VM has last host_id specified, trying to choose the same host: 1
  85. 2014-03-28 23:27:33,669 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Checking if host: 4 has enough capacity for requested CPU: 500 and requested RAM: 1073741824 , cpuOverprovisioningFactor: 1.0
  86. 2014-03-28 23:27:33,674 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  87. 2014-03-28 23:27:33,675 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) We need to allocate to the last host again, so checking if there is enough reserved capacity
  88. 2014-03-28 23:27:33,675 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Reserved CPU: 500 , Requested CPU: 500
  89. 2014-03-28 23:27:33,675 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Reserved RAM: 1073741824 , Requested RAM: 1073741824
  90. 2014-03-28 23:27:33,675 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Host has enough CPU and RAM available
  91. 2014-03-28 23:27:33,676 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) STATS: Can alloc CPU from host: 4, used: 0, reserved: 500, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:true ,considerReservedCapacity?: true
  92. 2014-03-28 23:27:33,676 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) STATS: Can alloc MEM from host: 4, used: 0, reserved: 1073741824, total: 35803627520; requested mem: 1073741824,alloc_from_last_host?:true ,considerReservedCapacity?: true
  93. 2014-03-28 23:27:33,676 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  94. 2014-03-28 23:27:33,679 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Host: 4 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  95. 2014-03-28 23:27:33,679 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) The last host of this VM is UP and has enough capacity
  96. 2014-03-28 23:27:33,679 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Now checking for suitable pools under zone: 1, pod: 1, cluster: 1
  97. 2014-03-28 23:27:33,681 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  98. 2014-03-28 23:27:33,682 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) We need to allocate to the last host again, so checking if there is enough reserved capacity
  99. 2014-03-28 23:27:33,682 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Reserved CPU: 500 , Requested CPU: 500
  100. 2014-03-28 23:27:33,682 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Reserved RAM: 536870912 , Requested RAM: 536870912
  101. 2014-03-28 23:27:33,682 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Host has enough CPU and RAM available
  102. 2014-03-28 23:27:33,682 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) STATS: Can alloc CPU from host: 1, used: 0, reserved: 500, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:true ,considerReservedCapacity?: true
  103. 2014-03-28 23:27:33,682 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) STATS: Can alloc MEM from host: 1, used: 0, reserved: 536870912, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:true ,considerReservedCapacity?: true
  104. 2014-03-28 23:27:33,685 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Checking suitable pools for volume (Id, Type): (2,ROOT)
  105. 2014-03-28 23:27:33,685 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) We need to allocate new storagepool for this volume
  106. 2014-03-28 23:27:33,685 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Host: 1 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  107. 2014-03-28 23:27:33,686 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) The last host of this VM is UP and has enough capacity
  108. 2014-03-28 23:27:33,686 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Now checking for suitable pools under zone: 1, pod: 1, cluster: 1
  109. 2014-03-28 23:27:33,688 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Calling StoragePoolAllocators to find suitable pools
  110. 2014-03-28 23:27:33,690 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking suitable pools for volume (Id, Type): (13,ROOT)
  111. 2014-03-28 23:27:33,691 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) We need to allocate new storagepool for this volume
  112. 2014-03-28 23:27:33,692 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Calling StoragePoolAllocators to find suitable pools
  113. 2014-03-28 23:27:33,693 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  114. 2014-03-28 23:27:33,693 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) ClusterScopeStoragePoolAllocator looking for storage pool
  115. 2014-03-28 23:27:33,694 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) Looking for pools in dc: 1 pod:1 cluster:1
  116. 2014-03-28 23:27:33,696 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) Found pools matching tags: [Pool[3|NetworkFilesystem]]
  117. 2014-03-28 23:27:33,697 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (secstorage-1:ctx-befa82ae) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  118. 2014-03-28 23:27:33,697 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator looking for storage pool
  119. 2014-03-28 23:27:33,698 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Looking for pools in dc: 1 pod:1 cluster:1
  120. 2014-03-28 23:27:33,700 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) Removing pool Pool[3|NetworkFilesystem] from avoid set, must have been inserted when searching for another disk's tag
  121. 2014-03-28 23:27:33,700 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Found pools matching tags: [Pool[3|NetworkFilesystem]]
  122. 2014-03-28 23:27:33,707 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) Checking if storage pool is suitable, name: null ,poolId: 3
  123. 2014-03-28 23:27:33,708 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Removing pool Pool[3|NetworkFilesystem] from avoid set, must have been inserted when searching for another disk's tag
  124. 2014-03-28 23:27:33,710 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Checking if storage pool is suitable, name: null ,poolId: 3
  125. 2014-03-28 23:27:33,715 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-c6b337fc) Checking pool 3 for storage, totalSize: 29906436096, usedBytes: 3239575552, usedPct: 0.10832369131517128, disable threshold: 0.85
  126. 2014-03-28 23:27:33,718 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool 3 for storage, totalSize: 29906436096, usedBytes: 3239575552, usedPct: 0.10832369131517128, disable threshold: 0.85
  127. 2014-03-28 23:27:33,727 DEBUG [c.c.s.StorageManagerImpl] (consoleproxy-1:ctx-c6b337fc) Checking pool: 3 for volume allocation [Vol[2|vm=2|ROOT]], maxSize : 59812872192, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  128. 2014-03-28 23:27:33,727 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (consoleproxy-1:ctx-c6b337fc) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  129. 2014-03-28 23:27:33,727 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  130. 2014-03-28 23:27:33,728 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool: 3 for volume allocation [Vol[13|vm=13|ROOT]], maxSize : 59812872192, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  131. 2014-03-28 23:27:33,728 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  132. 2014-03-28 23:27:33,728 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  133. 2014-03-28 23:27:33,729 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Checking if host: 4 can access any suitable storage pool for volume: ROOT
  134. 2014-03-28 23:27:33,729 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking if host: 1 can access any suitable storage pool for volume: ROOT
  135. 2014-03-28 23:27:33,730 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Host: 1 can access pool: 3
  136. 2014-03-28 23:27:33,730 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Host: 4 can access pool: 3
  137. 2014-03-28 23:27:33,730 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Found a potential host id: 1 name: xenserver-nmvotzdm and associated storage pools for this VM
  138. 2014-03-28 23:27:33,731 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Found a potential host id: 4 name: xenserver-sxdjiipt and associated storage pools for this VM
  139. 2014-03-28 23:27:33,731 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (consoleproxy-1:ctx-c6b337fc) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(4)-Storage(Volume(2|ROOT-->Pool(3))]
  140. 2014-03-28 23:27:33,731 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Deployment found - P0=VM[ConsoleProxy|v-2-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(4)-Storage(Volume(2|ROOT-->Pool(3))]
  141. 2014-03-28 23:27:33,731 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(13|ROOT-->Pool(3))]
  142. 2014-03-28 23:27:33,732 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Deployment found - P0=VM[SecondaryStorageVm|s-13-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(13|ROOT-->Pool(3))]
  143. 2014-03-28 23:27:33,744 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: 4 new host id: 4 host id before state transition: null
  144. 2014-03-28 23:27:33,744 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) VM starting again on the last host it was stopped on
  145. 2014-03-28 23:27:33,744 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: 1 new host id: 1 host id before state transition: null
  146. 2014-03-28 23:27:33,745 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM starting again on the last host it was stopped on
  147. 2014-03-28 23:27:33,764 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  148. 2014-03-28 23:27:33,764 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) We are allocating VM, increasing the used capacity of this host:1
  149. 2014-03-28 23:27:33,764 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used CPU: 0 , Free CPU:44300 ,Requested CPU: 500
  150. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used RAM: 0 , Free RAM:35266756608 ,Requested RAM: 536870912
  151. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) We are allocating VM to the last host again, so adjusting the reserved capacity if it is not less than required
  152. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Reserved CPU: 500 , Requested CPU: 500
  153. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Reserved RAM: 536870912 , Requested RAM: 536870912
  154. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) CPU STATS after allocation: for host: 1, old used: 0, old reserved: 500, actual total: 44800, total with overprovisioning: 44800; new used:500, reserved:0; requested cpu:500,alloc_from_last:true
  155. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) RAM STATS after allocation: for host: 1, old used: 0, old reserved: 536870912, total: 35803627520; new used: 536870912, reserved: 0; requested mem: 536870912,alloc_from_last:true
  156. 2014-03-28 23:27:33,765 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  157. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) We are allocating VM, increasing the used capacity of this host:4
  158. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Current Used CPU: 0 , Free CPU:44300 ,Requested CPU: 500
  159. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Current Used RAM: 0 , Free RAM:34729885696 ,Requested RAM: 1073741824
  160. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) We are allocating VM to the last host again, so adjusting the reserved capacity if it is not less than required
  161. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Reserved CPU: 500 , Requested CPU: 500
  162. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Reserved RAM: 1073741824 , Requested RAM: 1073741824
  163. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) CPU STATS after allocation: for host: 4, old used: 0, old reserved: 500, actual total: 44800, total with overprovisioning: 44800; new used:500, reserved:0; requested cpu:500,alloc_from_last:true
  164. 2014-03-28 23:27:33,766 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) RAM STATS after allocation: for host: 4, old used: 0, old reserved: 1073741824, total: 35803627520; new used: 1073741824, reserved: 0; requested mem: 1073741824,alloc_from_last:true
  165. 2014-03-28 23:27:33,768 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) VM is being created in podId: 1
  166. 2014-03-28 23:27:33,769 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) VM is being created in podId: 1
  167. 2014-03-28 23:27:33,788 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Network id=204 is already implemented
  168. 2014-03-28 23:27:33,788 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=204 is already implemented
  169. 2014-03-28 23:27:33,836 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Asking SecurityGroupProvider to prepare for Nic[5-2-bd17156e-47d8-4ab4-8eaa-d092a3448357-71.43.84.195]
  170. 2014-03-28 23:27:33,836 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  171. 2014-03-28 23:27:33,841 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  172. 2014-03-28 23:27:33,841 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Asking VirtualRouter to prepare for Nic[5-2-bd17156e-47d8-4ab4-8eaa-d092a3448357-71.43.84.195]
  173. 2014-03-28 23:27:33,855 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=202 is already implemented
  174. 2014-03-28 23:27:33,859 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Network id=202 is already implemented
  175. 2014-03-28 23:27:33,895 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=201 is already implemented
  176. 2014-03-28 23:27:33,896 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Network id=201 is already implemented
  177. 2014-03-28 23:27:33,925 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-c6b337fc) Allocated a nic NicProfile[7-2-bd17156e-47d8-4ab4-8eaa-d092a3448357-71.43.84.212-null for VM[ConsoleProxy|v-2-VM]
  178. 2014-03-28 23:27:33,925 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[50-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.208-null for VM[SecondaryStorageVm|s-13-VM]
  179. 2014-03-28 23:27:33,937 DEBUG [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-c6b337fc) Checking if we need to prepare 1 volumes for VM[ConsoleProxy|v-2-VM]
  180. 2014-03-28 23:27:33,939 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=203 is already implemented
  181. 2014-03-28 23:27:33,972 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[51-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.214-null for VM[SecondaryStorageVm|s-13-VM]
  182. 2014-03-28 23:27:33,977 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-13-VM]
  183. 2014-03-28 23:27:33,990 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-c6b337fc) template 1 is already in store:1, type:Image
  184. 2014-03-28 23:27:34,001 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:1, type:Image
  185. 2014-03-28 23:27:34,012 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:3, type:Primary
  186. 2014-03-28 23:27:34,013 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (consoleproxy-1:ctx-c6b337fc) template 1 is already in store:3, type:Primary
  187. 2014-03-28 23:27:34,015 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Found template routing-1 in storage pool 3 with VMTemplateStoragePool id: 94
  188. 2014-03-28 23:27:34,019 DEBUG [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-c6b337fc) Found template routing-1 in storage pool 3 with VMTemplateStoragePool id: 94
  189. 2014-03-28 23:27:34,026 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Acquire lock on VMTemplateStoragePool 94 with timeout 3600 seconds
  190. 2014-03-28 23:27:34,029 DEBUG [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-c6b337fc) Acquire lock on VMTemplateStoragePool 94 with timeout 3600 seconds
  191. 2014-03-28 23:27:34,031 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) lock is acquired for VMTemplateStoragePool 94
  192. 2014-03-28 23:27:34,041 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (secstorage-1:ctx-befa82ae) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
  193. 2014-03-28 23:27:34,060 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 1-16056333: Sending { Cmd , MgmtId: 126274761388196, via: 1(xenserver-nmvotzdm), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  194. 2014-03-28 23:27:34,062 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 1-16056333: Executing: { Cmd , MgmtId: 126274761388196, via: 1(xenserver-nmvotzdm), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  195. 2014-03-28 23:27:34,062 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-18:ctx-b3d44140) Seq 1-16056333: Executing request
  196. 2014-03-28 23:27:36,170 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-18:ctx-b3d44140) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-156a9da1-5de0-4ee0-8145-c355eddbc5c7
  197. 2014-03-28 23:27:36,171 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-18:ctx-b3d44140) can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  198. 2014-03-28 23:27:36,171 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-18:ctx-b3d44140) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  199. com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  200. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:848)
  201. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:918)
  202. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:75)
  203. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
  204. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:609)
  205. at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
  206. at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
  207. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
  208. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  209. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  210. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  211. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  212. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  213. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  214. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  215. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  216. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
  217. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
  218. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  219. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  220. at java.lang.Thread.run(Thread.java:701)
  221. 2014-03-28 23:27:36,173 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-18:ctx-b3d44140) Seq 1-16056333: Response Received:
  222. 2014-03-28 23:27:36,174 DEBUG [c.c.a.t.Request] (DirectAgent-18:ctx-b3d44140) Seq 1-16056333: Processing: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8","wait":0}}] }
  223. 2014-03-28 23:27:36,174 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 1-16056333: Received: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  224. 2014-03-28 23:27:36,186 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) releasing lock for VMTemplateStoragePool 94
  225. 2014-03-28 23:27:36,186 WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-befa82ae) Was unable to find lock for the key template_spool_ref94 and thread id 1118349204
  226. 2014-03-28 23:27:36,187 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  227. 2014-03-28 23:27:36,187 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Unable to contact resource.
  228. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  229. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1202)
  230. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1254)
  231. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:962)
  232. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  233. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  234. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  235. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  236. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  237. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  238. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  239. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  240. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  241. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  242. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  243. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  244. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  245. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  246. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  247. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  248. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  249. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  250. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  251. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  252. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  253. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  254. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  255. at java.lang.Thread.run(Thread.java:701)
  256. 2014-03-28 23:27:36,192 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Cleaning up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  257. 2014-03-28 23:27:36,197 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 1-16056334: Sending { Cmd , MgmtId: 126274761388196, via: 1(xenserver-nmvotzdm), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  258. 2014-03-28 23:27:36,197 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 1-16056334: Executing: { Cmd , MgmtId: 126274761388196, via: 1(xenserver-nmvotzdm), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  259. 2014-03-28 23:27:36,198 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-19:ctx-4f73b804) Seq 1-16056334: Executing request
  260. 2014-03-28 23:27:36,258 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-19:ctx-4f73b804) VM does not exist on XenServer3b043635-6b22-4b17-ae30-6de62f7d7457
  261. 2014-03-28 23:27:36,258 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-19:ctx-4f73b804) Seq 1-16056334: Response Received:
  262. 2014-03-28 23:27:36,258 DEBUG [c.c.a.t.Request] (DirectAgent-19:ctx-4f73b804) Seq 1-16056334: Processing: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
  263. 2014-03-28 23:27:36,259 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 1-16056334: Received: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, { StopAnswer } }
  264. 2014-03-28 23:27:36,282 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  265. 2014-03-28 23:27:36,284 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  266. 2014-03-28 23:27:36,287 DEBUG [c.c.n.g.ControlNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[49-13-null-null-null
  267. 2014-03-28 23:27:36,294 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=50
  268. 2014-03-28 23:27:36,294 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[50-13-null-null-null
  269. 2014-03-28 23:27:36,302 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=51
  270. 2014-03-28 23:27:36,302 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[51-13-null-null-null
  271. 2014-03-28 23:27:36,305 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully released network resources for the vm VM[SecondaryStorageVm|s-13-VM]
  272. 2014-03-28 23:27:36,306 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  273. 2014-03-28 23:27:36,311 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Deploy avoids pods: null, clusters: null, hosts: [1]
  274. 2014-03-28 23:27:36,312 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@32f56fb
  275. 2014-03-28 23:27:36,312 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 536870912
  276. 2014-03-28 23:27:36,312 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Is ROOT volume READY (pool already allocated)?: No
  277. 2014-03-28 23:27:36,312 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) This VM has last host_id specified, trying to choose the same host: 1
  278. 2014-03-28 23:27:36,314 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) The last host of this VM is in avoid set
  279. 2014-03-28 23:27:36,314 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Cannot choose the last host to deploy this VM
  280. 2014-03-28 23:27:36,314 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Searching resources only under specified Pod: 1
  281. 2014-03-28 23:27:36,314 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  282. 2014-03-28 23:27:36,324 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking resources in Cluster: 1 under Pod: 1
  283. 2014-03-28 23:27:36,324 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  284. 2014-03-28 23:27:36,329 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) FirstFitAllocator has 4 hosts to check for allocation: [Host[-1-Routing], Host[-4-Routing], Host[-2-Routing], Host[-3-Routing]]
  285. 2014-03-28 23:27:36,335 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found 4 hosts for allocation after prioritization: [Host[-1-Routing], Host[-4-Routing], Host[-2-Routing], Host[-3-Routing]]
  286. 2014-03-28 23:27:36,335 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
  287. 2014-03-28 23:27:36,335 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-nmvotzdm, hostId: 1 is in avoid set, skipping this and trying other available hosts
  288. 2014-03-28 23:27:36,341 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host: 4 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  289. 2014-03-28 23:27:36,342 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Checking if host: 4 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  290. 2014-03-28 23:27:36,344 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  291. 2014-03-28 23:27:36,344 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free CPU: 44300 , Requested CPU: 500
  292. 2014-03-28 23:27:36,344 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free RAM: 34729885696 , Requested RAM: 536870912
  293. 2014-03-28 23:27:36,344 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host has enough CPU and RAM available
  294. 2014-03-28 23:27:36,344 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 4, used: 500, reserved: 0, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  295. 2014-03-28 23:27:36,344 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 4, used: 1073741824, reserved: 0, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  296. 2014-03-28 23:27:36,345 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found a suitable host, adding to list: 4
  297. 2014-03-28 23:27:36,351 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host: 2 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  298. 2014-03-28 23:27:36,351 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  299. 2014-03-28 23:27:36,353 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  300. 2014-03-28 23:27:36,353 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free CPU: 44800 , Requested CPU: 500
  301. 2014-03-28 23:27:36,353 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free RAM: 35803627520 , Requested RAM: 536870912
  302. 2014-03-28 23:27:36,353 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host has enough CPU and RAM available
  303. 2014-03-28 23:27:36,353 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  304. 2014-03-28 23:27:36,353 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 2, used: 0, reserved: 0, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  305. 2014-03-28 23:27:36,353 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found a suitable host, adding to list: 2
  306. 2014-03-28 23:27:36,359 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host: 3 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  307. 2014-03-28 23:27:36,360 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Checking if host: 3 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  308. 2014-03-28 23:27:36,362 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  309. 2014-03-28 23:27:36,362 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free CPU: 44800 , Requested CPU: 500
  310. 2014-03-28 23:27:36,362 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free RAM: 35803627520 , Requested RAM: 536870912
  311. 2014-03-28 23:27:36,362 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host has enough CPU and RAM available
  312. 2014-03-28 23:27:36,362 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 3, used: 0, reserved: 0, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  313. 2014-03-28 23:27:36,362 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 3, used: 0, reserved: 0, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  314. 2014-03-28 23:27:36,362 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found a suitable host, adding to list: 3
  315. 2014-03-28 23:27:36,362 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host Allocator returning 3 suitable hosts
  316. 2014-03-28 23:27:36,364 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking suitable pools for volume (Id, Type): (13,ROOT)
  317. 2014-03-28 23:27:36,364 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) We need to allocate new storagepool for this volume
  318. 2014-03-28 23:27:36,364 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Calling StoragePoolAllocators to find suitable pools
  319. 2014-03-28 23:27:36,365 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (secstorage-1:ctx-befa82ae) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  320. 2014-03-28 23:27:36,365 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator looking for storage pool
  321. 2014-03-28 23:27:36,365 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Looking for pools in dc: 1 pod:1 cluster:1
  322. 2014-03-28 23:27:36,367 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Found pools matching tags: [Pool[3|NetworkFilesystem]]
  323. 2014-03-28 23:27:36,368 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Removing pool Pool[3|NetworkFilesystem] from avoid set, must have been inserted when searching for another disk's tag
  324. 2014-03-28 23:27:36,369 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Checking if storage pool is suitable, name: null ,poolId: 3
  325. 2014-03-28 23:27:36,373 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool 3 for storage, totalSize: 29906436096, usedBytes: 3239575552, usedPct: 0.10832369131517128, disable threshold: 0.85
  326. 2014-03-28 23:27:36,378 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool: 3 for volume allocation [Vol[13|vm=13|ROOT]], maxSize : 59812872192, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  327. 2014-03-28 23:27:36,378 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  328. 2014-03-28 23:27:36,378 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  329. 2014-03-28 23:27:36,378 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking if host: 4 can access any suitable storage pool for volume: ROOT
  330. 2014-03-28 23:27:36,378 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Host: 4 can access pool: 3
  331. 2014-03-28 23:27:36,379 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Found a potential host id: 4 name: xenserver-sxdjiipt and associated storage pools for this VM
  332. 2014-03-28 23:27:36,380 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(4)-Storage(Volume(13|ROOT-->Pool(3))]
  333. 2014-03-28 23:27:36,380 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Deployment found - P0=VM[SecondaryStorageVm|s-13-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(4)-Storage(Volume(13|ROOT-->Pool(3))]
  334. 2014-03-28 23:27:36,387 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: 1 new host id: 4 host id before state transition: 1
  335. 2014-03-28 23:27:36,394 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  336. 2014-03-28 23:27:36,394 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total RAM: 35803627776 and RAM after applying overprovisioning: 35803627520
  337. 2014-03-28 23:27:36,394 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release cpu from host: 1, old used: 500,reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  338. 2014-03-28 23:27:36,394 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release mem from host: 1, old used: 536870912,reserved: 0, total: 35803627520; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  339. 2014-03-28 23:27:36,404 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  340. 2014-03-28 23:27:36,404 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) We are allocating VM, increasing the used capacity of this host:4
  341. 2014-03-28 23:27:36,404 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used CPU: 500 , Free CPU:44300 ,Requested CPU: 500
  342. 2014-03-28 23:27:36,404 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used RAM: 1073741824 , Free RAM:34729885696 ,Requested RAM: 536870912
  343. 2014-03-28 23:27:36,404 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) CPU STATS after allocation: for host: 4, old used: 500, old reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used:1000, reserved:0; requested cpu:500,alloc_from_last:false
  344. 2014-03-28 23:27:36,404 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) RAM STATS after allocation: for host: 4, old used: 1073741824, old reserved: 0, total: 35803627520; new used: 1610612736, reserved: 0; requested mem: 536870912,alloc_from_last:false
  345. 2014-03-28 23:27:36,407 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) VM is being created in podId: 1
  346. 2014-03-28 23:27:36,419 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=204 is already implemented
  347. 2014-03-28 23:27:36,441 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  348. 2014-03-28 23:27:36,444 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  349. 2014-03-28 23:27:36,449 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=202 is already implemented
  350. 2014-03-28 23:27:36,469 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=201 is already implemented
  351. 2014-03-28 23:27:36,483 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[50-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.209-null for VM[SecondaryStorageVm|s-13-VM]
  352. 2014-03-28 23:27:36,489 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=203 is already implemented
  353. 2014-03-28 23:27:36,510 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[51-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.214-null for VM[SecondaryStorageVm|s-13-VM]
  354. 2014-03-28 23:27:36,515 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-13-VM]
  355. 2014-03-28 23:27:36,536 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:1, type:Image
  356. 2014-03-28 23:27:36,543 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (secstorage-1:ctx-befa82ae) Not found (templateId:1poolId:3) in template_spool_ref, persisting it
  357. 2014-03-28 23:27:36,552 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:3, type:Primary
  358. 2014-03-28 23:27:36,553 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Found template routing-1 in storage pool 3 with VMTemplateStoragePool id: 95
  359. 2014-03-28 23:27:36,565 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Acquire lock on VMTemplateStoragePool 95 with timeout 3600 seconds
  360. 2014-03-28 23:27:36,567 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) lock is acquired for VMTemplateStoragePool 95
  361. 2014-03-28 23:27:36,574 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (secstorage-1:ctx-befa82ae) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
  362. 2014-03-28 23:27:36,596 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307722: Sending { Cmd , MgmtId: 126274761388196, via: 3(xenserver-nqeazvvs), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  363. 2014-03-28 23:27:36,597 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307722: Executing: { Cmd , MgmtId: 126274761388196, via: 3(xenserver-nqeazvvs), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  364. 2014-03-28 23:27:36,598 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-6:ctx-1712ef15) Seq 3-376307722: Executing request
  365. 2014-03-28 23:27:38,714 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-6:ctx-1712ef15) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-36c7de0f-0a7c-4640-b77b-4314284f44df
  366. 2014-03-28 23:27:38,715 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-6:ctx-1712ef15) can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  367. 2014-03-28 23:27:38,716 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-6:ctx-1712ef15) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  368. com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  369. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:848)
  370. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:918)
  371. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:75)
  372. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
  373. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:609)
  374. at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
  375. at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
  376. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
  377. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  378. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  379. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  380. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  381. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  382. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  383. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  384. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  385. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
  386. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
  387. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  388. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  389. at java.lang.Thread.run(Thread.java:701)
  390. 2014-03-28 23:27:38,717 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-6:ctx-1712ef15) Seq 3-376307722: Response Received:
  391. 2014-03-28 23:27:38,718 DEBUG [c.c.a.t.Request] (DirectAgent-6:ctx-1712ef15) Seq 3-376307722: Processing: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8","wait":0}}] }
  392. 2014-03-28 23:27:38,718 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307722: Received: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  393. 2014-03-28 23:27:38,729 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) releasing lock for VMTemplateStoragePool 95
  394. 2014-03-28 23:27:38,729 WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-befa82ae) Was unable to find lock for the key template_spool_ref95 and thread id 1118349204
  395. 2014-03-28 23:27:38,730 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  396. 2014-03-28 23:27:38,730 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Unable to contact resource.
  397. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  398. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1202)
  399. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1254)
  400. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:962)
  401. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  402. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  403. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  404. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  405. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  406. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  407. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  408. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  409. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  410. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  411. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  412. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  413. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  414. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  415. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  416. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  417. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  418. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  419. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  420. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  421. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  422. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  423. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  424. at java.lang.Thread.run(Thread.java:701)
  425. 2014-03-28 23:27:38,734 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Cleaning up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  426. 2014-03-28 23:27:38,739 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 4-642908171: Sending { Cmd , MgmtId: 126274761388196, via: 4(xenserver-sxdjiipt), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  427. 2014-03-28 23:27:38,740 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 4-642908171: Executing: { Cmd , MgmtId: 126274761388196, via: 4(xenserver-sxdjiipt), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  428. 2014-03-28 23:27:38,740 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-7:ctx-8760b606) Seq 4-642908171: Executing request
  429. 2014-03-28 23:27:38,820 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-7:ctx-8760b606) VM does not exist on XenServerf7cc2535-abf0-4346-b946-b9194b90e340
  430. 2014-03-28 23:27:38,820 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-7:ctx-8760b606) Seq 4-642908171: Response Received:
  431. 2014-03-28 23:27:38,821 DEBUG [c.c.a.t.Request] (DirectAgent-7:ctx-8760b606) Seq 4-642908171: Processing: { Ans: , MgmtId: 126274761388196, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
  432. 2014-03-28 23:27:38,821 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 4-642908171: Received: { Ans: , MgmtId: 126274761388196, via: 4, Ver: v1, Flags: 10, { StopAnswer } }
  433. 2014-03-28 23:27:38,842 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  434. 2014-03-28 23:27:38,847 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  435. 2014-03-28 23:27:38,849 DEBUG [c.c.n.g.ControlNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[49-13-null-null-null
  436. 2014-03-28 23:27:38,856 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=50
  437. 2014-03-28 23:27:38,856 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[50-13-null-null-null
  438. 2014-03-28 23:27:38,864 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=51
  439. 2014-03-28 23:27:38,864 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[51-13-null-null-null
  440. 2014-03-28 23:27:38,871 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully released network resources for the vm VM[SecondaryStorageVm|s-13-VM]
  441. 2014-03-28 23:27:38,871 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  442. 2014-03-28 23:27:38,879 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Deploy avoids pods: null, clusters: null, hosts: [1, 4]
  443. 2014-03-28 23:27:38,880 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@32f56fb
  444. 2014-03-28 23:27:38,880 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 536870912
  445. 2014-03-28 23:27:38,880 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Is ROOT volume READY (pool already allocated)?: No
  446. 2014-03-28 23:27:38,880 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) This VM has last host_id specified, trying to choose the same host: 1
  447. 2014-03-28 23:27:38,882 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) The last host of this VM is in avoid set
  448. 2014-03-28 23:27:38,882 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Cannot choose the last host to deploy this VM
  449. 2014-03-28 23:27:38,882 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Searching resources only under specified Pod: 1
  450. 2014-03-28 23:27:38,882 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  451. 2014-03-28 23:27:38,896 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking resources in Cluster: 1 under Pod: 1
  452. 2014-03-28 23:27:38,896 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  453. 2014-03-28 23:27:38,901 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) FirstFitAllocator has 4 hosts to check for allocation: [Host[-3-Routing], Host[-1-Routing], Host[-2-Routing], Host[-4-Routing]]
  454. 2014-03-28 23:27:38,913 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found 4 hosts for allocation after prioritization: [Host[-3-Routing], Host[-1-Routing], Host[-2-Routing], Host[-4-Routing]]
  455. 2014-03-28 23:27:38,913 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
  456. 2014-03-28 23:27:38,922 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host: 3 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  457. 2014-03-28 23:27:38,922 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Checking if host: 3 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  458. 2014-03-28 23:27:38,927 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  459. 2014-03-28 23:27:38,927 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free CPU: 44800 , Requested CPU: 500
  460. 2014-03-28 23:27:38,927 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free RAM: 35803627520 , Requested RAM: 536870912
  461. 2014-03-28 23:27:38,927 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host has enough CPU and RAM available
  462. 2014-03-28 23:27:38,927 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 3, used: 0, reserved: 0, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  463. 2014-03-28 23:27:38,927 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 3, used: 0, reserved: 0, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  464. 2014-03-28 23:27:38,927 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found a suitable host, adding to list: 3
  465. 2014-03-28 23:27:38,927 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-nmvotzdm, hostId: 1 is in avoid set, skipping this and trying other available hosts
  466. 2014-03-28 23:27:38,937 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host: 2 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  467. 2014-03-28 23:27:38,937 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  468. 2014-03-28 23:27:38,942 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  469. 2014-03-28 23:27:38,942 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free CPU: 44800 , Requested CPU: 500
  470. 2014-03-28 23:27:38,942 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free RAM: 35803627520 , Requested RAM: 536870912
  471. 2014-03-28 23:27:38,942 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host has enough CPU and RAM available
  472. 2014-03-28 23:27:38,942 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  473. 2014-03-28 23:27:38,942 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 2, used: 0, reserved: 0, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  474. 2014-03-28 23:27:38,942 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found a suitable host, adding to list: 2
  475. 2014-03-28 23:27:38,942 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-sxdjiipt, hostId: 4 is in avoid set, skipping this and trying other available hosts
  476. 2014-03-28 23:27:38,942 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host Allocator returning 2 suitable hosts
  477. 2014-03-28 23:27:38,944 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking suitable pools for volume (Id, Type): (13,ROOT)
  478. 2014-03-28 23:27:38,944 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) We need to allocate new storagepool for this volume
  479. 2014-03-28 23:27:38,944 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Calling StoragePoolAllocators to find suitable pools
  480. 2014-03-28 23:27:38,945 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (secstorage-1:ctx-befa82ae) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  481. 2014-03-28 23:27:38,945 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator looking for storage pool
  482. 2014-03-28 23:27:38,945 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Looking for pools in dc: 1 pod:1 cluster:1
  483. 2014-03-28 23:27:38,946 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Found pools matching tags: [Pool[3|NetworkFilesystem]]
  484. 2014-03-28 23:27:38,948 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Removing pool Pool[3|NetworkFilesystem] from avoid set, must have been inserted when searching for another disk's tag
  485. 2014-03-28 23:27:38,949 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Checking if storage pool is suitable, name: null ,poolId: 3
  486. 2014-03-28 23:27:38,953 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool 3 for storage, totalSize: 29906436096, usedBytes: 3239575552, usedPct: 0.10832369131517128, disable threshold: 0.85
  487. 2014-03-28 23:27:38,957 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool: 3 for volume allocation [Vol[13|vm=13|ROOT]], maxSize : 59812872192, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  488. 2014-03-28 23:27:38,957 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  489. 2014-03-28 23:27:38,957 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  490. 2014-03-28 23:27:38,958 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking if host: 3 can access any suitable storage pool for volume: ROOT
  491. 2014-03-28 23:27:38,958 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Host: 3 can access pool: 3
  492. 2014-03-28 23:27:38,962 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Found a potential host id: 3 name: xenserver-nqeazvvs and associated storage pools for this VM
  493. 2014-03-28 23:27:38,964 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(3)-Storage(Volume(13|ROOT-->Pool(3))]
  494. 2014-03-28 23:27:38,964 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Deployment found - P0=VM[SecondaryStorageVm|s-13-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(3)-Storage(Volume(13|ROOT-->Pool(3))]
  495. 2014-03-28 23:27:38,974 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: 1 new host id: 3 host id before state transition: 4
  496. 2014-03-28 23:27:38,980 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  497. 2014-03-28 23:27:38,980 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total RAM: 35803627776 and RAM after applying overprovisioning: 35803627520
  498. 2014-03-28 23:27:38,980 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release cpu from host: 4, old used: 1000,reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
  499. 2014-03-28 23:27:38,980 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release mem from host: 4, old used: 1610612736,reserved: 0, total: 35803627520; new used: 1073741824,reserved:0; movedfromreserved: false,moveToReserveredfalse
  500. 2014-03-28 23:27:38,996 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  501. 2014-03-28 23:27:38,996 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) We are allocating VM, increasing the used capacity of this host:3
  502. 2014-03-28 23:27:38,996 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used CPU: 0 , Free CPU:44800 ,Requested CPU: 500
  503. 2014-03-28 23:27:38,996 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used RAM: 0 , Free RAM:35803627520 ,Requested RAM: 536870912
  504. 2014-03-28 23:27:38,996 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) CPU STATS after allocation: for host: 3, old used: 0, old reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
  505. 2014-03-28 23:27:38,996 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) RAM STATS after allocation: for host: 3, old used: 0, old reserved: 0, total: 35803627520; new used: 536870912, reserved: 0; requested mem: 536870912,alloc_from_last:false
  506. 2014-03-28 23:27:38,998 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) VM is being created in podId: 1
  507. 2014-03-28 23:27:39,013 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=204 is already implemented
  508. 2014-03-28 23:27:39,032 INFO [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-c6b337fc) Unable to acquire lock on VMTemplateStoragePool 94
  509. 2014-03-28 23:27:39,033 ERROR [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Failed to start instance VM[ConsoleProxy|v-2-VM]
  510. java.lang.NullPointerException
  511. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:454)
  512. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:603)
  513. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1190)
  514. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1254)
  515. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:962)
  516. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  517. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  518. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  519. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  520. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  521. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  522. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  523. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  524. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  525. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  526. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  527. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  528. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  529. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  530. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  531. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  532. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  533. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  534. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  535. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  536. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  537. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  538. at java.lang.Thread.run(Thread.java:701)
  539. 2014-03-28 23:27:39,038 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Cleaning up resources for the vm VM[ConsoleProxy|v-2-VM] in Starting state
  540. 2014-03-28 23:27:39,042 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c6b337fc) Seq 4-642908172: Sending { Cmd , MgmtId: 126274761388196, via: 4(xenserver-sxdjiipt), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-2-VM","wait":0}}] }
  541. 2014-03-28 23:27:39,044 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c6b337fc) Seq 4-642908172: Executing: { Cmd , MgmtId: 126274761388196, via: 4(xenserver-sxdjiipt), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-2-VM","wait":0}}] }
  542. 2014-03-28 23:27:39,044 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  543. 2014-03-28 23:27:39,045 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-2:ctx-85415853) Seq 4-642908172: Executing request
  544. 2014-03-28 23:27:39,049 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  545. 2014-03-28 23:27:39,055 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=202 is already implemented
  546. 2014-03-28 23:27:39,075 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=201 is already implemented
  547. 2014-03-28 23:27:39,090 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[50-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.208-null for VM[SecondaryStorageVm|s-13-VM]
  548. 2014-03-28 23:27:39,096 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=203 is already implemented
  549. 2014-03-28 23:27:39,132 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-85415853) VM does not exist on XenServerf7cc2535-abf0-4346-b946-b9194b90e340
  550. 2014-03-28 23:27:39,132 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-2:ctx-85415853) Seq 4-642908172: Response Received:
  551. 2014-03-28 23:27:39,132 DEBUG [c.c.a.t.Request] (DirectAgent-2:ctx-85415853) Seq 4-642908172: Processing: { Ans: , MgmtId: 126274761388196, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
  552. 2014-03-28 23:27:39,132 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c6b337fc) Seq 4-642908172: Received: { Ans: , MgmtId: 126274761388196, via: 4, Ver: v1, Flags: 10, { StopAnswer } }
  553. 2014-03-28 23:27:39,153 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[51-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.215-null for VM[SecondaryStorageVm|s-13-VM]
  554. 2014-03-28 23:27:39,159 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-13-VM]
  555. 2014-03-28 23:27:39,161 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Asking SecurityGroupProvider to release NicProfile[5-2-bd17156e-47d8-4ab4-8eaa-d092a3448357-71.43.84.195-vlan://untagged
  556. 2014-03-28 23:27:39,165 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-c6b337fc) Asking VirtualRouter to release NicProfile[5-2-bd17156e-47d8-4ab4-8eaa-d092a3448357-71.43.84.195-vlan://untagged
  557. 2014-03-28 23:27:39,171 DEBUG [c.c.n.g.ControlNetworkGuru] (consoleproxy-1:ctx-c6b337fc) Released nic: NicProfile[6-2-null-null-null
  558. 2014-03-28 23:27:39,188 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (consoleproxy-1:ctx-c6b337fc) Releasing ip address for reservationId=bd17156e-47d8-4ab4-8eaa-d092a3448357, instance=7
  559. 2014-03-28 23:27:39,188 DEBUG [c.c.n.g.PodBasedNetworkGuru] (consoleproxy-1:ctx-c6b337fc) Released nic: NicProfile[7-2-null-null-null
  560. 2014-03-28 23:27:39,195 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:1, type:Image
  561. 2014-03-28 23:27:39,196 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Successfully released network resources for the vm VM[ConsoleProxy|v-2-VM]
  562. 2014-03-28 23:27:39,196 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c6b337fc) Successfully cleanued up resources for the vm VM[ConsoleProxy|v-2-VM] in Starting state
  563. 2014-03-28 23:27:39,206 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (secstorage-1:ctx-befa82ae) Not found (templateId:1poolId:3) in template_spool_ref, persisting it
  564. 2014-03-28 23:27:39,206 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 4 new host id: null host id before state transition: 4
  565. 2014-03-28 23:27:39,218 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:3, type:Primary
  566. 2014-03-28 23:27:39,220 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Found template routing-1 in storage pool 3 with VMTemplateStoragePool id: 96
  567. 2014-03-28 23:27:39,225 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  568. 2014-03-28 23:27:39,225 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) Hosts's actual total RAM: 35803627776 and RAM after applying overprovisioning: 35803627520
  569. 2014-03-28 23:27:39,225 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) release cpu from host: 4, old used: 500,reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  570. 2014-03-28 23:27:39,225 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-c6b337fc) release mem from host: 4, old used: 1073741824,reserved: 0, total: 35803627520; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  571. 2014-03-28 23:27:39,228 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Acquire lock on VMTemplateStoragePool 96 with timeout 3600 seconds
  572. 2014-03-28 23:27:39,230 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-c6b337fc) Exception while trying to start console proxy
  573. com.cloud.exception.AgentUnavailableException: Resource [Host:4] is unreachable: Host 4: Unable to start instance due to null
  574. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1072)
  575. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  576. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  577. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:555)
  578. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:941)
  579. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1666)
  580. at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
  581. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:118)
  582. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  583. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  584. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  585. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  586. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  587. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  588. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  589. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  590. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  591. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  592. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  593. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  594. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  595. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  596. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  597. at java.lang.Thread.run(Thread.java:701)
  598. Caused by: java.lang.NullPointerException
  599. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:454)
  600. at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:603)
  601. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1190)
  602. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1254)
  603. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:962)
  604. ... 23 more
  605. 2014-03-28 23:27:39,233 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) lock is acquired for VMTemplateStoragePool 96
  606. 2014-03-28 23:27:39,237 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (secstorage-1:ctx-befa82ae) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
  607. 2014-03-28 23:27:39,251 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 2-1891631114: Sending { Cmd , MgmtId: 126274761388196, via: 2(xenserver-xzcdirqf), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  608. 2014-03-28 23:27:39,253 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 2-1891631114: Executing: { Cmd , MgmtId: 126274761388196, via: 2(xenserver-xzcdirqf), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  609. 2014-03-28 23:27:39,253 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-20:ctx-fbc3a17b) Seq 2-1891631114: Executing request
  610. 2014-03-28 23:27:41,368 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-20:ctx-fbc3a17b) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-736dbb9e-f9f5-4134-80df-8e2a0d163ef8
  611. 2014-03-28 23:27:41,368 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-20:ctx-fbc3a17b) can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  612. 2014-03-28 23:27:41,368 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-20:ctx-fbc3a17b) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  613. com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  614. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:848)
  615. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:918)
  616. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:75)
  617. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
  618. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:609)
  619. at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
  620. at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
  621. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
  622. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  623. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  624. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  625. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  626. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  627. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  628. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  629. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  630. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
  631. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
  632. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  633. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  634. at java.lang.Thread.run(Thread.java:701)
  635. 2014-03-28 23:27:41,369 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-20:ctx-fbc3a17b) Seq 2-1891631114: Response Received:
  636. 2014-03-28 23:27:41,369 DEBUG [c.c.a.t.Request] (DirectAgent-20:ctx-fbc3a17b) Seq 2-1891631114: Processing: { Ans: , MgmtId: 126274761388196, via: 2, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8","wait":0}}] }
  637. 2014-03-28 23:27:41,369 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 2-1891631114: Received: { Ans: , MgmtId: 126274761388196, via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  638. 2014-03-28 23:27:41,376 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) releasing lock for VMTemplateStoragePool 96
  639. 2014-03-28 23:27:41,376 WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-befa82ae) Was unable to find lock for the key template_spool_ref96 and thread id 1118349204
  640. 2014-03-28 23:27:41,376 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  641. 2014-03-28 23:27:41,377 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Unable to contact resource.
  642. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  643. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1202)
  644. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1254)
  645. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:962)
  646. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  647. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  648. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  649. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  650. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  651. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  652. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  653. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  654. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  655. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  656. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  657. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  658. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  659. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  660. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  661. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  662. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  663. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  664. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  665. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  666. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  667. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  668. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  669. at java.lang.Thread.run(Thread.java:701)
  670. 2014-03-28 23:27:41,380 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Cleaning up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  671. 2014-03-28 23:27:41,382 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307723: Sending { Cmd , MgmtId: 126274761388196, via: 3(xenserver-nqeazvvs), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  672. 2014-03-28 23:27:41,383 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307723: Executing: { Cmd , MgmtId: 126274761388196, via: 3(xenserver-nqeazvvs), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  673. 2014-03-28 23:27:41,383 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-10:ctx-16b1e65c) Seq 3-376307723: Executing request
  674. 2014-03-28 23:27:41,459 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-10:ctx-16b1e65c) VM does not exist on XenServer7d95fdcc-f439-4e2a-83a8-d7a79d69c92a
  675. 2014-03-28 23:27:41,459 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-10:ctx-16b1e65c) Seq 3-376307723: Response Received:
  676. 2014-03-28 23:27:41,460 DEBUG [c.c.a.t.Request] (DirectAgent-10:ctx-16b1e65c) Seq 3-376307723: Processing: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
  677. 2014-03-28 23:27:41,460 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307723: Received: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, { StopAnswer } }
  678. 2014-03-28 23:27:41,482 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  679. 2014-03-28 23:27:41,484 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  680. 2014-03-28 23:27:41,487 DEBUG [c.c.n.g.ControlNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[49-13-null-null-null
  681. 2014-03-28 23:27:41,493 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=50
  682. 2014-03-28 23:27:41,494 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[50-13-null-null-null
  683. 2014-03-28 23:27:41,501 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=51
  684. 2014-03-28 23:27:41,501 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[51-13-null-null-null
  685. 2014-03-28 23:27:41,504 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully released network resources for the vm VM[SecondaryStorageVm|s-13-VM]
  686. 2014-03-28 23:27:41,504 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  687. 2014-03-28 23:27:41,507 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Deploy avoids pods: null, clusters: null, hosts: [1, 3, 4]
  688. 2014-03-28 23:27:41,508 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@32f56fb
  689. 2014-03-28 23:27:41,508 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 536870912
  690. 2014-03-28 23:27:41,508 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Is ROOT volume READY (pool already allocated)?: No
  691. 2014-03-28 23:27:41,509 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) This VM has last host_id specified, trying to choose the same host: 1
  692. 2014-03-28 23:27:41,510 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) The last host of this VM is in avoid set
  693. 2014-03-28 23:27:41,510 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Cannot choose the last host to deploy this VM
  694. 2014-03-28 23:27:41,510 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Searching resources only under specified Pod: 1
  695. 2014-03-28 23:27:41,510 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  696. 2014-03-28 23:27:41,518 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking resources in Cluster: 1 under Pod: 1
  697. 2014-03-28 23:27:41,519 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  698. 2014-03-28 23:27:41,523 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) FirstFitAllocator has 4 hosts to check for allocation: [Host[-2-Routing], Host[-4-Routing], Host[-1-Routing], Host[-3-Routing]]
  699. 2014-03-28 23:27:41,528 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found 4 hosts for allocation after prioritization: [Host[-2-Routing], Host[-4-Routing], Host[-1-Routing], Host[-3-Routing]]
  700. 2014-03-28 23:27:41,528 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
  701. 2014-03-28 23:27:41,536 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host: 2 has cpu capability (cpu:16, speed:2800) to support requested CPU: 1 and requested speed: 500
  702. 2014-03-28 23:27:41,536 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
  703. 2014-03-28 23:27:41,539 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  704. 2014-03-28 23:27:41,539 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free CPU: 44800 , Requested CPU: 500
  705. 2014-03-28 23:27:41,539 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Free RAM: 35803627520 , Requested RAM: 536870912
  706. 2014-03-28 23:27:41,539 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host has enough CPU and RAM available
  707. 2014-03-28 23:27:41,539 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 44800, total with overprovisioning: 44800; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
  708. 2014-03-28 23:27:41,539 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 2, used: 0, reserved: 0, total: 35803627520; requested mem: 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: true
  709. 2014-03-28 23:27:41,539 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found a suitable host, adding to list: 2
  710. 2014-03-28 23:27:41,539 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-sxdjiipt, hostId: 4 is in avoid set, skipping this and trying other available hosts
  711. 2014-03-28 23:27:41,539 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-nmvotzdm, hostId: 1 is in avoid set, skipping this and trying other available hosts
  712. 2014-03-28 23:27:41,539 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-nqeazvvs, hostId: 3 is in avoid set, skipping this and trying other available hosts
  713. 2014-03-28 23:27:41,539 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host Allocator returning 1 suitable hosts
  714. 2014-03-28 23:27:41,541 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking suitable pools for volume (Id, Type): (13,ROOT)
  715. 2014-03-28 23:27:41,541 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) We need to allocate new storagepool for this volume
  716. 2014-03-28 23:27:41,541 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Calling StoragePoolAllocators to find suitable pools
  717. 2014-03-28 23:27:41,542 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (secstorage-1:ctx-befa82ae) LocalStoragePoolAllocator trying to find storage pool to fit the vm
  718. 2014-03-28 23:27:41,542 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator looking for storage pool
  719. 2014-03-28 23:27:41,542 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Looking for pools in dc: 1 pod:1 cluster:1
  720. 2014-03-28 23:27:41,543 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Found pools matching tags: [Pool[3|NetworkFilesystem]]
  721. 2014-03-28 23:27:41,545 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Removing pool Pool[3|NetworkFilesystem] from avoid set, must have been inserted when searching for another disk's tag
  722. 2014-03-28 23:27:41,546 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (secstorage-1:ctx-befa82ae) Checking if storage pool is suitable, name: null ,poolId: 3
  723. 2014-03-28 23:27:41,550 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool 3 for storage, totalSize: 29906436096, usedBytes: 3239575552, usedPct: 0.10832369131517128, disable threshold: 0.85
  724. 2014-03-28 23:27:41,554 DEBUG [c.c.s.StorageManagerImpl] (secstorage-1:ctx-befa82ae) Checking pool: 3 for volume allocation [Vol[13|vm=13|ROOT]], maxSize : 59812872192, totalAllocatedSize : 0, askingSize : 0, allocated disable threshold: 0.85
  725. 2014-03-28 23:27:41,554 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (secstorage-1:ctx-befa82ae) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
  726. 2014-03-28 23:27:41,554 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
  727. 2014-03-28 23:27:41,554 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking if host: 2 can access any suitable storage pool for volume: ROOT
  728. 2014-03-28 23:27:41,555 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Host: 2 can access pool: 3
  729. 2014-03-28 23:27:41,557 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Found a potential host id: 2 name: xenserver-xzcdirqf and associated storage pools for this VM
  730. 2014-03-28 23:27:41,559 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(13|ROOT-->Pool(3))]
  731. 2014-03-28 23:27:41,559 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Deployment found - P0=VM[SecondaryStorageVm|s-13-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(13|ROOT-->Pool(3))]
  732. 2014-03-28 23:27:41,581 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: 1 new host id: 2 host id before state transition: 3
  733. 2014-03-28 23:27:41,590 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  734. 2014-03-28 23:27:41,590 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total RAM: 35803627776 and RAM after applying overprovisioning: 35803627520
  735. 2014-03-28 23:27:41,590 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release cpu from host: 3, old used: 500,reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  736. 2014-03-28 23:27:41,590 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release mem from host: 3, old used: 536870912,reserved: 0, total: 35803627520; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  737. 2014-03-28 23:27:41,602 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  738. 2014-03-28 23:27:41,602 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) We are allocating VM, increasing the used capacity of this host:2
  739. 2014-03-28 23:27:41,602 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used CPU: 0 , Free CPU:44800 ,Requested CPU: 500
  740. 2014-03-28 23:27:41,602 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Current Used RAM: 0 , Free RAM:35803627520 ,Requested RAM: 536870912
  741. 2014-03-28 23:27:41,602 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) CPU STATS after allocation: for host: 2, old used: 0, old reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
  742. 2014-03-28 23:27:41,602 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) RAM STATS after allocation: for host: 2, old used: 0, old reserved: 0, total: 35803627520; new used: 536870912, reserved: 0; requested mem: 536870912,alloc_from_last:false
  743. 2014-03-28 23:27:41,604 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) VM is being created in podId: 1
  744. 2014-03-28 23:27:41,613 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=204 is already implemented
  745. 2014-03-28 23:27:41,637 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  746. 2014-03-28 23:27:41,639 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to prepare for Nic[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196]
  747. 2014-03-28 23:27:41,645 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=202 is already implemented
  748. 2014-03-28 23:27:41,663 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=201 is already implemented
  749. 2014-03-28 23:27:41,678 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[50-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.210-null for VM[SecondaryStorageVm|s-13-VM]
  750. 2014-03-28 23:27:41,689 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Network id=203 is already implemented
  751. 2014-03-28 23:27:41,709 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Allocated a nic NicProfile[51-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.209-null for VM[SecondaryStorageVm|s-13-VM]
  752. 2014-03-28 23:27:41,714 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Checking if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-13-VM]
  753. 2014-03-28 23:27:41,729 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:1, type:Image
  754. 2014-03-28 23:27:41,732 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (secstorage-1:ctx-befa82ae) Not found (templateId:1poolId:3) in template_spool_ref, persisting it
  755. 2014-03-28 23:27:41,737 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-befa82ae) template 1 is already in store:3, type:Primary
  756. 2014-03-28 23:27:41,739 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Found template routing-1 in storage pool 3 with VMTemplateStoragePool id: 97
  757. 2014-03-28 23:27:41,744 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) Acquire lock on VMTemplateStoragePool 97 with timeout 3600 seconds
  758. 2014-03-28 23:27:41,745 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) lock is acquired for VMTemplateStoragePool 97
  759. 2014-03-28 23:27:41,751 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (secstorage-1:ctx-befa82ae) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
  760. 2014-03-28 23:27:41,765 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307724: Sending { Cmd , MgmtId: 126274761388196, via: 3(xenserver-nqeazvvs), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  761. 2014-03-28 23:27:41,767 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307724: Executing: { Cmd , MgmtId: 126274761388196, via: 3(xenserver-nqeazvvs), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/1/","origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.1.216/secondary","_role":"Image"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2","uuid":"5c77ac86-b6ee-11e3-8937-72d8a24bf0a4","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"90f3ce31-9bed-3d8d-9260-3d498ded3bad","id":3,"poolType":"NetworkFilesystem","host":"192.168.1.216","path":"/primary","port":2049,"url":"NetworkFilesystem://192.168.1.216//primary/?ROLE=Primary&STOREUUID=90f3ce31-9bed-3d8d-9260-3d498ded3bad"}},"name":"routing-1","hypervisorType":"XenServer"}},"executeInSequence":false,"options":{},"wait":10800}}] }
  762. 2014-03-28 23:27:41,769 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-8:ctx-f57f13dd) Seq 3-376307724: Executing request
  763. 2014-03-28 23:27:43,880 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-8:ctx-f57f13dd) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-cd80ee1d-f996-45e3-8c04-122d4dab3afd
  764. 2014-03-28 23:27:43,881 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-8:ctx-f57f13dd) can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  765. 2014-03-28 23:27:43,881 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-8:ctx-f57f13dd) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  766. com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  767. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:848)
  768. at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:918)
  769. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:75)
  770. at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
  771. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:609)
  772. at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
  773. at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
  774. at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
  775. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  776. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  777. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  778. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  779. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  780. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  781. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  782. at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  783. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
  784. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
  785. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  786. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  787. at java.lang.Thread.run(Thread.java:701)
  788. 2014-03-28 23:27:43,882 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-8:ctx-f57f13dd) Seq 3-376307724: Response Received:
  789. 2014-03-28 23:27:43,883 DEBUG [c.c.a.t.Request] (DirectAgent-8:ctx-f57f13dd) Seq 3-376307724: Processing: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8","wait":0}}] }
  790. 2014-03-28 23:27:43,883 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 3-376307724: Received: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
  791. 2014-03-28 23:27:43,894 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-befa82ae) releasing lock for VMTemplateStoragePool 97
  792. 2014-03-28 23:27:43,894 WARN [c.c.u.d.Merovingian2] (secstorage-1:ctx-befa82ae) Was unable to find lock for the key template_spool_ref97 and thread id 1118349204
  793. 2014-03-28 23:27:43,894 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-befa82ae) Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  794. 2014-03-28 23:27:43,895 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Unable to contact resource.
  795. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to create Vol[13|vm=13|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 56c29764-8f12-0fb1-9006-c9bbabab2cd8
  796. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1202)
  797. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1254)
  798. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:962)
  799. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  800. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  801. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  802. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  803. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  804. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  805. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  806. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  807. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  808. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  809. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  810. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  811. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  812. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  813. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  814. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  815. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  816. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  817. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  818. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  819. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  820. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  821. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  822. at java.lang.Thread.run(Thread.java:701)
  823. 2014-03-28 23:27:43,899 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Cleaning up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  824. 2014-03-28 23:27:43,901 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 2-1891631115: Sending { Cmd , MgmtId: 126274761388196, via: 2(xenserver-xzcdirqf), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  825. 2014-03-28 23:27:43,906 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 2-1891631115: Executing: { Cmd , MgmtId: 126274761388196, via: 2(xenserver-xzcdirqf), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-13-VM","wait":0}}] }
  826. 2014-03-28 23:27:43,906 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-21:ctx-531915b5) Seq 2-1891631115: Executing request
  827. 2014-03-28 23:27:43,982 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-21:ctx-531915b5) VM does not exist on XenServer8285d0ae-4ed1-4669-b70e-0142efb1fe00
  828. 2014-03-28 23:27:43,983 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-21:ctx-531915b5) Seq 2-1891631115: Response Received:
  829. 2014-03-28 23:27:43,983 DEBUG [c.c.a.t.Request] (DirectAgent-21:ctx-531915b5) Seq 2-1891631115: Processing: { Ans: , MgmtId: 126274761388196, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
  830. 2014-03-28 23:27:43,983 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-befa82ae) Seq 2-1891631115: Received: { Ans: , MgmtId: 126274761388196, via: 2, Ver: v1, Flags: 10, { StopAnswer } }
  831. 2014-03-28 23:27:44,002 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking SecurityGroupProvider to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  832. 2014-03-28 23:27:44,005 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Asking VirtualRouter to release NicProfile[48-13-d7a59a72-8ffb-4fb7-b330-deac03f36ccd-71.43.84.196-vlan://untagged
  833. 2014-03-28 23:27:44,007 DEBUG [c.c.n.g.ControlNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[49-13-null-null-null
  834. 2014-03-28 23:27:44,013 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=50
  835. 2014-03-28 23:27:44,014 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[50-13-null-null-null
  836. 2014-03-28 23:27:44,021 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (secstorage-1:ctx-befa82ae) Releasing ip address for reservationId=d7a59a72-8ffb-4fb7-b330-deac03f36ccd, instance=51
  837. 2014-03-28 23:27:44,021 DEBUG [c.c.n.g.PodBasedNetworkGuru] (secstorage-1:ctx-befa82ae) Released nic: NicProfile[51-13-null-null-null
  838. 2014-03-28 23:27:44,024 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully released network resources for the vm VM[SecondaryStorageVm|s-13-VM]
  839. 2014-03-28 23:27:44,024 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-13-VM] in Starting state
  840. 2014-03-28 23:27:44,027 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Deploy avoids pods: null, clusters: null, hosts: [1, 2, 3, 4]
  841. 2014-03-28 23:27:44,028 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@32f56fb
  842. 2014-03-28 23:27:44,028 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: 536870912
  843. 2014-03-28 23:27:44,028 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Is ROOT volume READY (pool already allocated)?: No
  844. 2014-03-28 23:27:44,028 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) This VM has last host_id specified, trying to choose the same host: 1
  845. 2014-03-28 23:27:44,030 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) The last host of this VM is in avoid set
  846. 2014-03-28 23:27:44,030 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Cannot choose the last host to deploy this VM
  847. 2014-03-28 23:27:44,030 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Searching resources only under specified Pod: 1
  848. 2014-03-28 23:27:44,030 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  849. 2014-03-28 23:27:44,038 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Checking resources in Cluster: 1 under Pod: 1
  850. 2014-03-28 23:27:44,038 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
  851. 2014-03-28 23:27:44,044 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) FirstFitAllocator has 4 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing], Host[-4-Routing], Host[-3-Routing]]
  852. 2014-03-28 23:27:44,051 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Found 4 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing], Host[-4-Routing], Host[-3-Routing]]
  853. 2014-03-28 23:27:44,051 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
  854. 2014-03-28 23:27:44,051 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-xzcdirqf, hostId: 2 is in avoid set, skipping this and trying other available hosts
  855. 2014-03-28 23:27:44,051 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-nmvotzdm, hostId: 1 is in avoid set, skipping this and trying other available hosts
  856. 2014-03-28 23:27:44,051 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-sxdjiipt, hostId: 4 is in avoid set, skipping this and trying other available hosts
  857. 2014-03-28 23:27:44,052 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host name: xenserver-nqeazvvs, hostId: 3 is in avoid set, skipping this and trying other available hosts
  858. 2014-03-28 23:27:44,052 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (secstorage-1:ctx-befa82ae FirstFitRoutingAllocator) Host Allocator returning 0 suitable hosts
  859. 2014-03-28 23:27:44,052 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) No suitable hosts found
  860. 2014-03-28 23:27:44,052 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) No suitable hosts found under this Cluster: 1
  861. 2014-03-28 23:27:44,054 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (secstorage-1:ctx-befa82ae) Could not find suitable Deployment Destination for this VM under any clusters, returning.
  862. 2014-03-28 23:27:44,054 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Searching resources only under specified Pod: 1
  863. 2014-03-28 23:27:44,054 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Pod: 1
  864. 2014-03-28 23:27:44,057 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) Removing from the clusterId list these clusters from avoid set: [1]
  865. 2014-03-28 23:27:44,057 DEBUG [c.c.d.FirstFitPlanner] (secstorage-1:ctx-befa82ae) No clusters found after removing disabled clusters and clusters in avoid list, returning.
  866. 2014-03-28 23:27:44,064 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: 2
  867. 2014-03-28 23:27:44,070 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  868. 2014-03-28 23:27:44,070 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total RAM: 35803627776 and RAM after applying overprovisioning: 35803627520
  869. 2014-03-28 23:27:44,070 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release cpu from host: 2, old used: 500,reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  870. 2014-03-28 23:27:44,070 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release mem from host: 2, old used: 536870912,reserved: 0, total: 35803627520; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
  871. 2014-03-28 23:27:44,073 WARN [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-befa82ae) Exception while trying to start secondary storage vm
  872. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-13-VM]Scope=interface com.cloud.dc.DataCenter; id=1
  873. at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:921)
  874. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
  875. at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
  876. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
  877. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:694)
  878. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  879. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  880. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  881. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  882. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  883. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  884. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  885. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  886. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  887. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  888. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  889. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  890. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  891. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  892. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  893. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  894. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  895. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  896. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  897. at java.lang.Thread.run(Thread.java:701)
  898. 2014-03-28 23:27:44,074 INFO [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-befa82ae) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 13, will recycle it and start a new one
  899. 2014-03-28 23:27:44,079 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) VM is already stopped: VM[SecondaryStorageVm|s-13-VM]
  900. 2014-03-28 23:27:44,084 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: 1 new host id: null host id before state transition: null
  901. 2014-03-28 23:27:44,098 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total CPU: 44800 and CPU after applying overprovisioning: 44800
  902. 2014-03-28 23:27:44,098 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) Hosts's actual total RAM: 35803627776 and RAM after applying overprovisioning: 35803627520
  903. 2014-03-28 23:27:44,099 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release cpu from host: 1, old used: 0,reserved: 0, actual total: 44800, total with overprovisioning: 44800; new used: 0,reserved:0; movedfromreserved: true,moveToReserveredfalse
  904. 2014-03-28 23:27:44,099 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-befa82ae) release mem from host: 1, old used: 0,reserved: 0, total: 35803627520; new used: 0,reserved:0; movedfromreserved: true,moveToReserveredfalse
  905. 2014-03-28 23:27:44,103 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Destroying vm VM[SecondaryStorageVm|s-13-VM]
  906. 2014-03-28 23:27:44,103 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Cleaning up NICS
  907. 2014-03-28 23:27:44,103 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Cleaning network for vm: 13
  908. 2014-03-28 23:27:44,119 DEBUG [c.c.n.g.DirectNetworkGuru] (secstorage-1:ctx-befa82ae) Deallocate network: networkId: 204, ip: 71.43.84.196
  909. 2014-03-28 23:27:44,127 DEBUG [c.c.n.g.DirectNetworkGuru] (secstorage-1:ctx-befa82ae) remove nic 48 secondary ip
  910. 2014-03-28 23:27:44,131 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Removed nic id=48
  911. 2014-03-28 23:27:44,133 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Revoving nic secondary ip entry ...
  912. 2014-03-28 23:27:44,139 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Removed nic id=49
  913. 2014-03-28 23:27:44,143 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Revoving nic secondary ip entry ...
  914. 2014-03-28 23:27:44,157 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Removed nic id=50
  915. 2014-03-28 23:27:44,158 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Revoving nic secondary ip entry ...
  916. 2014-03-28 23:27:44,166 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Removed nic id=51
  917. 2014-03-28 23:27:44,167 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-befa82ae) Revoving nic secondary ip entry ...
  918. 2014-03-28 23:27:44,168 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-befa82ae) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
  919. 2014-03-28 23:27:44,175 WARN [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-befa82ae) Unexpected exception null
  920. java.lang.NullPointerException
  921. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.disconnectVolumesFromHost(VolumeOrchestrator.java:885)
  922. at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
  923. at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:449)
  924. at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:438)
  925. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.destroySecStorageVm(SecondaryStorageManagerImpl.java:983)
  926. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:714)
  927. at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1278)
  928. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
  929. at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
  930. at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
  931. at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
  932. at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
  933. at com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
  934. at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
  935. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  936. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  937. at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  938. at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
  939. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  940. at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
  941. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
  942. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
  943. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
  944. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
  945. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  946. at java.lang.Thread.run(Thread.java:701)
  947. 2014-03-28 23:27:45,426 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-098e3187) Found 0 routers to update status.
  948. 2014-03-28 23:27:45,430 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-098e3187) Found 0 networks to update RvR status.
  949. 2014-03-28 23:27:47,852 DEBUG [c.c.a.ApiServlet] (catalina-exec-14:ctx-2883393c) ===START=== 72.189.249.168 -- GET command=listCapabilities&response=json&sessionkey=XmPAPBNX4%2FQiRYggs89NcJc7h34%3D&_=1396063667697
  950. 2014-03-28 23:27:47,861 DEBUG [c.c.a.ApiServlet] (catalina-exec-14:ctx-2883393c) ===END=== 72.189.249.168 -- GET command=listCapabilities&response=json&sessionkey=XmPAPBNX4%2FQiRYggs89NcJc7h34%3D&_=1396063667697
  951. 2014-03-28 23:27:54,606 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-8:ctx-1775243c) Ping from 1(xenserver-nmvotzdm)
  952. 2014-03-28 23:27:55,140 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-21:ctx-e526a3b5) Ping from 2(xenserver-xzcdirqf)
  953. 2014-03-28 23:27:55,285 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-54:ctx-f22f82b0) Ping from 3(xenserver-nqeazvvs)
  954. 2014-03-28 23:27:55,363 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-8:ctx-662f1f5a) Ping from 4(xenserver-sxdjiipt)
  955. 2014-03-28 23:27:55,498 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-57:ctx-1f25b0df) Seq 2-1891631106: Executing request
  956. 2014-03-28 23:27:55,580 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-57:ctx-1f25b0df) Seq 2-1891631106: Response Received:
  957. 2014-03-28 23:27:55,580 DEBUG [c.c.a.t.Request] (DirectAgent-57:ctx-1f25b0df) Seq 2-1891631106: Processing: { Ans: , MgmtId: 126274761388196, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  958. 2014-03-28 23:27:55,582 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-8:ctx-7c5cc905) Seq 3-376307714: Executing request
  959. 2014-03-28 23:27:55,671 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-8:ctx-7c5cc905) Seq 3-376307714: Response Received:
  960. 2014-03-28 23:27:55,671 DEBUG [c.c.a.t.Request] (DirectAgent-8:ctx-7c5cc905) Seq 3-376307714: Processing: { Ans: , MgmtId: 126274761388196, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  961. 2014-03-28 23:27:55,711 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-56:ctx-17ed3e6e) Seq 4-642908163: Executing request
  962. 2014-03-28 23:27:55,799 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-56:ctx-17ed3e6e) Seq 4-642908163: Response Received:
  963. 2014-03-28 23:27:55,800 DEBUG [c.c.a.t.Request] (DirectAgent-56:ctx-17ed3e6e) Seq 4-642908163: Processing: { Ans: , MgmtId: 126274761388196, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
  964. 2014-03-28 23:27:55,988 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-59:ctx-ff6e31af) Seq 1-16056324: Executing request
  965. 2014-03-28 23:27:56,164 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-59:ctx-ff6e31af) Seq 1-16056324: Response Received:
  966. 2014-03-28 23:27:56,165 DEBUG [c.c.a.t.Request] (DirectAgent-59:ctx-ff6e31af) Seq 1-16056324: Processing: { Ans: , MgmtId: 126274761388196, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}] }
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!