API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jun 14th, 2023
29
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 111.49 KB | None | 0 0
raw download clone embed print report
  1. ======== Autoscaler status: 2023-06-08 02:42:18.721909 ========
  2. Node status
  3. ---------------------------------------------------------------
  4. Healthy:
  5. 1 ray_head_default
  6. Pending:
  7. (no pending nodes)
  8. Recent failures:
  9. (no failures)
  10.  
  11. Resources
  12. ---------------------------------------------------------------
  13. Usage:
  14. 1.0/1.0 CPU
  15. 0B/4.05GiB memory
  16. 2.61MiB/2.02GiB object_store_memory
  17.  
  18. Demands:
  19. (no resource demands)
  20. 2023-06-08 02:42:18,723 INFO autoscaler.py:466 -- The autoscaler took 0.099 seconds to complete the update iteration.
  21. 2023-06-08 02:42:23,751 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  22. 2023-06-08 02:42:23,830 INFO autoscaler.py:147 -- The autoscaler took 0.088 seconds to fetch the list of non-terminated nodes.
  23. 2023-06-08 02:42:23,831 INFO autoscaler.py:423 --
  24. ======== Autoscaler status: 2023-06-08 02:42:23.831277 ========
  25. Node status
  26. ---------------------------------------------------------------
  27. Healthy:
  28. 1 ray_head_default
  29. Pending:
  30. (no pending nodes)
  31. Recent failures:
  32. (no failures)
  33.  
  34. Resources
  35. ---------------------------------------------------------------
  36. Usage:
  37. 1.0/1.0 CPU
  38. 0B/4.05GiB memory
  39. 2.61MiB/2.02GiB object_store_memory
  40.  
  41. Demands:
  42. {'CPU': 2.0}: 1+ pending tasks/actors
  43. 2023-06-08 02:42:23,832 INFO autoscaler.py:1370 -- StandardAutoscaler: Queue 1 new nodes for launch
  44. 2023-06-08 02:42:23,832 INFO node_launcher.py:166 -- BaseNodeLauncher: Got 1 nodes to launch.
  45. 2023-06-08 02:42:23,842 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?alt=json
  46. 2023-06-08 02:42:24,770 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217343845-5fd9b11cb42f8-89e719e3-730c5c30 to finish...
  47. 2023-06-08 02:42:24,771 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217343845-5fd9b11cb42f8-89e719e3-730c5c30?alt=json
  48. 2023-06-08 02:42:29,848 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217343845-5fd9b11cb42f8-89e719e3-730c5c30?alt=json
  49. 2023-06-08 02:42:34,933 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217343845-5fd9b11cb42f8-89e719e3-730c5c30?alt=json
  50. 2023-06-08 02:42:40,027 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217343845-5fd9b11cb42f8-89e719e3-730c5c30?alt=json
  51. 2023-06-08 02:42:40,123 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217343845-5fd9b11cb42f8-89e719e3-730c5c30 finished.
  52. 2023-06-08 02:42:40,123 INFO node_launcher.py:166 -- BaseNodeLauncher: Launching 1 nodes, type ray_worker_small.
  53. 2023-06-08 02:42:40,124 INFO autoscaler.py:466 -- The autoscaler took 16.381 seconds to complete the update iteration.
  54. 2023-06-08 02:42:40,124 INFO monitor.py:429 -- :event_summary:Adding 1 node(s) of type ray_worker_small.
  55. 2023-06-08 02:42:45,158 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  56. 2023-06-08 02:42:45,245 INFO autoscaler.py:147 -- The autoscaler took 0.098 seconds to fetch the list of non-terminated nodes.
  57. 2023-06-08 02:42:45,246 INFO autoscaler.py:423 --
  58. ======== Autoscaler status: 2023-06-08 02:42:45.246236 ========
  59. Node status
  60. ---------------------------------------------------------------
  61. Healthy:
  62. 1 ray_head_default
  63. Pending:
  64. 10.128.0.21: ray_worker_small, uninitialized
  65. Recent failures:
  66. (no failures)
  67.  
  68. Resources
  69. ---------------------------------------------------------------
  70. Usage:
  71. 1.0/1.0 CPU
  72. 0B/4.05GiB memory
  73. 2.61MiB/2.02GiB object_store_memory
  74.  
  75. Demands:
  76. {'CPU': 2.0}: 1+ pending tasks/actors
  77. 2023-06-08 02:42:45,247 INFO autoscaler.py:1318 -- Creating new (spawn_updater) updater thread for node ray-debug-pose-as-service-worker-1fdd7a63-compute.
  78. 2023-06-08 02:42:45,259 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  79. 2023-06-08 02:42:45,353 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-1fdd7a63-compute/setLabels?alt=json
  80. 2023-06-08 02:42:45,584 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217365355-5fd9b13137b97-f8377e5d-ff962773 to finish...
  81. 2023-06-08 02:42:45,585 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217365355-5fd9b13137b97-f8377e5d-ff962773?alt=json
  82. 2023-06-08 02:42:50,671 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217365355-5fd9b13137b97-f8377e5d-ff962773?alt=json
  83. 2023-06-08 02:42:50,766 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217365355-5fd9b13137b97-f8377e5d-ff962773 finished.
  84. 2023-06-08 02:42:50,767 INFO autoscaler.py:466 -- The autoscaler took 5.619 seconds to complete the update iteration.
  85. 2023-06-08 02:42:55,799 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  86. 2023-06-08 02:42:55,914 INFO autoscaler.py:147 -- The autoscaler took 0.122 seconds to fetch the list of non-terminated nodes.
  87. 2023-06-08 02:42:55,914 INFO autoscaler.py:423 --
  88. ======== Autoscaler status: 2023-06-08 02:42:55.914667 ========
  89. Node status
  90. ---------------------------------------------------------------
  91. Healthy:
  92. 1 ray_head_default
  93. Pending:
  94. 10.128.0.21: ray_worker_small, waiting-for-ssh
  95. Recent failures:
  96. (no failures)
  97.  
  98. Resources
  99. ---------------------------------------------------------------
  100. Usage:
  101. 1.0/1.0 CPU
  102. 0B/4.05GiB memory
  103. 2.61MiB/2.02GiB object_store_memory
  104.  
  105. Demands:
  106. {'CPU': 2.0}: 1+ pending tasks/actors
  107. 2023-06-08 02:42:55,916 INFO autoscaler.py:466 -- The autoscaler took 0.124 seconds to complete the update iteration.
  108. 2023-06-08 02:43:00,943 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  109. 2023-06-08 02:43:01,046 INFO autoscaler.py:147 -- The autoscaler took 0.111 seconds to fetch the list of non-terminated nodes.
  110. 2023-06-08 02:43:01,047 INFO autoscaler.py:423 --
  111. ======== Autoscaler status: 2023-06-08 02:43:01.047137 ========
  112. Node status
  113. ---------------------------------------------------------------
  114. Healthy:
  115. 1 ray_head_default
  116. Pending:
  117. 10.128.0.21: ray_worker_small, waiting-for-ssh
  118. Recent failures:
  119. (no failures)
  120.  
  121. Resources
  122. ---------------------------------------------------------------
  123. Usage:
  124. 1.0/1.0 CPU
  125. 0B/4.05GiB memory
  126. 2.61MiB/2.02GiB object_store_memory
  127.  
  128. Demands:
  129. {'CPU': 2.0}: 1+ pending tasks/actors
  130. 2023-06-08 02:43:01,048 INFO autoscaler.py:466 -- The autoscaler took 0.113 seconds to complete the update iteration.
  131. 2023-06-08 02:43:06,079 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  132. 2023-06-08 02:43:06,166 INFO autoscaler.py:147 -- The autoscaler took 0.094 seconds to fetch the list of non-terminated nodes.
  133. 2023-06-08 02:43:06,166 INFO autoscaler.py:423 --
  134. ======== Autoscaler status: 2023-06-08 02:43:06.166369 ========
  135. Node status
  136. ---------------------------------------------------------------
  137. Healthy:
  138. 1 ray_head_default
  139. Pending:
  140. 10.128.0.21: ray_worker_small, waiting-for-ssh
  141. Recent failures:
  142. (no failures)
  143.  
  144. Resources
  145. ---------------------------------------------------------------
  146. Usage:
  147. 1.0/1.0 CPU
  148. 0B/4.05GiB memory
  149. 2.61MiB/2.02GiB object_store_memory
  150.  
  151. Demands:
  152. {'CPU': 2.0}: 1+ pending tasks/actors
  153. 2023-06-08 02:43:06,167 INFO autoscaler.py:466 -- The autoscaler took 0.096 seconds to complete the update iteration.
  154. 2023-06-08 02:43:11,196 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  155. 2023-06-08 02:43:11,280 INFO autoscaler.py:147 -- The autoscaler took 0.091 seconds to fetch the list of non-terminated nodes.
  156. 2023-06-08 02:43:11,280 INFO autoscaler.py:423 --
  157. ======== Autoscaler status: 2023-06-08 02:43:11.280759 ========
  158. Node status
  159. ---------------------------------------------------------------
  160. Healthy:
  161. 1 ray_head_default
  162. Pending:
  163. 10.128.0.21: ray_worker_small, waiting-for-ssh
  164. Recent failures:
  165. (no failures)
  166.  
  167. Resources
  168. ---------------------------------------------------------------
  169. Usage:
  170. 1.0/1.0 CPU
  171. 0B/4.05GiB memory
  172. 2.61MiB/2.02GiB object_store_memory
  173.  
  174. Demands:
  175. {'CPU': 2.0}: 1+ pending tasks/actors
  176. 2023-06-08 02:43:11,282 INFO autoscaler.py:466 -- The autoscaler took 0.094 seconds to complete the update iteration.
  177. 2023-06-08 02:43:12,061 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  178. 2023-06-08 02:43:12,157 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-1fdd7a63-compute/setLabels?alt=json
  179. 2023-06-08 02:43:12,400 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217392160-5fd9b14ac7d15-af20c8f4-06ca8263 to finish...
  180. 2023-06-08 02:43:12,402 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217392160-5fd9b14ac7d15-af20c8f4-06ca8263?alt=json
  181. 2023-06-08 02:43:17,490 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217392160-5fd9b14ac7d15-af20c8f4-06ca8263?alt=json
  182. 2023-06-08 02:43:17,570 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217392160-5fd9b14ac7d15-af20c8f4-06ca8263 finished.
  183. 2023-06-08 02:43:17,579 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  184. 2023-06-08 02:43:17,714 INFO autoscaler.py:147 -- The autoscaler took 1.412 seconds to fetch the list of non-terminated nodes.
  185. 2023-06-08 02:43:17,714 INFO autoscaler.py:423 --
  186. ======== Autoscaler status: 2023-06-08 02:43:17.714425 ========
  187. Node status
  188. ---------------------------------------------------------------
  189. Healthy:
  190. 1 ray_head_default
  191. Pending:
  192. 10.128.0.21: ray_worker_small, syncing-files
  193. Recent failures:
  194. (no failures)
  195.  
  196. Resources
  197. ---------------------------------------------------------------
  198. Usage:
  199. 1.0/1.0 CPU
  200. 0B/4.05GiB memory
  201. 2.61MiB/2.02GiB object_store_memory
  202.  
  203. Demands:
  204. {'CPU': 2.0}: 1+ pending tasks/actors
  205. 2023-06-08 02:43:17,716 INFO autoscaler.py:466 -- The autoscaler took 1.414 seconds to complete the update iteration.
  206. 2023-06-08 02:43:18,963 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  207. 2023-06-08 02:43:19,075 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-1fdd7a63-compute/setLabels?alt=json
  208. 2023-06-08 02:43:19,292 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217399077-5fd9b15160af2-67903277-98db4988 to finish...
  209. 2023-06-08 02:43:19,293 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217399077-5fd9b15160af2-67903277-98db4988?alt=json
  210. 2023-06-08 02:43:24,374 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217399077-5fd9b15160af2-67903277-98db4988?alt=json
  211. 2023-06-08 02:43:24,457 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217399077-5fd9b15160af2-67903277-98db4988 finished.
  212. 2023-06-08 02:43:24,467 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  213. 2023-06-08 02:43:24,556 INFO autoscaler.py:147 -- The autoscaler took 1.821 seconds to fetch the list of non-terminated nodes.
  214. 2023-06-08 02:43:24,556 INFO autoscaler.py:423 --
  215. ======== Autoscaler status: 2023-06-08 02:43:24.556750 ========
  216. Node status
  217. ---------------------------------------------------------------
  218. Healthy:
  219. 1 ray_head_default
  220. Pending:
  221. 10.128.0.21: ray_worker_small, setting-up
  222. Recent failures:
  223. (no failures)
  224.  
  225. Resources
  226. ---------------------------------------------------------------
  227. Usage:
  228. 1.0/1.0 CPU
  229. 0B/4.05GiB memory
  230. 2.61MiB/2.02GiB object_store_memory
  231.  
  232. Demands:
  233. {'CPU': 2.0}: 1+ pending tasks/actors
  234. 2023-06-08 02:43:24,558 INFO autoscaler.py:466 -- The autoscaler took 1.823 seconds to complete the update iteration.
  235. 2023-06-08 02:43:29,583 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  236. 2023-06-08 02:43:29,692 INFO autoscaler.py:147 -- The autoscaler took 0.117 seconds to fetch the list of non-terminated nodes.
  237. 2023-06-08 02:43:29,693 INFO autoscaler.py:423 --
  238. ======== Autoscaler status: 2023-06-08 02:43:29.693052 ========
  239. Node status
  240. ---------------------------------------------------------------
  241. Healthy:
  242. 1 ray_head_default
  243. Pending:
  244. 10.128.0.21: ray_worker_small, setting-up
  245. Recent failures:
  246. (no failures)
  247.  
  248. Resources
  249. ---------------------------------------------------------------
  250. Usage:
  251. 1.0/1.0 CPU
  252. 0B/4.05GiB memory
  253. 2.61MiB/2.02GiB object_store_memory
  254.  
  255. Demands:
  256. {'CPU': 2.0}: 1+ pending tasks/actors
  257. 2023-06-08 02:43:29,694 INFO autoscaler.py:466 -- The autoscaler took 0.119 seconds to complete the update iteration.
  258. 2023-06-08 02:43:34,721 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  259. 2023-06-08 02:43:34,804 INFO autoscaler.py:147 -- The autoscaler took 0.09 seconds to fetch the list of non-terminated nodes.
  260. 2023-06-08 02:43:34,804 INFO autoscaler.py:423 --
  261. ======== Autoscaler status: 2023-06-08 02:43:34.804462 ========
  262. Node status
  263. ---------------------------------------------------------------
  264. Healthy:
  265. 1 ray_head_default
  266. Pending:
  267. 10.128.0.21: ray_worker_small, setting-up
  268. Recent failures:
  269. (no failures)
  270.  
  271. Resources
  272. ---------------------------------------------------------------
  273. Usage:
  274. 1.0/1.0 CPU
  275. 0B/4.05GiB memory
  276. 2.61MiB/2.02GiB object_store_memory
  277.  
  278. Demands:
  279. {'CPU': 2.0}: 1+ pending tasks/actors
  280. 2023-06-08 02:43:34,806 INFO autoscaler.py:466 -- The autoscaler took 0.092 seconds to complete the update iteration.
  281. 2023-06-08 02:43:39,835 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  282. 2023-06-08 02:43:39,933 INFO autoscaler.py:147 -- The autoscaler took 0.106 seconds to fetch the list of non-terminated nodes.
  283. 2023-06-08 02:43:39,934 INFO autoscaler.py:423 --
  284. ======== Autoscaler status: 2023-06-08 02:43:39.934015 ========
  285. Node status
  286. ---------------------------------------------------------------
  287. Healthy:
  288. 1 ray_head_default
  289. Pending:
  290. 10.128.0.21: ray_worker_small, setting-up
  291. Recent failures:
  292. (no failures)
  293.  
  294. Resources
  295. ---------------------------------------------------------------
  296. Usage:
  297. 1.0/1.0 CPU
  298. 0B/4.05GiB memory
  299. 2.61MiB/2.02GiB object_store_memory
  300.  
  301. Demands:
  302. {'CPU': 2.0}: 1+ pending tasks/actors
  303. 2023-06-08 02:43:39,935 INFO autoscaler.py:466 -- The autoscaler took 0.108 seconds to complete the update iteration.
  304. 2023-06-08 02:43:44,961 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  305. 2023-06-08 02:43:45,052 INFO autoscaler.py:147 -- The autoscaler took 0.1 seconds to fetch the list of non-terminated nodes.
  306. 2023-06-08 02:43:45,052 INFO autoscaler.py:423 --
  307. ======== Autoscaler status: 2023-06-08 02:43:45.052394 ========
  308. Node status
  309. ---------------------------------------------------------------
  310. Healthy:
  311. 1 ray_head_default
  312. Pending:
  313. 10.128.0.21: ray_worker_small, setting-up
  314. Recent failures:
  315. (no failures)
  316.  
  317. Resources
  318. ---------------------------------------------------------------
  319. Usage:
  320. 1.0/1.0 CPU
  321. 0B/4.05GiB memory
  322. 2.61MiB/2.02GiB object_store_memory
  323.  
  324. Demands:
  325. {'CPU': 2.0}: 1+ pending tasks/actors
  326. 2023-06-08 02:43:45,054 INFO autoscaler.py:466 -- The autoscaler took 0.103 seconds to complete the update iteration.
  327. 2023-06-08 02:43:50,081 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  328. 2023-06-08 02:43:50,181 INFO autoscaler.py:147 -- The autoscaler took 0.108 seconds to fetch the list of non-terminated nodes.
  329. 2023-06-08 02:43:50,182 INFO autoscaler.py:423 --
  330. ======== Autoscaler status: 2023-06-08 02:43:50.182125 ========
  331. Node status
  332. ---------------------------------------------------------------
  333. Healthy:
  334. 1 ray_head_default
  335. Pending:
  336. 10.128.0.21: ray_worker_small, setting-up
  337. Recent failures:
  338. (no failures)
  339.  
  340. Resources
  341. ---------------------------------------------------------------
  342. Usage:
  343. 1.0/1.0 CPU
  344. 0B/4.05GiB memory
  345. 2.61MiB/2.02GiB object_store_memory
  346.  
  347. Demands:
  348. {'CPU': 2.0}: 1+ pending tasks/actors
  349. 2023-06-08 02:43:50,183 INFO autoscaler.py:466 -- The autoscaler took 0.11 seconds to complete the update iteration.
  350. 2023-06-08 02:43:55,212 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  351. 2023-06-08 02:43:55,276 INFO autoscaler.py:147 -- The autoscaler took 0.071 seconds to fetch the list of non-terminated nodes.
  352. 2023-06-08 02:43:55,276 INFO autoscaler.py:423 --
  353. ======== Autoscaler status: 2023-06-08 02:43:55.276737 ========
  354. Node status
  355. ---------------------------------------------------------------
  356. Healthy:
  357. 1 ray_head_default
  358. Pending:
  359. 10.128.0.21: ray_worker_small, setting-up
  360. Recent failures:
  361. (no failures)
  362.  
  363. Resources
  364. ---------------------------------------------------------------
  365. Usage:
  366. 1.0/1.0 CPU
  367. 0B/4.05GiB memory
  368. 2.61MiB/2.02GiB object_store_memory
  369.  
  370. Demands:
  371. {'CPU': 2.0}: 1+ pending tasks/actors
  372. 2023-06-08 02:43:55,278 INFO autoscaler.py:466 -- The autoscaler took 0.073 seconds to complete the update iteration.
  373. 2023-06-08 02:44:00,306 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  374. 2023-06-08 02:44:00,398 INFO autoscaler.py:147 -- The autoscaler took 0.101 seconds to fetch the list of non-terminated nodes.
  375. 2023-06-08 02:44:00,399 INFO autoscaler.py:423 --
  376. ======== Autoscaler status: 2023-06-08 02:44:00.399003 ========
  377. Node status
  378. ---------------------------------------------------------------
  379. Healthy:
  380. 1 ray_head_default
  381. Pending:
  382. 10.128.0.21: ray_worker_small, setting-up
  383. Recent failures:
  384. (no failures)
  385.  
  386. Resources
  387. ---------------------------------------------------------------
  388. Usage:
  389. 1.0/1.0 CPU
  390. 0B/4.05GiB memory
  391. 2.61MiB/2.02GiB object_store_memory
  392.  
  393. Demands:
  394. {'CPU': 2.0}: 1+ pending tasks/actors
  395. 2023-06-08 02:44:00,400 INFO autoscaler.py:466 -- The autoscaler took 0.102 seconds to complete the update iteration.
  396. 2023-06-08 02:44:05,430 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  397. 2023-06-08 02:44:05,537 INFO autoscaler.py:147 -- The autoscaler took 0.117 seconds to fetch the list of non-terminated nodes.
  398. 2023-06-08 02:44:05,538 INFO autoscaler.py:423 --
  399. ======== Autoscaler status: 2023-06-08 02:44:05.538224 ========
  400. Node status
  401. ---------------------------------------------------------------
  402. Healthy:
  403. 1 ray_head_default
  404. Pending:
  405. 10.128.0.21: ray_worker_small, setting-up
  406. Recent failures:
  407. (no failures)
  408.  
  409. Resources
  410. ---------------------------------------------------------------
  411. Usage:
  412. 1.0/1.0 CPU
  413. 0B/4.05GiB memory
  414. 2.61MiB/2.02GiB object_store_memory
  415.  
  416. Demands:
  417. {'CPU': 2.0}: 1+ pending tasks/actors
  418. 2023-06-08 02:44:05,539 INFO autoscaler.py:466 -- The autoscaler took 0.119 seconds to complete the update iteration.
  419. 2023-06-08 02:44:10,563 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  420. 2023-06-08 02:44:10,632 INFO autoscaler.py:147 -- The autoscaler took 0.077 seconds to fetch the list of non-terminated nodes.
  421. 2023-06-08 02:44:10,632 INFO autoscaler.py:423 --
  422. ======== Autoscaler status: 2023-06-08 02:44:10.632416 ========
  423. Node status
  424. ---------------------------------------------------------------
  425. Healthy:
  426. 1 ray_head_default
  427. Pending:
  428. 10.128.0.21: ray_worker_small, setting-up
  429. Recent failures:
  430. (no failures)
  431.  
  432. Resources
  433. ---------------------------------------------------------------
  434. Usage:
  435. 1.0/1.0 CPU
  436. 0B/4.05GiB memory
  437. 2.61MiB/2.02GiB object_store_memory
  438.  
  439. Demands:
  440. {'CPU': 2.0}: 1+ pending tasks/actors
  441. 2023-06-08 02:44:10,634 INFO autoscaler.py:466 -- The autoscaler took 0.079 seconds to complete the update iteration.
  442. 2023-06-08 02:44:15,658 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  443. 2023-06-08 02:44:15,734 INFO autoscaler.py:147 -- The autoscaler took 0.084 seconds to fetch the list of non-terminated nodes.
  444. 2023-06-08 02:44:15,734 INFO autoscaler.py:423 --
  445. ======== Autoscaler status: 2023-06-08 02:44:15.734654 ========
  446. Node status
  447. ---------------------------------------------------------------
  448. Healthy:
  449. 1 ray_head_default
  450. Pending:
  451. 10.128.0.21: ray_worker_small, setting-up
  452. Recent failures:
  453. (no failures)
  454.  
  455. Resources
  456. ---------------------------------------------------------------
  457. Usage:
  458. 1.0/1.0 CPU
  459. 0B/4.05GiB memory
  460. 2.61MiB/2.02GiB object_store_memory
  461.  
  462. Demands:
  463. {'CPU': 2.0}: 1+ pending tasks/actors
  464. 2023-06-08 02:44:15,736 INFO autoscaler.py:466 -- The autoscaler took 0.085 seconds to complete the update iteration.
  465. 2023-06-08 02:44:20,779 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  466. 2023-06-08 02:44:20,865 INFO autoscaler.py:147 -- The autoscaler took 0.103 seconds to fetch the list of non-terminated nodes.
  467. 2023-06-08 02:44:20,865 INFO autoscaler.py:423 --
  468. ======== Autoscaler status: 2023-06-08 02:44:20.865563 ========
  469. Node status
  470. ---------------------------------------------------------------
  471. Healthy:
  472. 1 ray_head_default
  473. Pending:
  474. 10.128.0.21: ray_worker_small, setting-up
  475. Recent failures:
  476. (no failures)
  477.  
  478. Resources
  479. ---------------------------------------------------------------
  480. Usage:
  481. 1.0/1.0 CPU
  482. 0B/4.05GiB memory
  483. 2.61MiB/2.02GiB object_store_memory
  484.  
  485. Demands:
  486. {'CPU': 2.0}: 1+ pending tasks/actors
  487. 2023-06-08 02:44:20,867 INFO autoscaler.py:466 -- The autoscaler took 0.105 seconds to complete the update iteration.
  488. 2023-06-08 02:44:25,895 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  489. 2023-06-08 02:44:25,968 INFO autoscaler.py:147 -- The autoscaler took 0.083 seconds to fetch the list of non-terminated nodes.
  490. 2023-06-08 02:44:25,969 INFO autoscaler.py:423 --
  491. ======== Autoscaler status: 2023-06-08 02:44:25.969262 ========
  492. Node status
  493. ---------------------------------------------------------------
  494. Healthy:
  495. 1 ray_head_default
  496. Pending:
  497. 10.128.0.21: ray_worker_small, setting-up
  498. Recent failures:
  499. (no failures)
  500.  
  501. Resources
  502. ---------------------------------------------------------------
  503. Usage:
  504. 1.0/1.0 CPU
  505. 0B/4.05GiB memory
  506. 2.61MiB/2.02GiB object_store_memory
  507.  
  508. Demands:
  509. {'CPU': 2.0}: 1+ pending tasks/actors
  510. 2023-06-08 02:44:25,970 INFO autoscaler.py:466 -- The autoscaler took 0.085 seconds to complete the update iteration.
  511. 2023-06-08 02:44:30,995 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  512. 2023-06-08 02:44:31,058 INFO autoscaler.py:147 -- The autoscaler took 0.071 seconds to fetch the list of non-terminated nodes.
  513. 2023-06-08 02:44:31,059 INFO autoscaler.py:423 --
  514. ======== Autoscaler status: 2023-06-08 02:44:31.059087 ========
  515. Node status
  516. ---------------------------------------------------------------
  517. Healthy:
  518. 1 ray_head_default
  519. Pending:
  520. 10.128.0.21: ray_worker_small, setting-up
  521. Recent failures:
  522. (no failures)
  523.  
  524. Resources
  525. ---------------------------------------------------------------
  526. Usage:
  527. 1.0/1.0 CPU
  528. 0B/4.05GiB memory
  529. 2.61MiB/2.02GiB object_store_memory
  530.  
  531. Demands:
  532. {'CPU': 2.0}: 1+ pending tasks/actors
  533. 2023-06-08 02:44:31,060 INFO autoscaler.py:466 -- The autoscaler took 0.073 seconds to complete the update iteration.
  534. 2023-06-08 02:44:36,087 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  535. 2023-06-08 02:44:36,170 INFO autoscaler.py:147 -- The autoscaler took 0.091 seconds to fetch the list of non-terminated nodes.
  536. 2023-06-08 02:44:36,171 INFO autoscaler.py:423 --
  537. ======== Autoscaler status: 2023-06-08 02:44:36.171062 ========
  538. Node status
  539. ---------------------------------------------------------------
  540. Healthy:
  541. 1 ray_head_default
  542. Pending:
  543. 10.128.0.21: ray_worker_small, setting-up
  544. Recent failures:
  545. (no failures)
  546.  
  547. Resources
  548. ---------------------------------------------------------------
  549. Usage:
  550. 1.0/1.0 CPU
  551. 0B/4.05GiB memory
  552. 2.61MiB/2.02GiB object_store_memory
  553.  
  554. Demands:
  555. {'CPU': 2.0}: 1+ pending tasks/actors
  556. 2023-06-08 02:44:36,172 INFO autoscaler.py:466 -- The autoscaler took 0.093 seconds to complete the update iteration.
  557. 2023-06-08 02:44:41,200 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  558. 2023-06-08 02:44:41,285 INFO autoscaler.py:147 -- The autoscaler took 0.093 seconds to fetch the list of non-terminated nodes.
  559. 2023-06-08 02:44:41,286 INFO autoscaler.py:423 --
  560. ======== Autoscaler status: 2023-06-08 02:44:41.285875 ========
  561. Node status
  562. ---------------------------------------------------------------
  563. Healthy:
  564. 1 ray_head_default
  565. Pending:
  566. 10.128.0.21: ray_worker_small, setting-up
  567. Recent failures:
  568. (no failures)
  569.  
  570. Resources
  571. ---------------------------------------------------------------
  572. Usage:
  573. 1.0/1.0 CPU
  574. 0B/4.05GiB memory
  575. 2.61MiB/2.02GiB object_store_memory
  576.  
  577. Demands:
  578. {'CPU': 2.0}: 1+ pending tasks/actors
  579. 2023-06-08 02:44:41,287 INFO autoscaler.py:466 -- The autoscaler took 0.095 seconds to complete the update iteration.
  580. 2023-06-08 02:44:46,315 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  581. 2023-06-08 02:44:46,420 INFO autoscaler.py:147 -- The autoscaler took 0.116 seconds to fetch the list of non-terminated nodes.
  582. 2023-06-08 02:44:46,420 INFO autoscaler.py:423 --
  583. ======== Autoscaler status: 2023-06-08 02:44:46.420438 ========
  584. Node status
  585. ---------------------------------------------------------------
  586. Healthy:
  587. 1 ray_head_default
  588. Pending:
  589. 10.128.0.21: ray_worker_small, setting-up
  590. Recent failures:
  591. (no failures)
  592.  
  593. Resources
  594. ---------------------------------------------------------------
  595. Usage:
  596. 1.0/1.0 CPU
  597. 0B/4.05GiB memory
  598. 2.61MiB/2.02GiB object_store_memory
  599.  
  600. Demands:
  601. {'CPU': 2.0}: 1+ pending tasks/actors
  602. 2023-06-08 02:44:46,422 INFO autoscaler.py:466 -- The autoscaler took 0.118 seconds to complete the update iteration.
  603. 2023-06-08 02:44:51,451 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  604. 2023-06-08 02:44:51,537 INFO autoscaler.py:147 -- The autoscaler took 0.094 seconds to fetch the list of non-terminated nodes.
  605. 2023-06-08 02:44:51,538 INFO autoscaler.py:423 --
  606. ======== Autoscaler status: 2023-06-08 02:44:51.537998 ========
  607. Node status
  608. ---------------------------------------------------------------
  609. Healthy:
  610. 1 ray_head_default
  611. Pending:
  612. 10.128.0.21: ray_worker_small, setting-up
  613. Recent failures:
  614. (no failures)
  615.  
  616. Resources
  617. ---------------------------------------------------------------
  618. Usage:
  619. 1.0/1.0 CPU
  620. 0B/4.05GiB memory
  621. 2.61MiB/2.02GiB object_store_memory
  622.  
  623. Demands:
  624. {'CPU': 2.0}: 1+ pending tasks/actors
  625. 2023-06-08 02:44:51,539 INFO autoscaler.py:466 -- The autoscaler took 0.096 seconds to complete the update iteration.
  626. 2023-06-08 02:44:56,567 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  627. 2023-06-08 02:44:56,656 INFO autoscaler.py:147 -- The autoscaler took 0.097 seconds to fetch the list of non-terminated nodes.
  628. 2023-06-08 02:44:56,657 INFO autoscaler.py:423 --
  629. ======== Autoscaler status: 2023-06-08 02:44:56.657100 ========
  630. Node status
  631. ---------------------------------------------------------------
  632. Healthy:
  633. 1 ray_head_default
  634. Pending:
  635. 10.128.0.21: ray_worker_small, setting-up
  636. Recent failures:
  637. (no failures)
  638.  
  639. Resources
  640. ---------------------------------------------------------------
  641. Usage:
  642. 1.0/1.0 CPU
  643. 0B/4.05GiB memory
  644. 2.61MiB/2.02GiB object_store_memory
  645.  
  646. Demands:
  647. {'CPU': 2.0}: 1+ pending tasks/actors
  648. 2023-06-08 02:44:56,658 INFO autoscaler.py:466 -- The autoscaler took 0.099 seconds to complete the update iteration.
  649. 2023-06-08 02:45:01,686 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  650. 2023-06-08 02:45:01,771 INFO autoscaler.py:147 -- The autoscaler took 0.094 seconds to fetch the list of non-terminated nodes.
  651. 2023-06-08 02:45:01,771 INFO autoscaler.py:423 --
  652. ======== Autoscaler status: 2023-06-08 02:45:01.771861 ========
  653. Node status
  654. ---------------------------------------------------------------
  655. Healthy:
  656. 1 ray_head_default
  657. Pending:
  658. 10.128.0.21: ray_worker_small, setting-up
  659. Recent failures:
  660. (no failures)
  661.  
  662. Resources
  663. ---------------------------------------------------------------
  664. Usage:
  665. 1.0/1.0 CPU
  666. 0B/4.05GiB memory
  667. 2.61MiB/2.02GiB object_store_memory
  668.  
  669. Demands:
  670. {'CPU': 2.0}: 1+ pending tasks/actors
  671. 2023-06-08 02:45:01,773 INFO autoscaler.py:466 -- The autoscaler took 0.096 seconds to complete the update iteration.
  672. 2023-06-08 02:45:06,800 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  673. 2023-06-08 02:45:06,889 INFO autoscaler.py:147 -- The autoscaler took 0.098 seconds to fetch the list of non-terminated nodes.
  674. 2023-06-08 02:45:06,890 INFO autoscaler.py:423 --
  675. ======== Autoscaler status: 2023-06-08 02:45:06.890270 ========
  676. Node status
  677. ---------------------------------------------------------------
  678. Healthy:
  679. 1 ray_head_default
  680. Pending:
  681. 10.128.0.21: ray_worker_small, setting-up
  682. Recent failures:
  683. (no failures)
  684.  
  685. Resources
  686. ---------------------------------------------------------------
  687. Usage:
  688. 1.0/1.0 CPU
  689. 0B/4.05GiB memory
  690. 2.61MiB/2.02GiB object_store_memory
  691.  
  692. Demands:
  693. {'CPU': 2.0}: 1+ pending tasks/actors
  694. 2023-06-08 02:45:06,891 INFO autoscaler.py:466 -- The autoscaler took 0.1 seconds to complete the update iteration.
  695. 2023-06-08 02:45:11,921 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  696. 2023-06-08 02:45:11,986 INFO autoscaler.py:147 -- The autoscaler took 0.074 seconds to fetch the list of non-terminated nodes.
  697. 2023-06-08 02:45:11,987 INFO autoscaler.py:423 --
  698. ======== Autoscaler status: 2023-06-08 02:45:11.987217 ========
  699. Node status
  700. ---------------------------------------------------------------
  701. Healthy:
  702. 1 ray_head_default
  703. Pending:
  704. 10.128.0.21: ray_worker_small, setting-up
  705. Recent failures:
  706. (no failures)
  707.  
  708. Resources
  709. ---------------------------------------------------------------
  710. Usage:
  711. 1.0/1.0 CPU
  712. 0B/4.05GiB memory
  713. 2.61MiB/2.02GiB object_store_memory
  714.  
  715. Demands:
  716. {'CPU': 2.0}: 1+ pending tasks/actors
  717. 2023-06-08 02:45:11,988 INFO autoscaler.py:466 -- The autoscaler took 0.076 seconds to complete the update iteration.
  718. 2023-06-08 02:45:17,012 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  719. 2023-06-08 02:45:17,091 INFO autoscaler.py:147 -- The autoscaler took 0.087 seconds to fetch the list of non-terminated nodes.
  720. 2023-06-08 02:45:17,091 INFO autoscaler.py:423 --
  721. ======== Autoscaler status: 2023-06-08 02:45:17.091726 ========
  722. Node status
  723. ---------------------------------------------------------------
  724. Healthy:
  725. 1 ray_head_default
  726. Pending:
  727. 10.128.0.21: ray_worker_small, setting-up
  728. Recent failures:
  729. (no failures)
  730.  
  731. Resources
  732. ---------------------------------------------------------------
  733. Usage:
  734. 1.0/1.0 CPU
  735. 0B/4.05GiB memory
  736. 2.61MiB/2.02GiB object_store_memory
  737.  
  738. Demands:
  739. {'CPU': 2.0}: 1+ pending tasks/actors
  740. 2023-06-08 02:45:17,093 INFO autoscaler.py:466 -- The autoscaler took 0.089 seconds to complete the update iteration.
  741. 2023-06-08 02:45:22,122 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  742. 2023-06-08 02:45:22,200 INFO autoscaler.py:147 -- The autoscaler took 0.087 seconds to fetch the list of non-terminated nodes.
  743. 2023-06-08 02:45:22,201 INFO autoscaler.py:423 --
  744. ======== Autoscaler status: 2023-06-08 02:45:22.201070 ========
  745. Node status
  746. ---------------------------------------------------------------
  747. Healthy:
  748. 1 ray_head_default
  749. Pending:
  750. 10.128.0.21: ray_worker_small, setting-up
  751. Recent failures:
  752. (no failures)
  753.  
  754. Resources
  755. ---------------------------------------------------------------
  756. Usage:
  757. 1.0/1.0 CPU
  758. 0B/4.05GiB memory
  759. 2.61MiB/2.02GiB object_store_memory
  760.  
  761. Demands:
  762. {'CPU': 2.0}: 1+ pending tasks/actors
  763. 2023-06-08 02:45:22,202 INFO autoscaler.py:466 -- The autoscaler took 0.089 seconds to complete the update iteration.
  764. 2023-06-08 02:45:27,228 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  765. 2023-06-08 02:45:27,300 INFO autoscaler.py:147 -- The autoscaler took 0.082 seconds to fetch the list of non-terminated nodes.
  766. 2023-06-08 02:45:27,301 INFO autoscaler.py:423 --
  767. ======== Autoscaler status: 2023-06-08 02:45:27.301296 ========
  768. Node status
  769. ---------------------------------------------------------------
  770. Healthy:
  771. 1 ray_head_default
  772. Pending:
  773. 10.128.0.21: ray_worker_small, setting-up
  774. Recent failures:
  775. (no failures)
  776.  
  777. Resources
  778. ---------------------------------------------------------------
  779. Usage:
  780. 1.0/1.0 CPU
  781. 0B/4.05GiB memory
  782. 2.61MiB/2.02GiB object_store_memory
  783.  
  784. Demands:
  785. {'CPU': 2.0}: 1+ pending tasks/actors
  786. 2023-06-08 02:45:27,302 INFO autoscaler.py:466 -- The autoscaler took 0.084 seconds to complete the update iteration.
  787. 2023-06-08 02:45:32,327 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  788. 2023-06-08 02:45:32,425 INFO autoscaler.py:147 -- The autoscaler took 0.106 seconds to fetch the list of non-terminated nodes.
  789. 2023-06-08 02:45:32,425 INFO autoscaler.py:423 --
  790. ======== Autoscaler status: 2023-06-08 02:45:32.425565 ========
  791. Node status
  792. ---------------------------------------------------------------
  793. Healthy:
  794. 1 ray_head_default
  795. Pending:
  796. 10.128.0.21: ray_worker_small, setting-up
  797. Recent failures:
  798. (no failures)
  799.  
  800. Resources
  801. ---------------------------------------------------------------
  802. Usage:
  803. 1.0/1.0 CPU
  804. 0B/4.05GiB memory
  805. 2.61MiB/2.02GiB object_store_memory
  806.  
  807. Demands:
  808. {'CPU': 2.0}: 1+ pending tasks/actors
  809. 2023-06-08 02:45:32,427 INFO autoscaler.py:466 -- The autoscaler took 0.108 seconds to complete the update iteration.
  810. 2023-06-08 02:45:37,451 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  811. 2023-06-08 02:45:37,548 INFO autoscaler.py:147 -- The autoscaler took 0.105 seconds to fetch the list of non-terminated nodes.
  812. 2023-06-08 02:45:37,549 INFO autoscaler.py:423 --
  813. ======== Autoscaler status: 2023-06-08 02:45:37.549245 ========
  814. Node status
  815. ---------------------------------------------------------------
  816. Healthy:
  817. 1 ray_head_default
  818. Pending:
  819. 10.128.0.21: ray_worker_small, setting-up
  820. Recent failures:
  821. (no failures)
  822.  
  823. Resources
  824. ---------------------------------------------------------------
  825. Usage:
  826. 1.0/1.0 CPU
  827. 0B/4.05GiB memory
  828. 2.61MiB/2.02GiB object_store_memory
  829.  
  830. Demands:
  831. {'CPU': 2.0}: 1+ pending tasks/actors
  832. 2023-06-08 02:45:37,550 INFO autoscaler.py:466 -- The autoscaler took 0.107 seconds to complete the update iteration.
  833. 2023-06-08 02:45:41,287 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  834. 2023-06-08 02:45:41,370 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-1fdd7a63-compute/setLabels?alt=json
  835. 2023-06-08 02:45:41,625 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217541373-5fd9b1d914e42-ba087226-20ecd8c5 to finish...
  836. 2023-06-08 02:45:41,626 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217541373-5fd9b1d914e42-ba087226-20ecd8c5?alt=json
  837. 2023-06-08 02:45:46,710 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217541373-5fd9b1d914e42-ba087226-20ecd8c5?alt=json
  838. 2023-06-08 02:45:46,794 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217541373-5fd9b1d914e42-ba087226-20ecd8c5 finished.
  839. 2023-06-08 02:45:46,805 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  840. 2023-06-08 02:45:46,898 INFO autoscaler.py:147 -- The autoscaler took 4.327 seconds to fetch the list of non-terminated nodes.
  841. 2023-06-08 02:45:46,898 INFO autoscaler.py:423 --
  842. ======== Autoscaler status: 2023-06-08 02:45:46.898559 ========
  843. Node status
  844. ---------------------------------------------------------------
  845. Healthy:
  846. 1 ray_head_default
  847. 1 ray_worker_small
  848. Pending:
  849. (no pending nodes)
  850. Recent failures:
  851. (no failures)
  852.  
  853. Resources
  854. ---------------------------------------------------------------
  855. Usage:
  856. 3.0/3.0 CPU
  857. 0B/9.39GiB memory
  858. 2.61MiB/4.31GiB object_store_memory
  859.  
  860. Demands:
  861. {'CPU': 2.0}: 1+ pending tasks/actors
  862. 2023-06-08 02:45:46,901 INFO autoscaler.py:1370 -- StandardAutoscaler: Queue 1 new nodes for launch
  863. 2023-06-08 02:45:46,902 INFO node_launcher.py:166 -- BaseNodeLauncher: Got 1 nodes to launch.
  864. 2023-06-08 02:45:46,926 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?alt=json
  865. 2023-06-08 02:45:48,042 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217546929-5fd9b1de6149d-ed7d5cc5-4b7f5ad5 to finish...
  866. 2023-06-08 02:45:48,043 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217546929-5fd9b1de6149d-ed7d5cc5-4b7f5ad5?alt=json
  867. 2023-06-08 02:45:53,131 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217546929-5fd9b1de6149d-ed7d5cc5-4b7f5ad5?alt=json
  868. 2023-06-08 02:45:58,223 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217546929-5fd9b1de6149d-ed7d5cc5-4b7f5ad5?alt=json
  869. 2023-06-08 02:45:58,295 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217546929-5fd9b1de6149d-ed7d5cc5-4b7f5ad5 finished.
  870. 2023-06-08 02:45:58,296 INFO node_launcher.py:166 -- BaseNodeLauncher: Launching 1 nodes, type ray_worker_small.
  871. 2023-06-08 02:45:58,296 INFO autoscaler.py:466 -- The autoscaler took 15.726 seconds to complete the update iteration.
  872. 2023-06-08 02:45:58,296 INFO monitor.py:429 -- :event_summary:Resized to 3 CPUs.
  873. 2023-06-08 02:45:58,297 INFO monitor.py:429 -- :event_summary:Adding 1 node(s) of type ray_worker_small.
  874. 2023-06-08 02:46:03,324 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  875. 2023-06-08 02:46:03,435 INFO autoscaler.py:147 -- The autoscaler took 0.119 seconds to fetch the list of non-terminated nodes.
  876. 2023-06-08 02:46:03,435 INFO autoscaler.py:423 --
  877. ======== Autoscaler status: 2023-06-08 02:46:03.435530 ========
  878. Node status
  879. ---------------------------------------------------------------
  880. Healthy:
  881. 1 ray_head_default
  882. 1 ray_worker_small
  883. Pending:
  884. 10.128.0.27: ray_worker_small, uninitialized
  885. Recent failures:
  886. (no failures)
  887.  
  888. Resources
  889. ---------------------------------------------------------------
  890. Usage:
  891. 3.0/3.0 CPU
  892. 0B/9.39GiB memory
  893. 0B/4.31GiB object_store_memory
  894.  
  895. Demands:
  896. (no resource demands)
  897. 2023-06-08 02:46:03,437 INFO autoscaler.py:1318 -- Creating new (spawn_updater) updater thread for node ray-debug-pose-as-service-worker-262d4584-compute.
  898. 2023-06-08 02:46:03,447 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  899. 2023-06-08 02:46:03,532 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-262d4584-compute/setLabels?alt=json
  900. 2023-06-08 02:46:03,798 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217563536-5fd9b1ee37c20-4a7fcbbb-c6501ca3 to finish...
  901. 2023-06-08 02:46:03,799 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217563536-5fd9b1ee37c20-4a7fcbbb-c6501ca3?alt=json
  902. 2023-06-08 02:46:08,897 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217563536-5fd9b1ee37c20-4a7fcbbb-c6501ca3?alt=json
  903. 2023-06-08 02:46:08,969 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217563536-5fd9b1ee37c20-4a7fcbbb-c6501ca3 finished.
  904. 2023-06-08 02:46:08,971 INFO autoscaler.py:466 -- The autoscaler took 5.655 seconds to complete the update iteration.
  905. 2023-06-08 02:46:14,001 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  906. 2023-06-08 02:46:14,103 INFO autoscaler.py:147 -- The autoscaler took 0.112 seconds to fetch the list of non-terminated nodes.
  907. 2023-06-08 02:46:14,104 INFO autoscaler.py:423 --
  908. ======== Autoscaler status: 2023-06-08 02:46:14.104215 ========
  909. Node status
  910. ---------------------------------------------------------------
  911. Healthy:
  912. 1 ray_head_default
  913. 1 ray_worker_small
  914. Pending:
  915. 10.128.0.27: ray_worker_small, waiting-for-ssh
  916. Recent failures:
  917. (no failures)
  918.  
  919. Resources
  920. ---------------------------------------------------------------
  921. Usage:
  922. 3.0/3.0 CPU
  923. 0B/9.39GiB memory
  924. 0B/4.31GiB object_store_memory
  925.  
  926. Demands:
  927. (no resource demands)
  928. 2023-06-08 02:46:14,106 INFO autoscaler.py:466 -- The autoscaler took 0.115 seconds to complete the update iteration.
  929. 2023-06-08 02:46:19,141 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  930. 2023-06-08 02:46:19,212 INFO autoscaler.py:147 -- The autoscaler took 0.079 seconds to fetch the list of non-terminated nodes.
  931. 2023-06-08 02:46:19,212 INFO autoscaler.py:423 --
  932. ======== Autoscaler status: 2023-06-08 02:46:19.212860 ========
  933. Node status
  934. ---------------------------------------------------------------
  935. Healthy:
  936. 1 ray_head_default
  937. 1 ray_worker_small
  938. Pending:
  939. 10.128.0.27: ray_worker_small, waiting-for-ssh
  940. Recent failures:
  941. (no failures)
  942.  
  943. Resources
  944. ---------------------------------------------------------------
  945. Usage:
  946. 3.0/3.0 CPU
  947. 0B/9.39GiB memory
  948. 0B/4.31GiB object_store_memory
  949.  
  950. Demands:
  951. (no resource demands)
  952. 2023-06-08 02:46:19,217 INFO autoscaler.py:466 -- The autoscaler took 0.084 seconds to complete the update iteration.
  953. 2023-06-08 02:46:24,243 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  954. 2023-06-08 02:46:24,348 INFO autoscaler.py:147 -- The autoscaler took 0.113 seconds to fetch the list of non-terminated nodes.
  955. 2023-06-08 02:46:24,348 INFO autoscaler.py:423 --
  956. ======== Autoscaler status: 2023-06-08 02:46:24.348657 ========
  957. Node status
  958. ---------------------------------------------------------------
  959. Healthy:
  960. 1 ray_head_default
  961. 1 ray_worker_small
  962. Pending:
  963. 10.128.0.27: ray_worker_small, waiting-for-ssh
  964. Recent failures:
  965. (no failures)
  966.  
  967. Resources
  968. ---------------------------------------------------------------
  969. Usage:
  970. 3.0/3.0 CPU
  971. 0B/9.39GiB memory
  972. 0B/4.31GiB object_store_memory
  973.  
  974. Demands:
  975. (no resource demands)
  976. 2023-06-08 02:46:24,350 INFO autoscaler.py:466 -- The autoscaler took 0.116 seconds to complete the update iteration.
  977. 2023-06-08 02:46:29,376 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  978. 2023-06-08 02:46:29,469 INFO autoscaler.py:147 -- The autoscaler took 0.101 seconds to fetch the list of non-terminated nodes.
  979. 2023-06-08 02:46:29,470 INFO autoscaler.py:423 --
  980. ======== Autoscaler status: 2023-06-08 02:46:29.470220 ========
  981. Node status
  982. ---------------------------------------------------------------
  983. Healthy:
  984. 1 ray_head_default
  985. 1 ray_worker_small
  986. Pending:
  987. 10.128.0.27: ray_worker_small, waiting-for-ssh
  988. Recent failures:
  989. (no failures)
  990.  
  991. Resources
  992. ---------------------------------------------------------------
  993. Usage:
  994. 3.0/3.0 CPU
  995. 0B/9.39GiB memory
  996. 0B/4.31GiB object_store_memory
  997.  
  998. Demands:
  999. (no resource demands)
  1000. 2023-06-08 02:46:29,472 INFO autoscaler.py:466 -- The autoscaler took 0.103 seconds to complete the update iteration.
  1001. 2023-06-08 02:46:31,447 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1002. 2023-06-08 02:46:31,543 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-262d4584-compute/setLabels?alt=json
  1003. 2023-06-08 02:46:31,820 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217591545-5fd9b208edfe5-129bd347-47b56dd1 to finish...
  1004. 2023-06-08 02:46:31,822 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217591545-5fd9b208edfe5-129bd347-47b56dd1?alt=json
  1005. 2023-06-08 02:46:36,898 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217591545-5fd9b208edfe5-129bd347-47b56dd1?alt=json
  1006. 2023-06-08 02:46:36,983 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217591545-5fd9b208edfe5-129bd347-47b56dd1 finished.
  1007. 2023-06-08 02:46:36,994 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1008. 2023-06-08 02:46:37,073 INFO autoscaler.py:147 -- The autoscaler took 2.581 seconds to fetch the list of non-terminated nodes.
  1009. 2023-06-08 02:46:37,074 INFO autoscaler.py:423 --
  1010. ======== Autoscaler status: 2023-06-08 02:46:37.074336 ========
  1011. Node status
  1012. ---------------------------------------------------------------
  1013. Healthy:
  1014. 1 ray_head_default
  1015. 1 ray_worker_small
  1016. Pending:
  1017. 10.128.0.27: ray_worker_small, syncing-files
  1018. Recent failures:
  1019. (no failures)
  1020.  
  1021. Resources
  1022. ---------------------------------------------------------------
  1023. Usage:
  1024. 3.0/3.0 CPU
  1025. 0B/9.39GiB memory
  1026. 0B/4.31GiB object_store_memory
  1027.  
  1028. Demands:
  1029. (no resource demands)
  1030. 2023-06-08 02:46:37,076 INFO autoscaler.py:466 -- The autoscaler took 2.583 seconds to complete the update iteration.
  1031. 2023-06-08 02:46:37,983 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1032. 2023-06-08 02:46:38,069 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-262d4584-compute/setLabels?alt=json
  1033. 2023-06-08 02:46:38,256 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217598071-5fd9b20f2737c-f05562a5-52c56374 to finish...
  1034. 2023-06-08 02:46:38,257 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217598071-5fd9b20f2737c-f05562a5-52c56374?alt=json
  1035. 2023-06-08 02:46:43,355 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217598071-5fd9b20f2737c-f05562a5-52c56374?alt=json
  1036. 2023-06-08 02:46:43,411 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217598071-5fd9b20f2737c-f05562a5-52c56374 finished.
  1037. 2023-06-08 02:46:43,420 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1038. 2023-06-08 02:46:43,498 INFO autoscaler.py:147 -- The autoscaler took 1.401 seconds to fetch the list of non-terminated nodes.
  1039. 2023-06-08 02:46:43,498 INFO autoscaler.py:423 --
  1040. ======== Autoscaler status: 2023-06-08 02:46:43.498778 ========
  1041. Node status
  1042. ---------------------------------------------------------------
  1043. Healthy:
  1044. 1 ray_head_default
  1045. 1 ray_worker_small
  1046. Pending:
  1047. 10.128.0.27: ray_worker_small, setting-up
  1048. Recent failures:
  1049. (no failures)
  1050.  
  1051. Resources
  1052. ---------------------------------------------------------------
  1053. Usage:
  1054. 3.0/3.0 CPU
  1055. 0B/9.39GiB memory
  1056. 0B/4.31GiB object_store_memory
  1057.  
  1058. Demands:
  1059. (no resource demands)
  1060. 2023-06-08 02:46:43,500 INFO autoscaler.py:466 -- The autoscaler took 1.403 seconds to complete the update iteration.
  1061. 2023-06-08 02:46:48,544 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1062. 2023-06-08 02:46:48,624 INFO autoscaler.py:147 -- The autoscaler took 0.095 seconds to fetch the list of non-terminated nodes.
  1063. 2023-06-08 02:46:48,624 INFO autoscaler.py:423 --
  1064. ======== Autoscaler status: 2023-06-08 02:46:48.624541 ========
  1065. Node status
  1066. ---------------------------------------------------------------
  1067. Healthy:
  1068. 1 ray_head_default
  1069. 1 ray_worker_small
  1070. Pending:
  1071. 10.128.0.27: ray_worker_small, setting-up
  1072. Recent failures:
  1073. (no failures)
  1074.  
  1075. Resources
  1076. ---------------------------------------------------------------
  1077. Usage:
  1078. 3.0/3.0 CPU
  1079. 0B/9.39GiB memory
  1080. 0B/4.31GiB object_store_memory
  1081.  
  1082. Demands:
  1083. (no resource demands)
  1084. 2023-06-08 02:46:48,626 INFO autoscaler.py:466 -- The autoscaler took 0.097 seconds to complete the update iteration.
  1085. 2023-06-08 02:46:53,677 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1086. 2023-06-08 02:46:53,776 INFO autoscaler.py:147 -- The autoscaler took 0.114 seconds to fetch the list of non-terminated nodes.
  1087. 2023-06-08 02:46:53,777 INFO autoscaler.py:423 --
  1088. ======== Autoscaler status: 2023-06-08 02:46:53.777266 ========
  1089. Node status
  1090. ---------------------------------------------------------------
  1091. Healthy:
  1092. 1 ray_head_default
  1093. 1 ray_worker_small
  1094. Pending:
  1095. 10.128.0.27: ray_worker_small, setting-up
  1096. Recent failures:
  1097. (no failures)
  1098.  
  1099. Resources
  1100. ---------------------------------------------------------------
  1101. Usage:
  1102. 3.0/3.0 CPU
  1103. 0B/9.39GiB memory
  1104. 0B/4.31GiB object_store_memory
  1105.  
  1106. Demands:
  1107. (no resource demands)
  1108. 2023-06-08 02:46:53,779 INFO autoscaler.py:466 -- The autoscaler took 0.117 seconds to complete the update iteration.
  1109. 2023-06-08 02:46:58,832 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1110. 2023-06-08 02:46:58,913 INFO autoscaler.py:147 -- The autoscaler took 0.1 seconds to fetch the list of non-terminated nodes.
  1111. 2023-06-08 02:46:58,914 INFO autoscaler.py:423 --
  1112. ======== Autoscaler status: 2023-06-08 02:46:58.914415 ========
  1113. Node status
  1114. ---------------------------------------------------------------
  1115. Healthy:
  1116. 1 ray_head_default
  1117. 1 ray_worker_small
  1118. Pending:
  1119. 10.128.0.27: ray_worker_small, setting-up
  1120. Recent failures:
  1121. (no failures)
  1122.  
  1123. Resources
  1124. ---------------------------------------------------------------
  1125. Usage:
  1126. 3.0/3.0 CPU
  1127. 0B/9.39GiB memory
  1128. 0B/4.31GiB object_store_memory
  1129.  
  1130. Demands:
  1131. (no resource demands)
  1132. 2023-06-08 02:46:58,916 INFO autoscaler.py:466 -- The autoscaler took 0.102 seconds to complete the update iteration.
  1133. 2023-06-08 02:47:03,968 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1134. 2023-06-08 02:47:04,056 INFO autoscaler.py:147 -- The autoscaler took 0.104 seconds to fetch the list of non-terminated nodes.
  1135. 2023-06-08 02:47:04,056 INFO autoscaler.py:423 --
  1136. ======== Autoscaler status: 2023-06-08 02:47:04.056686 ========
  1137. Node status
  1138. ---------------------------------------------------------------
  1139. Healthy:
  1140. 1 ray_head_default
  1141. 1 ray_worker_small
  1142. Pending:
  1143. 10.128.0.27: ray_worker_small, setting-up
  1144. Recent failures:
  1145. (no failures)
  1146.  
  1147. Resources
  1148. ---------------------------------------------------------------
  1149. Usage:
  1150. 3.0/3.0 CPU
  1151. 0B/9.39GiB memory
  1152. 0B/4.31GiB object_store_memory
  1153.  
  1154. Demands:
  1155. (no resource demands)
  1156. 2023-06-08 02:47:04,058 INFO autoscaler.py:466 -- The autoscaler took 0.107 seconds to complete the update iteration.
  1157. 2023-06-08 02:47:09,112 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1158. 2023-06-08 02:47:09,184 INFO autoscaler.py:147 -- The autoscaler took 0.088 seconds to fetch the list of non-terminated nodes.
  1159. 2023-06-08 02:47:09,184 INFO autoscaler.py:423 --
  1160. ======== Autoscaler status: 2023-06-08 02:47:09.184796 ========
  1161. Node status
  1162. ---------------------------------------------------------------
  1163. Healthy:
  1164. 1 ray_head_default
  1165. 1 ray_worker_small
  1166. Pending:
  1167. 10.128.0.27: ray_worker_small, setting-up
  1168. Recent failures:
  1169. (no failures)
  1170.  
  1171. Resources
  1172. ---------------------------------------------------------------
  1173. Usage:
  1174. 3.0/3.0 CPU
  1175. 0B/9.39GiB memory
  1176. 0B/4.31GiB object_store_memory
  1177.  
  1178. Demands:
  1179. (no resource demands)
  1180. 2023-06-08 02:47:09,186 INFO autoscaler.py:466 -- The autoscaler took 0.09 seconds to complete the update iteration.
  1181. 2023-06-08 02:47:14,212 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1182. 2023-06-08 02:47:14,303 INFO autoscaler.py:147 -- The autoscaler took 0.099 seconds to fetch the list of non-terminated nodes.
  1183. 2023-06-08 02:47:14,303 INFO autoscaler.py:423 --
  1184. ======== Autoscaler status: 2023-06-08 02:47:14.303624 ========
  1185. Node status
  1186. ---------------------------------------------------------------
  1187. Healthy:
  1188. 1 ray_head_default
  1189. 1 ray_worker_small
  1190. Pending:
  1191. 10.128.0.27: ray_worker_small, setting-up
  1192. Recent failures:
  1193. (no failures)
  1194.  
  1195. Resources
  1196. ---------------------------------------------------------------
  1197. Usage:
  1198. 3.0/3.0 CPU
  1199. 0B/9.39GiB memory
  1200. 0B/4.31GiB object_store_memory
  1201.  
  1202. Demands:
  1203. (no resource demands)
  1204. 2023-06-08 02:47:14,305 INFO autoscaler.py:466 -- The autoscaler took 0.101 seconds to complete the update iteration.
  1205. 2023-06-08 02:47:19,333 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1206. 2023-06-08 02:47:19,505 INFO autoscaler.py:147 -- The autoscaler took 0.18 seconds to fetch the list of non-terminated nodes.
  1207. 2023-06-08 02:47:19,505 INFO autoscaler.py:423 --
  1208. ======== Autoscaler status: 2023-06-08 02:47:19.505736 ========
  1209. Node status
  1210. ---------------------------------------------------------------
  1211. Healthy:
  1212. 1 ray_head_default
  1213. 1 ray_worker_small
  1214. Pending:
  1215. 10.128.0.27: ray_worker_small, setting-up
  1216. Recent failures:
  1217. (no failures)
  1218.  
  1219. Resources
  1220. ---------------------------------------------------------------
  1221. Usage:
  1222. 3.0/3.0 CPU
  1223. 0B/9.39GiB memory
  1224. 0B/4.31GiB object_store_memory
  1225.  
  1226. Demands:
  1227. (no resource demands)
  1228. 2023-06-08 02:47:19,507 INFO autoscaler.py:466 -- The autoscaler took 0.183 seconds to complete the update iteration.
  1229. 2023-06-08 02:47:24,554 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1230. 2023-06-08 02:47:24,649 INFO autoscaler.py:147 -- The autoscaler took 0.11 seconds to fetch the list of non-terminated nodes.
  1231. 2023-06-08 02:47:24,650 INFO autoscaler.py:423 --
  1232. ======== Autoscaler status: 2023-06-08 02:47:24.650074 ========
  1233. Node status
  1234. ---------------------------------------------------------------
  1235. Healthy:
  1236. 1 ray_head_default
  1237. 1 ray_worker_small
  1238. Pending:
  1239. 10.128.0.27: ray_worker_small, setting-up
  1240. Recent failures:
  1241. (no failures)
  1242.  
  1243. Resources
  1244. ---------------------------------------------------------------
  1245. Usage:
  1246. 3.0/3.0 CPU
  1247. 0B/9.39GiB memory
  1248. 0B/4.31GiB object_store_memory
  1249.  
  1250. Demands:
  1251. (no resource demands)
  1252. 2023-06-08 02:47:24,652 INFO autoscaler.py:466 -- The autoscaler took 0.112 seconds to complete the update iteration.
  1253. 2023-06-08 02:47:29,705 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1254. 2023-06-08 02:47:29,802 INFO autoscaler.py:147 -- The autoscaler took 0.112 seconds to fetch the list of non-terminated nodes.
  1255. 2023-06-08 02:47:29,803 INFO autoscaler.py:423 --
  1256. ======== Autoscaler status: 2023-06-08 02:47:29.803031 ========
  1257. Node status
  1258. ---------------------------------------------------------------
  1259. Healthy:
  1260. 1 ray_head_default
  1261. 1 ray_worker_small
  1262. Pending:
  1263. 10.128.0.27: ray_worker_small, setting-up
  1264. Recent failures:
  1265. (no failures)
  1266.  
  1267. Resources
  1268. ---------------------------------------------------------------
  1269. Usage:
  1270. 3.0/3.0 CPU
  1271. 0B/9.39GiB memory
  1272. 0B/4.31GiB object_store_memory
  1273.  
  1274. Demands:
  1275. (no resource demands)
  1276. 2023-06-08 02:47:29,805 INFO autoscaler.py:466 -- The autoscaler took 0.116 seconds to complete the update iteration.
  1277. 2023-06-08 02:47:34,857 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1278. 2023-06-08 02:47:34,939 INFO autoscaler.py:147 -- The autoscaler took 0.097 seconds to fetch the list of non-terminated nodes.
  1279. 2023-06-08 02:47:34,939 INFO autoscaler.py:423 --
  1280. ======== Autoscaler status: 2023-06-08 02:47:34.939591 ========
  1281. Node status
  1282. ---------------------------------------------------------------
  1283. Healthy:
  1284. 1 ray_head_default
  1285. 1 ray_worker_small
  1286. Pending:
  1287. 10.128.0.27: ray_worker_small, setting-up
  1288. Recent failures:
  1289. (no failures)
  1290.  
  1291. Resources
  1292. ---------------------------------------------------------------
  1293. Usage:
  1294. 3.0/3.0 CPU
  1295. 0B/9.39GiB memory
  1296. 0B/4.31GiB object_store_memory
  1297.  
  1298. Demands:
  1299. (no resource demands)
  1300. 2023-06-08 02:47:34,942 INFO autoscaler.py:466 -- The autoscaler took 0.099 seconds to complete the update iteration.
  1301. 2023-06-08 02:47:39,992 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1302. 2023-06-08 02:47:40,043 INFO autoscaler.py:147 -- The autoscaler took 0.069 seconds to fetch the list of non-terminated nodes.
  1303. 2023-06-08 02:47:40,044 INFO autoscaler.py:423 --
  1304. ======== Autoscaler status: 2023-06-08 02:47:40.044259 ========
  1305. Node status
  1306. ---------------------------------------------------------------
  1307. Healthy:
  1308. 1 ray_head_default
  1309. 1 ray_worker_small
  1310. Pending:
  1311. 10.128.0.27: ray_worker_small, setting-up
  1312. Recent failures:
  1313. (no failures)
  1314.  
  1315. Resources
  1316. ---------------------------------------------------------------
  1317. Usage:
  1318. 3.0/3.0 CPU
  1319. 0B/9.39GiB memory
  1320. 0B/4.31GiB object_store_memory
  1321.  
  1322. Demands:
  1323. (no resource demands)
  1324. 2023-06-08 02:47:40,046 INFO autoscaler.py:466 -- The autoscaler took 0.072 seconds to complete the update iteration.
  1325. 2023-06-08 02:47:45,072 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1326. 2023-06-08 02:47:45,166 INFO autoscaler.py:147 -- The autoscaler took 0.103 seconds to fetch the list of non-terminated nodes.
  1327. 2023-06-08 02:47:45,167 INFO autoscaler.py:423 --
  1328. ======== Autoscaler status: 2023-06-08 02:47:45.167468 ========
  1329. Node status
  1330. ---------------------------------------------------------------
  1331. Healthy:
  1332. 1 ray_head_default
  1333. 1 ray_worker_small
  1334. Pending:
  1335. 10.128.0.27: ray_worker_small, setting-up
  1336. Recent failures:
  1337. (no failures)
  1338.  
  1339. Resources
  1340. ---------------------------------------------------------------
  1341. Usage:
  1342. 3.0/3.0 CPU
  1343. 0B/9.39GiB memory
  1344. 0B/4.31GiB object_store_memory
  1345.  
  1346. Demands:
  1347. (no resource demands)
  1348. 2023-06-08 02:47:45,170 INFO autoscaler.py:466 -- The autoscaler took 0.106 seconds to complete the update iteration.
  1349. 2023-06-08 02:47:50,197 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1350. 2023-06-08 02:47:50,280 INFO autoscaler.py:147 -- The autoscaler took 0.092 seconds to fetch the list of non-terminated nodes.
  1351. 2023-06-08 02:47:50,281 INFO autoscaler.py:423 --
  1352. ======== Autoscaler status: 2023-06-08 02:47:50.281293 ========
  1353. Node status
  1354. ---------------------------------------------------------------
  1355. Healthy:
  1356. 1 ray_head_default
  1357. 1 ray_worker_small
  1358. Pending:
  1359. 10.128.0.27: ray_worker_small, setting-up
  1360. Recent failures:
  1361. (no failures)
  1362.  
  1363. Resources
  1364. ---------------------------------------------------------------
  1365. Usage:
  1366. 3.0/3.0 CPU
  1367. 0B/9.39GiB memory
  1368. 0B/4.31GiB object_store_memory
  1369.  
  1370. Demands:
  1371. (no resource demands)
  1372. 2023-06-08 02:47:50,283 INFO autoscaler.py:466 -- The autoscaler took 0.095 seconds to complete the update iteration.
  1373. 2023-06-08 02:47:55,313 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1374. 2023-06-08 02:47:55,402 INFO autoscaler.py:147 -- The autoscaler took 0.098 seconds to fetch the list of non-terminated nodes.
  1375. 2023-06-08 02:47:55,403 INFO autoscaler.py:423 --
  1376. ======== Autoscaler status: 2023-06-08 02:47:55.402970 ========
  1377. Node status
  1378. ---------------------------------------------------------------
  1379. Healthy:
  1380. 1 ray_head_default
  1381. 1 ray_worker_small
  1382. Pending:
  1383. 10.128.0.27: ray_worker_small, setting-up
  1384. Recent failures:
  1385. (no failures)
  1386.  
  1387. Resources
  1388. ---------------------------------------------------------------
  1389. Usage:
  1390. 3.0/3.0 CPU
  1391. 0B/9.39GiB memory
  1392. 0B/4.31GiB object_store_memory
  1393.  
  1394. Demands:
  1395. (no resource demands)
  1396. 2023-06-08 02:47:55,405 INFO autoscaler.py:466 -- The autoscaler took 0.101 seconds to complete the update iteration.
  1397. 2023-06-08 02:48:00,433 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1398. 2023-06-08 02:48:00,542 INFO autoscaler.py:147 -- The autoscaler took 0.117 seconds to fetch the list of non-terminated nodes.
  1399. 2023-06-08 02:48:00,543 INFO autoscaler.py:423 --
  1400. ======== Autoscaler status: 2023-06-08 02:48:00.543171 ========
  1401. Node status
  1402. ---------------------------------------------------------------
  1403. Healthy:
  1404. 1 ray_head_default
  1405. 1 ray_worker_small
  1406. Pending:
  1407. 10.128.0.27: ray_worker_small, setting-up
  1408. Recent failures:
  1409. (no failures)
  1410.  
  1411. Resources
  1412. ---------------------------------------------------------------
  1413. Usage:
  1414. 3.0/3.0 CPU
  1415. 0B/9.39GiB memory
  1416. 0B/4.31GiB object_store_memory
  1417.  
  1418. Demands:
  1419. (no resource demands)
  1420. 2023-06-08 02:48:00,546 INFO autoscaler.py:466 -- The autoscaler took 0.12 seconds to complete the update iteration.
  1421. 2023-06-08 02:48:05,572 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1422. 2023-06-08 02:48:05,666 INFO autoscaler.py:147 -- The autoscaler took 0.103 seconds to fetch the list of non-terminated nodes.
  1423. 2023-06-08 02:48:05,666 INFO autoscaler.py:423 --
  1424. ======== Autoscaler status: 2023-06-08 02:48:05.666767 ========
  1425. Node status
  1426. ---------------------------------------------------------------
  1427. Healthy:
  1428. 1 ray_head_default
  1429. 1 ray_worker_small
  1430. Pending:
  1431. 10.128.0.27: ray_worker_small, setting-up
  1432. Recent failures:
  1433. (no failures)
  1434.  
  1435. Resources
  1436. ---------------------------------------------------------------
  1437. Usage:
  1438. 3.0/3.0 CPU
  1439. 0B/9.39GiB memory
  1440. 0B/4.31GiB object_store_memory
  1441.  
  1442. Demands:
  1443. (no resource demands)
  1444. 2023-06-08 02:48:05,669 INFO autoscaler.py:466 -- The autoscaler took 0.106 seconds to complete the update iteration.
  1445. 2023-06-08 02:48:10,696 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1446. 2023-06-08 02:48:10,781 INFO autoscaler.py:147 -- The autoscaler took 0.092 seconds to fetch the list of non-terminated nodes.
  1447. 2023-06-08 02:48:10,781 INFO autoscaler.py:423 --
  1448. ======== Autoscaler status: 2023-06-08 02:48:10.781569 ========
  1449. Node status
  1450. ---------------------------------------------------------------
  1451. Healthy:
  1452. 1 ray_head_default
  1453. 1 ray_worker_small
  1454. Pending:
  1455. 10.128.0.27: ray_worker_small, setting-up
  1456. Recent failures:
  1457. (no failures)
  1458.  
  1459. Resources
  1460. ---------------------------------------------------------------
  1461. Usage:
  1462. 3.0/3.0 CPU
  1463. 0B/9.39GiB memory
  1464. 0B/4.31GiB object_store_memory
  1465.  
  1466. Demands:
  1467. (no resource demands)
  1468. 2023-06-08 02:48:10,783 INFO autoscaler.py:466 -- The autoscaler took 0.094 seconds to complete the update iteration.
  1469. 2023-06-08 02:48:15,812 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1470. 2023-06-08 02:48:15,889 INFO autoscaler.py:147 -- The autoscaler took 0.085 seconds to fetch the list of non-terminated nodes.
  1471. 2023-06-08 02:48:15,890 INFO autoscaler.py:423 --
  1472. ======== Autoscaler status: 2023-06-08 02:48:15.889910 ========
  1473. Node status
  1474. ---------------------------------------------------------------
  1475. Healthy:
  1476. 1 ray_head_default
  1477. 1 ray_worker_small
  1478. Pending:
  1479. 10.128.0.27: ray_worker_small, setting-up
  1480. Recent failures:
  1481. (no failures)
  1482.  
  1483. Resources
  1484. ---------------------------------------------------------------
  1485. Usage:
  1486. 3.0/3.0 CPU
  1487. 0B/9.39GiB memory
  1488. 0B/4.31GiB object_store_memory
  1489.  
  1490. Demands:
  1491. (no resource demands)
  1492. 2023-06-08 02:48:15,892 INFO autoscaler.py:466 -- The autoscaler took 0.088 seconds to complete the update iteration.
  1493. 2023-06-08 02:48:20,926 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1494. 2023-06-08 02:48:21,022 INFO autoscaler.py:147 -- The autoscaler took 0.111 seconds to fetch the list of non-terminated nodes.
  1495. 2023-06-08 02:48:21,023 INFO autoscaler.py:423 --
  1496. ======== Autoscaler status: 2023-06-08 02:48:21.023431 ========
  1497. Node status
  1498. ---------------------------------------------------------------
  1499. Healthy:
  1500. 1 ray_head_default
  1501. 1 ray_worker_small
  1502. Pending:
  1503. 10.128.0.27: ray_worker_small, setting-up
  1504. Recent failures:
  1505. (no failures)
  1506.  
  1507. Resources
  1508. ---------------------------------------------------------------
  1509. Usage:
  1510. 3.0/3.0 CPU
  1511. 0B/9.39GiB memory
  1512. 0B/4.31GiB object_store_memory
  1513.  
  1514. Demands:
  1515. (no resource demands)
  1516. 2023-06-08 02:48:21,025 INFO autoscaler.py:466 -- The autoscaler took 0.114 seconds to complete the update iteration.
  1517. 2023-06-08 02:48:26,052 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1518. 2023-06-08 02:48:26,112 INFO autoscaler.py:147 -- The autoscaler took 0.069 seconds to fetch the list of non-terminated nodes.
  1519. 2023-06-08 02:48:26,113 INFO autoscaler.py:423 --
  1520. ======== Autoscaler status: 2023-06-08 02:48:26.113461 ========
  1521. Node status
  1522. ---------------------------------------------------------------
  1523. Healthy:
  1524. 1 ray_head_default
  1525. 1 ray_worker_small
  1526. Pending:
  1527. 10.128.0.27: ray_worker_small, setting-up
  1528. Recent failures:
  1529. (no failures)
  1530.  
  1531. Resources
  1532. ---------------------------------------------------------------
  1533. Usage:
  1534. 3.0/3.0 CPU
  1535. 0B/9.39GiB memory
  1536. 0B/4.31GiB object_store_memory
  1537.  
  1538. Demands:
  1539. (no resource demands)
  1540. 2023-06-08 02:48:26,116 INFO autoscaler.py:466 -- The autoscaler took 0.072 seconds to complete the update iteration.
  1541. 2023-06-08 02:48:31,143 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1542. 2023-06-08 02:48:31,245 INFO autoscaler.py:147 -- The autoscaler took 0.111 seconds to fetch the list of non-terminated nodes.
  1543. 2023-06-08 02:48:31,246 INFO autoscaler.py:423 --
  1544. ======== Autoscaler status: 2023-06-08 02:48:31.246023 ========
  1545. Node status
  1546. ---------------------------------------------------------------
  1547. Healthy:
  1548. 1 ray_head_default
  1549. 1 ray_worker_small
  1550. Pending:
  1551. 10.128.0.27: ray_worker_small, setting-up
  1552. Recent failures:
  1553. (no failures)
  1554.  
  1555. Resources
  1556. ---------------------------------------------------------------
  1557. Usage:
  1558. 3.0/3.0 CPU
  1559. 0B/9.39GiB memory
  1560. 0B/4.31GiB object_store_memory
  1561.  
  1562. Demands:
  1563. (no resource demands)
  1564. 2023-06-08 02:48:31,247 INFO autoscaler.py:466 -- The autoscaler took 0.113 seconds to complete the update iteration.
  1565. 2023-06-08 02:48:36,276 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1566. 2023-06-08 02:48:36,365 INFO autoscaler.py:147 -- The autoscaler took 0.097 seconds to fetch the list of non-terminated nodes.
  1567. 2023-06-08 02:48:36,365 INFO autoscaler.py:423 --
  1568. ======== Autoscaler status: 2023-06-08 02:48:36.365494 ========
  1569. Node status
  1570. ---------------------------------------------------------------
  1571. Healthy:
  1572. 1 ray_head_default
  1573. 1 ray_worker_small
  1574. Pending:
  1575. 10.128.0.27: ray_worker_small, setting-up
  1576. Recent failures:
  1577. (no failures)
  1578.  
  1579. Resources
  1580. ---------------------------------------------------------------
  1581. Usage:
  1582. 3.0/3.0 CPU
  1583. 0B/9.39GiB memory
  1584. 0B/4.31GiB object_store_memory
  1585.  
  1586. Demands:
  1587. (no resource demands)
  1588. 2023-06-08 02:48:36,367 INFO autoscaler.py:466 -- The autoscaler took 0.099 seconds to complete the update iteration.
  1589. 2023-06-08 02:48:41,392 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1590. 2023-06-08 02:48:41,482 INFO autoscaler.py:147 -- The autoscaler took 0.099 seconds to fetch the list of non-terminated nodes.
  1591. 2023-06-08 02:48:41,482 INFO autoscaler.py:423 --
  1592. ======== Autoscaler status: 2023-06-08 02:48:41.482578 ========
  1593. Node status
  1594. ---------------------------------------------------------------
  1595. Healthy:
  1596. 1 ray_head_default
  1597. 1 ray_worker_small
  1598. Pending:
  1599. 10.128.0.27: ray_worker_small, setting-up
  1600. Recent failures:
  1601. (no failures)
  1602.  
  1603. Resources
  1604. ---------------------------------------------------------------
  1605. Usage:
  1606. 3.0/3.0 CPU
  1607. 0B/9.39GiB memory
  1608. 0B/4.31GiB object_store_memory
  1609.  
  1610. Demands:
  1611. (no resource demands)
  1612. 2023-06-08 02:48:41,484 INFO autoscaler.py:466 -- The autoscaler took 0.101 seconds to complete the update iteration.
  1613. 2023-06-08 02:48:46,511 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1614. 2023-06-08 02:48:46,599 INFO autoscaler.py:147 -- The autoscaler took 0.096 seconds to fetch the list of non-terminated nodes.
  1615. 2023-06-08 02:48:46,600 INFO autoscaler.py:423 --
  1616. ======== Autoscaler status: 2023-06-08 02:48:46.599931 ========
  1617. Node status
  1618. ---------------------------------------------------------------
  1619. Healthy:
  1620. 1 ray_head_default
  1621. 1 ray_worker_small
  1622. Pending:
  1623. 10.128.0.27: ray_worker_small, setting-up
  1624. Recent failures:
  1625. (no failures)
  1626.  
  1627. Resources
  1628. ---------------------------------------------------------------
  1629. Usage:
  1630. 3.0/3.0 CPU
  1631. 0B/9.39GiB memory
  1632. 0B/4.31GiB object_store_memory
  1633.  
  1634. Demands:
  1635. (no resource demands)
  1636. 2023-06-08 02:48:46,601 INFO autoscaler.py:466 -- The autoscaler took 0.098 seconds to complete the update iteration.
  1637. 2023-06-08 02:48:51,628 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1638. 2023-06-08 02:48:51,703 INFO autoscaler.py:147 -- The autoscaler took 0.083 seconds to fetch the list of non-terminated nodes.
  1639. 2023-06-08 02:48:51,703 INFO autoscaler.py:423 --
  1640. ======== Autoscaler status: 2023-06-08 02:48:51.703913 ========
  1641. Node status
  1642. ---------------------------------------------------------------
  1643. Healthy:
  1644. 1 ray_head_default
  1645. 1 ray_worker_small
  1646. Pending:
  1647. 10.128.0.27: ray_worker_small, setting-up
  1648. Recent failures:
  1649. (no failures)
  1650.  
  1651. Resources
  1652. ---------------------------------------------------------------
  1653. Usage:
  1654. 3.0/3.0 CPU
  1655. 0B/9.39GiB memory
  1656. 0B/4.31GiB object_store_memory
  1657.  
  1658. Demands:
  1659. (no resource demands)
  1660. 2023-06-08 02:48:51,705 INFO autoscaler.py:466 -- The autoscaler took 0.085 seconds to complete the update iteration.
  1661. 2023-06-08 02:48:56,732 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1662. 2023-06-08 02:48:56,818 INFO autoscaler.py:147 -- The autoscaler took 0.094 seconds to fetch the list of non-terminated nodes.
  1663. 2023-06-08 02:48:56,819 INFO autoscaler.py:423 --
  1664. ======== Autoscaler status: 2023-06-08 02:48:56.819346 ========
  1665. Node status
  1666. ---------------------------------------------------------------
  1667. Healthy:
  1668. 1 ray_head_default
  1669. 1 ray_worker_small
  1670. Pending:
  1671. 10.128.0.27: ray_worker_small, setting-up
  1672. Recent failures:
  1673. (no failures)
  1674.  
  1675. Resources
  1676. ---------------------------------------------------------------
  1677. Usage:
  1678. 3.0/3.0 CPU
  1679. 0B/9.39GiB memory
  1680. 0B/4.31GiB object_store_memory
  1681.  
  1682. Demands:
  1683. (no resource demands)
  1684. 2023-06-08 02:48:56,821 INFO autoscaler.py:466 -- The autoscaler took 0.097 seconds to complete the update iteration.
  1685. 2023-06-08 02:48:58,855 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1686. 2023-06-08 02:48:58,970 INFO discovery.py:873 -- URL being requested: POST https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-262d4584-compute/setLabels?alt=json
  1687. 2023-06-08 02:48:59,177 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217738973-5fd9b295871cb-00603d26-31cd74c3 to finish...
  1688. 2023-06-08 02:48:59,178 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217738973-5fd9b295871cb-00603d26-31cd74c3?alt=json
  1689. 2023-06-08 02:49:04,273 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217738973-5fd9b295871cb-00603d26-31cd74c3?alt=json
  1690. 2023-06-08 02:49:04,360 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217738973-5fd9b295871cb-00603d26-31cd74c3 finished.
  1691. 2023-06-08 02:49:04,369 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1692. 2023-06-08 02:49:04,459 INFO autoscaler.py:147 -- The autoscaler took 2.62 seconds to fetch the list of non-terminated nodes.
  1693. 2023-06-08 02:49:04,460 INFO autoscaler.py:423 --
  1694. ======== Autoscaler status: 2023-06-08 02:49:04.460006 ========
  1695. Node status
  1696. ---------------------------------------------------------------
  1697. Healthy:
  1698. 1 ray_head_default
  1699. 2 ray_worker_small
  1700. Pending:
  1701. (no pending nodes)
  1702. Recent failures:
  1703. (no failures)
  1704.  
  1705. Resources
  1706. ---------------------------------------------------------------
  1707. Usage:
  1708. 3.0/5.0 CPU
  1709. 0B/14.74GiB memory
  1710. 0B/6.61GiB object_store_memory
  1711.  
  1712. Demands:
  1713. (no resource demands)
  1714. 2023-06-08 02:49:04,462 INFO autoscaler.py:466 -- The autoscaler took 2.623 seconds to complete the update iteration.
  1715. 2023-06-08 02:49:04,462 INFO monitor.py:429 -- :event_summary:Resized to 5 CPUs.
  1716. 2023-06-08 02:49:09,490 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1717. 2023-06-08 02:49:09,561 INFO autoscaler.py:147 -- The autoscaler took 0.078 seconds to fetch the list of non-terminated nodes.
  1718. 2023-06-08 02:49:09,561 INFO autoscaler.py:423 --
  1719. ======== Autoscaler status: 2023-06-08 02:49:09.561416 ========
  1720. Node status
  1721. ---------------------------------------------------------------
  1722. Healthy:
  1723. 1 ray_head_default
  1724. 2 ray_worker_small
  1725. Pending:
  1726. (no pending nodes)
  1727. Recent failures:
  1728. (no failures)
  1729.  
  1730. Resources
  1731. ---------------------------------------------------------------
  1732. Usage:
  1733. 1.0/5.0 CPU
  1734. 0B/14.74GiB memory
  1735. 0B/6.61GiB object_store_memory
  1736.  
  1737. Demands:
  1738. (no resource demands)
  1739. 2023-06-08 02:49:09,563 INFO autoscaler.py:466 -- The autoscaler took 0.08 seconds to complete the update iteration.
  1740. 2023-06-08 02:49:14,594 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1741. 2023-06-08 02:49:14,693 INFO autoscaler.py:147 -- The autoscaler took 0.108 seconds to fetch the list of non-terminated nodes.
  1742. 2023-06-08 02:49:14,693 INFO autoscaler.py:423 --
  1743. ======== Autoscaler status: 2023-06-08 02:49:14.693645 ========
  1744. Node status
  1745. ---------------------------------------------------------------
  1746. Healthy:
  1747. 1 ray_head_default
  1748. 2 ray_worker_small
  1749. Pending:
  1750. (no pending nodes)
  1751. Recent failures:
  1752. (no failures)
  1753.  
  1754. Resources
  1755. ---------------------------------------------------------------
  1756. Usage:
  1757. 1.0/5.0 CPU
  1758. 0B/14.74GiB memory
  1759. 0B/6.61GiB object_store_memory
  1760.  
  1761. Demands:
  1762. (no resource demands)
  1763. 2023-06-08 02:49:14,696 INFO autoscaler.py:466 -- The autoscaler took 0.111 seconds to complete the update iteration.
  1764. 2023-06-08 02:49:19,720 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1765. 2023-06-08 02:49:19,820 INFO autoscaler.py:147 -- The autoscaler took 0.109 seconds to fetch the list of non-terminated nodes.
  1766. 2023-06-08 02:49:19,820 INFO autoscaler.py:423 --
  1767. ======== Autoscaler status: 2023-06-08 02:49:19.820887 ========
  1768. Node status
  1769. ---------------------------------------------------------------
  1770. Healthy:
  1771. 1 ray_head_default
  1772. 2 ray_worker_small
  1773. Pending:
  1774. (no pending nodes)
  1775. Recent failures:
  1776. (no failures)
  1777.  
  1778. Resources
  1779. ---------------------------------------------------------------
  1780. Usage:
  1781. 1.0/5.0 CPU
  1782. 0B/14.74GiB memory
  1783. 0B/6.61GiB object_store_memory
  1784.  
  1785. Demands:
  1786. (no resource demands)
  1787. 2023-06-08 02:49:19,823 INFO autoscaler.py:466 -- The autoscaler took 0.112 seconds to complete the update iteration.
  1788. 2023-06-08 02:49:24,851 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1789. 2023-06-08 02:49:24,948 INFO autoscaler.py:147 -- The autoscaler took 0.106 seconds to fetch the list of non-terminated nodes.
  1790. 2023-06-08 02:49:24,949 INFO autoscaler.py:423 --
  1791. ======== Autoscaler status: 2023-06-08 02:49:24.949301 ========
  1792. Node status
  1793. ---------------------------------------------------------------
  1794. Healthy:
  1795. 1 ray_head_default
  1796. 2 ray_worker_small
  1797. Pending:
  1798. (no pending nodes)
  1799. Recent failures:
  1800. (no failures)
  1801.  
  1802. Resources
  1803. ---------------------------------------------------------------
  1804. Usage:
  1805. 1.0/5.0 CPU
  1806. 0B/14.74GiB memory
  1807. 0B/6.61GiB object_store_memory
  1808.  
  1809. Demands:
  1810. (no resource demands)
  1811. 2023-06-08 02:49:24,951 INFO autoscaler.py:466 -- The autoscaler took 0.108 seconds to complete the update iteration.
  1812. 2023-06-08 02:49:29,976 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1813. 2023-06-08 02:49:30,097 INFO autoscaler.py:147 -- The autoscaler took 0.129 seconds to fetch the list of non-terminated nodes.
  1814. 2023-06-08 02:49:30,097 INFO autoscaler.py:423 --
  1815. ======== Autoscaler status: 2023-06-08 02:49:30.097551 ========
  1816. Node status
  1817. ---------------------------------------------------------------
  1818. Healthy:
  1819. 1 ray_head_default
  1820. 2 ray_worker_small
  1821. Pending:
  1822. (no pending nodes)
  1823. Recent failures:
  1824. (no failures)
  1825.  
  1826. Resources
  1827. ---------------------------------------------------------------
  1828. Usage:
  1829. 1.0/5.0 CPU
  1830. 0B/14.74GiB memory
  1831. 0B/6.61GiB object_store_memory
  1832.  
  1833. Demands:
  1834. (no resource demands)
  1835. 2023-06-08 02:49:30,099 INFO autoscaler.py:466 -- The autoscaler took 0.132 seconds to complete the update iteration.
  1836. 2023-06-08 02:49:35,129 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1837. 2023-06-08 02:49:35,245 INFO autoscaler.py:147 -- The autoscaler took 0.125 seconds to fetch the list of non-terminated nodes.
  1838. 2023-06-08 02:49:35,245 INFO autoscaler.py:423 --
  1839. ======== Autoscaler status: 2023-06-08 02:49:35.245672 ========
  1840. Node status
  1841. ---------------------------------------------------------------
  1842. Healthy:
  1843. 1 ray_head_default
  1844. 2 ray_worker_small
  1845. Pending:
  1846. (no pending nodes)
  1847. Recent failures:
  1848. (no failures)
  1849.  
  1850. Resources
  1851. ---------------------------------------------------------------
  1852. Usage:
  1853. 1.0/5.0 CPU
  1854. 0B/14.74GiB memory
  1855. 0B/6.61GiB object_store_memory
  1856.  
  1857. Demands:
  1858. (no resource demands)
  1859. 2023-06-08 02:49:35,247 INFO autoscaler.py:466 -- The autoscaler took 0.127 seconds to complete the update iteration.
  1860. 2023-06-08 02:49:40,271 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1861. 2023-06-08 02:49:40,374 INFO autoscaler.py:147 -- The autoscaler took 0.111 seconds to fetch the list of non-terminated nodes.
  1862. 2023-06-08 02:49:40,374 INFO autoscaler.py:423 --
  1863. ======== Autoscaler status: 2023-06-08 02:49:40.374913 ========
  1864. Node status
  1865. ---------------------------------------------------------------
  1866. Healthy:
  1867. 1 ray_head_default
  1868. 2 ray_worker_small
  1869. Pending:
  1870. (no pending nodes)
  1871. Recent failures:
  1872. (no failures)
  1873.  
  1874. Resources
  1875. ---------------------------------------------------------------
  1876. Usage:
  1877. 1.0/5.0 CPU
  1878. 0B/14.74GiB memory
  1879. 0B/6.61GiB object_store_memory
  1880.  
  1881. Demands:
  1882. (no resource demands)
  1883. 2023-06-08 02:49:40,376 INFO autoscaler.py:466 -- The autoscaler took 0.114 seconds to complete the update iteration.
  1884. 2023-06-08 02:49:45,406 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1885. 2023-06-08 02:49:45,505 INFO autoscaler.py:147 -- The autoscaler took 0.11 seconds to fetch the list of non-terminated nodes.
  1886. 2023-06-08 02:49:45,506 INFO autoscaler.py:423 --
  1887. ======== Autoscaler status: 2023-06-08 02:49:45.506266 ========
  1888. Node status
  1889. ---------------------------------------------------------------
  1890. Healthy:
  1891. 1 ray_head_default
  1892. 2 ray_worker_small
  1893. Pending:
  1894. (no pending nodes)
  1895. Recent failures:
  1896. (no failures)
  1897.  
  1898. Resources
  1899. ---------------------------------------------------------------
  1900. Usage:
  1901. 1.0/5.0 CPU
  1902. 0B/14.74GiB memory
  1903. 0B/6.61GiB object_store_memory
  1904.  
  1905. Demands:
  1906. (no resource demands)
  1907. 2023-06-08 02:49:45,508 INFO autoscaler.py:466 -- The autoscaler took 0.113 seconds to complete the update iteration.
  1908. 2023-06-08 02:49:50,538 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1909. 2023-06-08 02:49:50,614 INFO autoscaler.py:147 -- The autoscaler took 0.084 seconds to fetch the list of non-terminated nodes.
  1910. 2023-06-08 02:49:50,614 INFO autoscaler.py:423 --
  1911. ======== Autoscaler status: 2023-06-08 02:49:50.614886 ========
  1912. Node status
  1913. ---------------------------------------------------------------
  1914. Healthy:
  1915. 1 ray_head_default
  1916. 2 ray_worker_small
  1917. Pending:
  1918. (no pending nodes)
  1919. Recent failures:
  1920. (no failures)
  1921.  
  1922. Resources
  1923. ---------------------------------------------------------------
  1924. Usage:
  1925. 1.0/5.0 CPU
  1926. 0B/14.74GiB memory
  1927. 0B/6.61GiB object_store_memory
  1928.  
  1929. Demands:
  1930. (no resource demands)
  1931. 2023-06-08 02:49:50,617 INFO autoscaler.py:466 -- The autoscaler took 0.088 seconds to complete the update iteration.
  1932. 2023-06-08 02:49:55,648 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1933. 2023-06-08 02:49:55,728 INFO autoscaler.py:147 -- The autoscaler took 0.088 seconds to fetch the list of non-terminated nodes.
  1934. 2023-06-08 02:49:55,729 INFO autoscaler.py:423 --
  1935. ======== Autoscaler status: 2023-06-08 02:49:55.728907 ========
  1936. Node status
  1937. ---------------------------------------------------------------
  1938. Healthy:
  1939. 1 ray_head_default
  1940. 2 ray_worker_small
  1941. Pending:
  1942. (no pending nodes)
  1943. Recent failures:
  1944. (no failures)
  1945.  
  1946. Resources
  1947. ---------------------------------------------------------------
  1948. Usage:
  1949. 1.0/5.0 CPU
  1950. 0B/14.74GiB memory
  1951. 0B/6.61GiB object_store_memory
  1952.  
  1953. Demands:
  1954. (no resource demands)
  1955. 2023-06-08 02:49:55,731 INFO autoscaler.py:466 -- The autoscaler took 0.091 seconds to complete the update iteration.
  1956. 2023-06-08 02:50:00,758 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1957. 2023-06-08 02:50:00,851 INFO autoscaler.py:147 -- The autoscaler took 0.101 seconds to fetch the list of non-terminated nodes.
  1958. 2023-06-08 02:50:00,851 INFO autoscaler.py:423 --
  1959. ======== Autoscaler status: 2023-06-08 02:50:00.851476 ========
  1960. Node status
  1961. ---------------------------------------------------------------
  1962. Healthy:
  1963. 1 ray_head_default
  1964. 2 ray_worker_small
  1965. Pending:
  1966. (no pending nodes)
  1967. Recent failures:
  1968. (no failures)
  1969.  
  1970. Resources
  1971. ---------------------------------------------------------------
  1972. Usage:
  1973. 1.0/5.0 CPU
  1974. 0B/14.74GiB memory
  1975. 0B/6.61GiB object_store_memory
  1976.  
  1977. Demands:
  1978. (no resource demands)
  1979. 2023-06-08 02:50:00,853 INFO autoscaler.py:466 -- The autoscaler took 0.104 seconds to complete the update iteration.
  1980. 2023-06-08 02:50:05,881 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  1981. 2023-06-08 02:50:05,991 INFO autoscaler.py:147 -- The autoscaler took 0.118 seconds to fetch the list of non-terminated nodes.
  1982. 2023-06-08 02:50:05,991 INFO autoscaler.py:423 --
  1983. ======== Autoscaler status: 2023-06-08 02:50:05.991876 ========
  1984. Node status
  1985. ---------------------------------------------------------------
  1986. Healthy:
  1987. 1 ray_head_default
  1988. 2 ray_worker_small
  1989. Pending:
  1990. (no pending nodes)
  1991. Recent failures:
  1992. (no failures)
  1993.  
  1994. Resources
  1995. ---------------------------------------------------------------
  1996. Usage:
  1997. 1.0/5.0 CPU
  1998. 0B/14.74GiB memory
  1999. 0B/6.61GiB object_store_memory
  2000.  
  2001. Demands:
  2002. (no resource demands)
  2003. 2023-06-08 02:50:05,993 INFO autoscaler.py:590 -- StandardAutoscaler: Terminating the node with id ray-debug-pose-as-service-worker-262d4584-compute and ip 10.128.0.27. (idle)
  2004. 2023-06-08 02:50:05,993 INFO autoscaler.py:544 -- Node last used: Thu Jun 8 02:49:01 2023.
  2005. 2023-06-08 02:50:05,993 INFO autoscaler.py:590 -- StandardAutoscaler: Terminating the node with id ray-debug-pose-as-service-worker-1fdd7a63-compute and ip 10.128.0.21. (idle)
  2006. 2023-06-08 02:50:05,993 INFO autoscaler.py:544 -- Node last used: Thu Jun 8 02:49:01 2023.
  2007. 2023-06-08 02:50:05,993 INFO autoscaler.py:678 -- Draining 2 raylet(s).
  2008. 2023-06-08 02:50:05,995 INFO node_provider.py:172 -- NodeProvider: ray-debug-pose-as-service-worker-262d4584-compute: Terminating node
  2009. 2023-06-08 02:50:06,018 INFO discovery.py:873 -- URL being requested: DELETE https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-262d4584-compute?alt=json
  2010. 2023-06-08 02:50:06,411 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e to finish...
  2011. 2023-06-08 02:50:06,413 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2012. 2023-06-08 02:50:11,494 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2013. 2023-06-08 02:50:16,567 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2014. 2023-06-08 02:50:21,644 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2015. 2023-06-08 02:50:26,706 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2016. 2023-06-08 02:50:31,797 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2017. 2023-06-08 02:50:36,887 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2018. 2023-06-08 02:50:41,970 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2019. 2023-06-08 02:50:47,079 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2020. 2023-06-08 02:50:52,150 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2021. 2023-06-08 02:50:57,266 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e?alt=json
  2022. 2023-06-08 02:50:57,348 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217806021-5fd9b2d5781d4-556ea2fe-8904509e finished.
  2023. 2023-06-08 02:50:57,348 INFO node_provider.py:172 -- NodeProvider: ray-debug-pose-as-service-worker-1fdd7a63-compute: Terminating node
  2024. 2023-06-08 02:50:57,358 INFO discovery.py:873 -- URL being requested: DELETE https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances/ray-debug-pose-as-service-worker-1fdd7a63-compute?alt=json
  2025. 2023-06-08 02:50:57,813 INFO node.py:311 -- wait_for_compute_zone_operation: Waiting for operation operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff to finish...
  2026. 2023-06-08 02:50:57,815 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2027. 2023-06-08 02:51:02,923 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2028. 2023-06-08 02:51:07,991 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2029. 2023-06-08 02:51:13,074 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2030. 2023-06-08 02:51:18,142 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2031. 2023-06-08 02:51:23,212 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2032. 2023-06-08 02:51:28,287 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2033. 2023-06-08 02:51:33,390 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2034. 2023-06-08 02:51:38,468 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2035. 2023-06-08 02:51:43,554 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2036. 2023-06-08 02:51:48,664 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2037. 2023-06-08 02:51:53,768 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/operations/operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff?alt=json
  2038. 2023-06-08 02:51:53,855 INFO node.py:330 -- wait_for_compute_zone_operation: Operation operation-1686217857360-5fd9b3066e3ae-5a272e5b-7fe16bff finished.
  2039. 2023-06-08 02:51:53,856 INFO autoscaler.py:466 -- The autoscaler took 107.983 seconds to complete the update iteration.
  2040. 2023-06-08 02:51:53,856 INFO monitor.py:429 -- :event_summary:Removing 2 nodes of type ray_worker_small (idle).
  2041. 2023-06-08 02:51:58,883 INFO discovery.py:873 -- URL being requested: GET https://compute.googleapis.com/compute/v1/projects/pose-as-service/zones/us-central1-a/instances?filter=%28%28status+%3D+PROVISIONING%29+OR+%28status+%3D+RUNNING%29+OR+%28status+%3D+STAGING%29%29+AND+%28labels.ray-cluster-name+%3D+debug-pose-as-service%29&alt=json
  2042. 2023-06-08 02:51:58,985 INFO autoscaler.py:147 -- The autoscaler took 0.11 seconds to fetch the list of non-terminated nodes.
  2043. 2023-06-08 02:51:58,985 INFO load_metrics.py:161 -- LoadMetrics: Removed ip: 10.128.0.27.
  2044. 2023-06-08 02:51:58,985 INFO load_metrics.py:161 -- LoadMetrics: Removed ip: 10.128.0.21.
  2045. 2023-06-08 02:51:58,985 INFO load_metrics.py:164 -- LoadMetrics: Removed 2 stale ip mappings: {'10.128.0.27', '10.128.0.21'} not in {'10.128.0.19'}
  2046. 2023-06-08 02:51:58,985 INFO autoscaler.py:423 --
  2047. ======== Autoscaler status: 2023-06-08 02:51:58.985797 ========
  2048. Node status
  2049. ---------------------------------------------------------------
  2050. Healthy:
  2051. 1 ray_head_default
  2052. Pending:
  2053. (no pending nodes)
  2054. Recent failures:
  2055. (no failures)
  2056.  
  2057. Resources
  2058. ---------------------------------------------------------------
  2059. Usage:
  2060. 1.0/1.0 CPU
  2061. 0B/4.05GiB memory
  2062. 0B/2.02GiB object_store_memory
  2063.  
  2064. Demands:
  2065. (no resource demands)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!