API tools faq
paste
Guest User

raylet.out

a guest
Apr 29th, 2022
177
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 35.24 KB | None | 0 0
raw download clone embed print report
  1. [2022-04-30 14:40:13,578 I 13292 13512] (raylet.exe) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
  2. [2022-04-30 14:40:13,580 I 13292 13512] (raylet.exe) store_runner.cc:32: Allowing the Plasma store to use up to 8.22634GB of memory.
  3. [2022-04-30 14:40:13,580 I 13292 13512] (raylet.exe) store_runner.cc:48: Starting object store with directory C:\Users\harsh\AppData\Local\Temp, fallback C:\Users\harsh\AppData\Local\Temp\ray, and huge page support disabled
  4. [2022-04-30 14:40:13,582 I 13292 16268] (raylet.exe) store.cc:546: ========== Plasma store: =================
  5. Current usage: 0 / 8.22634 GB
  6. - num bytes created total: 0
  7. 0 pending objects of total size 0MB
  8. - objects spillable: 0
  9. - bytes spillable: 0
  10. - objects unsealed: 0
  11. - bytes unsealed: 0
  12. - objects in use: 0
  13. - bytes in use: 0
  14. - objects evictable: 0
  15. - bytes evictable: 0
  16.  
  17. - objects created by worker: 0
  18. - bytes created by worker: 0
  19. - objects restored: 0
  20. - bytes restored: 0
  21. - objects received: 0
  22. - bytes received: 0
  23. - objects errored: 0
  24. - bytes errored: 0
  25.  
  26. [2022-04-30 14:40:13,595 I 13292 13512] (raylet.exe) grpc_server.cc:105: ObjectManager server started, listening on port 50696.
  27. [2022-04-30 14:40:13,606 I 13292 13512] (raylet.exe) node_manager.cc:329: Initializing NodeManager with ID e0f8ba6be143948050509a614680afc5af42df37b975aa18c8483869
  28. [2022-04-30 14:40:13,608 I 13292 13512] (raylet.exe) grpc_server.cc:105: NodeManager server started, listening on port 50698.
  29. [2022-04-30 14:40:13,670 I 13292 5940] (raylet.exe) agent_manager.cc:88: Monitor agent process with pid 14932, register timeout 30000ms.
  30. [2022-04-30 14:40:13,674 I 13292 13512] (raylet.exe) raylet.cc:111: Raylet of id, e0f8ba6be143948050509a614680afc5af42df37b975aa18c8483869 started. Raylet consists of node_manager and object_manager. node_manager address: 127.0.0.1:50698 object_manager address: 127.0.0.1:50696 hostname: 127.0.0.1
  31. [2022-04-30 14:40:13,679 I 13292 13512] (raylet.exe) node_manager.cc:539: [state-dump] Event stats:
  32. [state-dump]
  33. [state-dump]
  34. [state-dump] Global stats: 19 total (9 active)
  35. [state-dump] Queueing time: mean = 9.132 ms, max = 93.713 ms, min = 9.300 us, total = 173.508 ms
  36. [state-dump] Execution time: mean = 5.410 ms, total = 102.791 ms
  37. [state-dump] Event stats:
  38. [state-dump] PeriodicalRunner.RunFnPeriodically - 7 total (1 active, 1 running), CPU time: mean = 531.114 us, total = 3.718 ms
  39. [state-dump] NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  40. [state-dump] NodeManager.deadline_timer.record_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  41. [state-dump] NodeManager.deadline_timer.flush_free_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  42. [state-dump] NodeManagerService.grpc_server.RequestResourceReport - 1 total (0 active), CPU time: mean = 98.800 us, total = 98.800 us
  43. [state-dump] UNKNOWN - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  44. [state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  45. [state-dump] NodeInfoGcsService.grpc_client.GetInternalConfig - 1 total (0 active), CPU time: mean = 97.826 ms, total = 97.826 ms
  46. [state-dump] HeartbeatInfoGcsService.grpc_client.ReportHeartbeat - 1 total (0 active), CPU time: mean = 6.200 us, total = 6.200 us
  47. [state-dump] RayletWorkerPool.deadline_timer.kill_idle_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  48. [state-dump] GcsClient.deadline_timer.check_gcs_service_address - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  49. [state-dump] NodeInfoGcsService.grpc_client.RegisterNode - 1 total (0 active), CPU time: mean = 1.142 ms, total = 1.142 ms
  50. [state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  51. [state-dump]
  52. [state-dump] NodeManager:
  53. [state-dump] InitialConfigResources: {object_store_memory: 7.661379 GiB}, {GPU: 1.000000}, {node:127.0.0.1: 1.000000}, {CPU: 16.000000}, {memory: 15.322758 GiB}
  54. [state-dump] ClusterTaskManager:
  55. [state-dump] ========== Node: e0f8ba6be143948050509a614680afc5af42df37b975aa18c8483869 =================
  56. [state-dump] Infeasible queue length: 0
  57. [state-dump] Schedule queue length: 0
  58. [state-dump] Dispatch queue length: 0
  59. [state-dump] num_waiting_for_resource: 0
  60. [state-dump] num_waiting_for_plasma_memory: 0
  61. [state-dump] num_waiting_for_remote_node_resources: 0
  62. [state-dump] num_worker_not_started_by_job_config_not_exist: 0
  63. [state-dump] num_worker_not_started_by_registration_timeout: 0
  64. [state-dump] num_worker_not_started_by_process_rate_limit: 0
  65. [state-dump] num_tasks_waiting_for_workers: 0
  66. [state-dump] num_cancelled_tasks: 0
  67. [state-dump] cluster_resource_scheduler state:
  68. [state-dump] Local id: -1720304899554234539 Local resources: {
  69. [state-dump] CPU: ([160000]:[160000])
  70. [state-dump] MEM: ([164526858240000]:[164526858240000])
  71. [state-dump] GPU: ([10000]:[10000])
  72. [state-dump] OBJECT_STORE_MEM: ([82263429120000]:[82263429120000])
  73. [state-dump] node:127.0.0.1:([10000]:[10000])
  74. [state-dump] }
  75. [state-dump] node id: -1720304899554234539 {
  76. [state-dump] CPU: (160000:160000)
  77. [state-dump] MEM: (164526858240000:164526858240000)
  78. [state-dump] GPU: (10000:10000)
  79. [state-dump] OBJECT_STORE_MEM: (82263429120000:82263429120000)
  80. [state-dump] node:127.0.0.1:(10000:10000)
  81. [state-dump] }
  82. [state-dump]
  83. [state-dump] Waiting tasks size: 0
  84. [state-dump] Number of executing tasks: 0
  85. [state-dump] Number of pinned task arguments: 0
  86. [state-dump] Resource usage {
  87. [state-dump] }
  88. [state-dump] Running tasks by scheduling class:
  89. [state-dump] ==================================================
  90. [state-dump]
  91. [state-dump] ClusterResources:
  92. [state-dump] LocalObjectManager:
  93. [state-dump] - num pinned objects: 0
  94. [state-dump] - pinned objects size: 0
  95. [state-dump] - num objects pending restore: 0
  96. [state-dump] - num objects pending spill: 0
  97. [state-dump] - num bytes pending spill: 0
  98. [state-dump] - cumulative spill requests: 0
  99. [state-dump] - cumulative restore requests: 0
  100. [state-dump]
  101. [state-dump] ObjectManager:
  102. [state-dump] - num local objects: 0
  103. [state-dump] - num unfulfilled push requests: 0
  104. [state-dump] - num pull requests: 0
  105. [state-dump] - num chunks received total: 0
  106. [state-dump] - num chunks received failed (all): 0
  107. [state-dump] - num chunks received failed / cancelled: 0
  108. [state-dump] - num chunks received failed / plasma error: 0
  109. [state-dump] Event stats:
  110. [state-dump] Global stats: 0 total (0 active)
  111. [state-dump] Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
  112. [state-dump] Execution time: mean = -nan(ind) s, total = 0.000 s
  113. [state-dump] Event stats:
  114. [state-dump] PushManager:
  115. [state-dump] - num pushes in flight: 0
  116. [state-dump] - num chunks in flight: 0
  117. [state-dump] - num chunks remaining: 0
  118. [state-dump] - max chunks allowed: 409
  119. [state-dump] OwnershipBasedObjectDirectory:
  120. [state-dump] - num listeners: 0
  121. [state-dump] - cumulative location updates: 0
  122. [state-dump] - num location updates per second: 0.000
  123. [state-dump] - num location lookups per second: 0.000
  124. [state-dump] - num locations added per second: 0.000
  125. [state-dump] - num locations removed per second: 0.000
  126. [state-dump] BufferPool:
  127. [state-dump] - create buffer state map size: 0
  128. [state-dump] PullManager:
  129. [state-dump] - num bytes available for pulled objects: 8226342912
  130. [state-dump] - num bytes being pulled (all): 0
  131. [state-dump] - num bytes being pulled / pinned: 0
  132. [state-dump] - num get request bundles: 0
  133. [state-dump] - num wait request bundles: 0
  134. [state-dump] - num task request bundles: 0
  135. [state-dump] - first get request bundle: N/A
  136. [state-dump] - first wait request bundle: N/A
  137. [state-dump] - first task request bundle: N/A
  138. [state-dump] - num objects queued: 0
  139. [state-dump] - num objects actively pulled (all): 0
  140. [state-dump] - num objects actively pulled / pinned: 0
  141. [state-dump] - num bundles being pulled: 0
  142. [state-dump] - num pull retries: 0
  143. [state-dump] - max timeout seconds: 0
  144. [state-dump] - max timeout request is already processed. No entry.
  145. [state-dump]
  146. [state-dump] WorkerPool:
  147. [state-dump] - registered jobs: 0
  148. [state-dump] - process_failed_job_config_missing: 0
  149. [state-dump] - process_failed_rate_limited: 0
  150. [state-dump] - process_failed_pending_registration: 0
  151. [state-dump] - process_failed_runtime_env_setup_failed: 0
  152. [state-dump] - num PYTHON workers: 0
  153. [state-dump] - num PYTHON drivers: 0
  154. [state-dump] - num object spill callbacks queued: 0
  155. [state-dump] - num object restore queued: 0
  156. [state-dump] - num util functions queued: 0
  157. [state-dump] - num idle workers: 0
  158. [state-dump] [runtime env manager] ID to URIs table:
  159. [state-dump] [runtime env manager] URIs reference table:
  160. [state-dump] TaskDependencyManager:
  161. [state-dump] - task deps map size: 0
  162. [state-dump] - get req map size: 0
  163. [state-dump] - wait req map size: 0
  164. [state-dump] - local objects map size: 0
  165. [state-dump] WaitManager:
  166. [state-dump] - num active wait requests: 0
  167. [state-dump] Subscriber:
  168. [state-dump] Channel WORKER_OBJECT_EVICTION
  169. [state-dump] - cumulative subscribe requests: 0
  170. [state-dump] - cumulative unsubscribe requests: 0
  171. [state-dump] - active subscribed publishers: 0
  172. [state-dump] - cumulative published messages: 0
  173. [state-dump] - cumulative processed messages: 0
  174. [state-dump] Channel WORKER_REF_REMOVED_CHANNEL
  175. [state-dump] - cumulative subscribe requests: 0
  176. [state-dump] - cumulative unsubscribe requests: 0
  177. [state-dump] - active subscribed publishers: 0
  178. [state-dump] - cumulative published messages: 0
  179. [state-dump] - cumulative processed messages: 0
  180. [state-dump] Channel WORKER_OBJECT_LOCATIONS_CHANNEL
  181. [state-dump] - cumulative subscribe requests: 0
  182. [state-dump] - cumulative unsubscribe requests: 0
  183. [state-dump] - active subscribed publishers: 0
  184. [state-dump] - cumulative published messages: 0
  185. [state-dump] - cumulative processed messages: 0
  186. [state-dump] num async plasma notifications: 0
  187. [state-dump] Remote node managers:
  188. [state-dump] Event stats:
  189. [state-dump] Global stats: 19 total (9 active)
  190. [state-dump] Queueing time: mean = 9.132 ms, max = 93.713 ms, min = 9.300 us, total = 173.508 ms
  191. [state-dump] Execution time: mean = 5.410 ms, total = 102.791 ms
  192. [state-dump] Event stats:
  193. [state-dump] PeriodicalRunner.RunFnPeriodically - 7 total (1 active, 1 running), CPU time: mean = 531.114 us, total = 3.718 ms
  194. [state-dump] NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  195. [state-dump] NodeManager.deadline_timer.record_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  196. [state-dump] NodeManager.deadline_timer.flush_free_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  197. [state-dump] NodeManagerService.grpc_server.RequestResourceReport - 1 total (0 active), CPU time: mean = 98.800 us, total = 98.800 us
  198. [state-dump] UNKNOWN - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  199. [state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  200. [state-dump] NodeInfoGcsService.grpc_client.GetInternalConfig - 1 total (0 active), CPU time: mean = 97.826 ms, total = 97.826 ms
  201. [state-dump] HeartbeatInfoGcsService.grpc_client.ReportHeartbeat - 1 total (0 active), CPU time: mean = 6.200 us, total = 6.200 us
  202. [state-dump] RayletWorkerPool.deadline_timer.kill_idle_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  203. [state-dump] GcsClient.deadline_timer.check_gcs_service_address - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  204. [state-dump] NodeInfoGcsService.grpc_client.RegisterNode - 1 total (0 active), CPU time: mean = 1.142 ms, total = 1.142 ms
  205. [state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
  206. [state-dump] DebugString() time ms: 0
  207. [state-dump]
  208. [state-dump]
  209. [2022-04-30 14:40:13,680 I 13292 13512] (raylet.exe) accessor.cc:625: Received notification for node id = e0f8ba6be143948050509a614680afc5af42df37b975aa18c8483869, IsAlive = 1
  210. [2022-04-30 14:40:13,971 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 2064, the token 0
  211. [2022-04-30 14:40:13,975 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 13768, the token 1
  212. [2022-04-30 14:40:13,980 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 15516, the token 2
  213. [2022-04-30 14:40:13,984 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 3040, the token 3
  214. [2022-04-30 14:40:13,990 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 9616, the token 4
  215. [2022-04-30 14:40:13,996 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 9388, the token 5
  216. [2022-04-30 14:40:14,003 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 16004, the token 6
  217. [2022-04-30 14:40:14,010 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 14956, the token 7
  218. [2022-04-30 14:40:14,018 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 6980, the token 8
  219. [2022-04-30 14:40:14,025 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 5956, the token 9
  220. [2022-04-30 14:40:14,033 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 3624, the token 10
  221. [2022-04-30 14:40:14,046 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 4468, the token 11
  222. [2022-04-30 14:40:14,058 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 11608, the token 12
  223. [2022-04-30 14:40:14,070 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 16348, the token 13
  224. [2022-04-30 14:40:14,089 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 2208, the token 14
  225. [2022-04-30 14:40:14,101 I 13292 13512] (raylet.exe) worker_pool.cc:485: Started worker process of 1 worker(s) with pid 14340, the token 15
  226. [2022-04-30 14:40:24,100 I 13292 16268] (raylet.exe) object_store.cc:35: Object store current usage 8e-09 / 8.22634 GB.
  227. [2022-04-30 14:40:25,061 I 13292 13512] (raylet.exe) node_manager.cc:580: New job has started. Job id 01000000 Driver pid 10900 is dead: 0 driver address: 127.0.0.1
  228. [2022-04-30 14:40:27,961 W 13292 9692] (raylet.exe) metric_exporter.cc:207: [1] Export metrics to agent failed: GrpcUnknown: RPC Error message: Method not found!; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
  229. [2022-04-30 14:40:27,961 I 13292 13512] (raylet.exe) agent_manager.cc:40: HandleRegisterAgent, ip: 127.0.0.1, port: 56809, pid: 16324
  230. [2022-04-30 14:40:33,337 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 359ec6ce30d3ca2dffffffffffffffffffffffff0100000001000000, aborting
  231. [2022-04-30 14:40:33,524 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000001000000, aborting
  232. [2022-04-30 14:40:33,541 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000001000000, aborting
  233. [2022-04-30 14:40:33,548 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 2751d69548dba956ffffffffffffffffffffffff0100000001000000, aborting
  234. [2022-04-30 14:40:33,560 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000001000000, aborting
  235. [2022-04-30 14:40:33,567 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 80e22aed7718a125ffffffffffffffffffffffff0100000001000000, aborting
  236. [2022-04-30 14:40:33,567 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 16310a0f0a45af5cffffffffffffffffffffffff0100000001000000, aborting
  237. [2022-04-30 14:40:33,573 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 85748392bcd969ccffffffffffffffffffffffff0100000001000000, aborting
  238. [2022-04-30 14:40:33,573 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object e0dc174c83599034ffffffffffffffffffffffff0100000001000000, aborting
  239. [2022-04-30 14:40:33,584 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 8849b62d89cb30f9ffffffffffffffffffffffff0100000001000000, aborting
  240. [2022-04-30 14:40:33,590 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 32d950ec0ccf9d2affffffffffffffffffffffff0100000001000000, aborting
  241. [2022-04-30 14:40:33,592 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c2668a65bda616c1ffffffffffffffffffffffff0100000001000000, aborting
  242. [2022-04-30 14:40:33,600 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object f91b78d7db9a6593ffffffffffffffffffffffff0100000001000000, aborting
  243. [2022-04-30 14:40:33,603 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object f4402ec78d3a2607ffffffffffffffffffffffff0100000001000000, aborting
  244. [2022-04-30 14:40:33,604 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 71b133a11e1c461cffffffffffffffffffffffff0100000001000000, aborting
  245. [2022-04-30 14:40:33,622 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object d695f922effe6d99ffffffffffffffffffffffff0100000001000000, aborting
  246. [2022-04-30 14:40:33,632 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c8ef45ccd0112571ffffffffffffffffffffffff0100000001000000, aborting
  247. [2022-04-30 14:40:33,641 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 1e8ff6d236132784ffffffffffffffffffffffff0100000001000000, aborting
  248. [2022-04-30 14:40:33,664 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 359ec6ce30d3ca2dffffffffffffffffffffffff0100000001000000, aborting
  249. [2022-04-30 14:40:33,720 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 82891771158d68c1ffffffffffffffffffffffff0100000001000000, aborting
  250. [2022-04-30 14:40:33,741 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object a631fe8d231813bfffffffffffffffffffffffff0100000001000000, aborting
  251. [2022-04-30 14:40:33,742 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000003000000, aborting
  252. [2022-04-30 14:40:33,749 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object cae5e964086715a4ffffffffffffffffffffffff0100000001000000, aborting
  253. [2022-04-30 14:40:33,751 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c54e76759b2a0c10ffffffffffffffffffffffff0100000001000000, aborting
  254. [2022-04-30 14:40:33,752 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 347cc60e0bb3da74ffffffffffffffffffffffff0100000001000000, aborting
  255. [2022-04-30 14:40:33,756 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 18b2ad3c688fb947ffffffffffffffffffffffff0100000001000000, aborting
  256. [2022-04-30 14:40:33,758 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 3d3e27c54ed1f5cfffffffffffffffffffffffff0100000001000000, aborting
  257. [2022-04-30 14:40:33,759 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 1e360ffa862f8fe3ffffffffffffffffffffffff0100000001000000, aborting
  258. [2022-04-30 14:40:33,762 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 239c2f70c73fbf73ffffffffffffffffffffffff0100000001000000, aborting
  259. [2022-04-30 14:40:33,782 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c76a79b2875a7251ffffffffffffffffffffffff0100000001000000, aborting
  260. [2022-04-30 14:40:33,782 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object dc746dc61b2c1923ffffffffffffffffffffffff0100000001000000, aborting
  261. [2022-04-30 14:40:33,788 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 79cc316456d39201ffffffffffffffffffffffff0100000001000000, aborting
  262. [2022-04-30 14:40:33,804 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object a02c24b8b7fc0a31ffffffffffffffffffffffff0100000001000000, aborting
  263. [2022-04-30 14:40:33,824 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 5d4b8d1788f12d2dffffffffffffffffffffffff0100000001000000, aborting
  264. [2022-04-30 14:40:33,824 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 88543757a8df6d2fffffffffffffffffffffffff0100000001000000, aborting
  265. [2022-04-30 14:40:33,832 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object bcb4fef46b376cafffffffffffffffffffffffff0100000001000000, aborting
  266. [2022-04-30 14:40:33,833 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 04a86d731af48e35ffffffffffffffffffffffff0100000001000000, aborting
  267. [2022-04-30 14:40:33,855 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object f2de7ac0316578aaffffffffffffffffffffffff0100000001000000, aborting
  268. [2022-04-30 14:40:33,874 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 4f4ef6205ce35f90ffffffffffffffffffffffff0100000001000000, aborting
  269. [2022-04-30 14:40:33,877 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 64ac0404a8f0916fffffffffffffffffffffffff0100000001000000, aborting
  270. [2022-04-30 14:40:33,881 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 5168ff79929289e3ffffffffffffffffffffffff0100000001000000, aborting
  271. [2022-04-30 14:40:33,900 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object ae46b8beecd25f3affffffffffffffffffffffff0100000001000000, aborting
  272. [2022-04-30 14:40:33,910 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 17ed96eaf1aa4b2affffffffffffffffffffffff0100000001000000, aborting
  273. [2022-04-30 14:40:33,930 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c7528efcb2fd36edffffffffffffffffffffffff0100000001000000, aborting
  274. [2022-04-30 14:40:33,930 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000007000000, aborting
  275. [2022-04-30 14:40:33,935 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object b5f40f7c7d38fc79ffffffffffffffffffffffff0100000001000000, aborting
  276. [2022-04-30 14:40:33,938 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object ec502c4fdc3aeab0ffffffffffffffffffffffff0100000001000000, aborting
  277. [2022-04-30 14:40:33,942 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 261bd10b0466d7e8ffffffffffffffffffffffff0100000001000000, aborting
  278. [2022-04-30 14:40:33,942 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 465c0fb8d6cb3cdcffffffffffffffffffffffff0100000001000000, aborting
  279. [2022-04-30 14:40:33,942 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000002000000, aborting
  280. [2022-04-30 14:40:33,945 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 7109b8141612f944ffffffffffffffffffffffff0100000001000000, aborting
  281. [2022-04-30 14:40:33,948 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 0cb7b64917b5af44ffffffffffffffffffffffff0100000001000000, aborting
  282. [2022-04-30 14:40:33,958 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object cf9aed5eec5a308bffffffffffffffffffffffff0100000001000000, aborting
  283. [2022-04-30 14:40:33,964 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 9a0afb4ce5b46f16ffffffffffffffffffffffff0100000001000000, aborting
  284. [2022-04-30 14:40:33,967 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 8c4854248414f633ffffffffffffffffffffffff0100000001000000, aborting
  285. [2022-04-30 14:40:33,967 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object a8485d936ac2e7ccffffffffffffffffffffffff0100000001000000, aborting
  286. [2022-04-30 14:40:33,967 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object b9a008d165a7e804ffffffffffffffffffffffff0100000001000000, aborting
  287. [2022-04-30 14:40:33,970 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 28c7376153a43fb1ffffffffffffffffffffffff0100000001000000, aborting
  288. [2022-04-30 14:40:33,976 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 7486c9c5cb2b345effffffffffffffffffffffff0100000001000000, aborting
  289. [2022-04-30 14:40:33,977 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object b1d906d2acc455b1ffffffffffffffffffffffff0100000001000000, aborting
  290. [2022-04-30 14:40:33,980 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object a98b912db1b8ed14ffffffffffffffffffffffff0100000001000000, aborting
  291. [2022-04-30 14:40:33,997 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 0c025bfe7d0aed89ffffffffffffffffffffffff0100000001000000, aborting
  292. [2022-04-30 14:40:33,998 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object b58f0ee91e0a9599ffffffffffffffffffffffff0100000001000000, aborting
  293. [2022-04-30 14:40:33,998 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000004000000, aborting
  294. [2022-04-30 14:40:34,000 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object d99fa8d5c923d7c8ffffffffffffffffffffffff0100000001000000, aborting
  295. [2022-04-30 14:40:34,044 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object a46bb8530f4064fdffffffffffffffffffffffff0100000001000000, aborting
  296. [2022-04-30 14:40:34,085 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 6efb86ef2d286c40ffffffffffffffffffffffff0100000001000000, aborting
  297. [2022-04-30 14:40:34,088 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object ab92ac1da64ea740ffffffffffffffffffffffff0100000001000000, aborting
  298. [2022-04-30 14:40:34,110 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 38e143cc5dfd1165ffffffffffffffffffffffff0100000001000000, aborting
  299. [2022-04-30 14:40:34,133 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object ddbc5f4ddd8bc822ffffffffffffffffffffffff0100000001000000, aborting
  300. [2022-04-30 14:40:34,135 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object aea28d72824b5c00ffffffffffffffffffffffff0100000001000000, aborting
  301. [2022-04-30 14:40:34,143 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 877e0687cd603f1affffffffffffffffffffffff0100000001000000, aborting
  302. [2022-04-30 14:40:34,150 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object f50e24e4365654caffffffffffffffffffffffff0100000001000000, aborting
  303. [2022-04-30 14:40:34,154 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 9382939a56d17e30ffffffffffffffffffffffff0100000001000000, aborting
  304. [2022-04-30 14:40:34,173 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 6710354e78dd8e3affffffffffffffffffffffff0100000001000000, aborting
  305. [2022-04-30 14:40:34,185 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c49845173a8098c6ffffffffffffffffffffffff0100000001000000, aborting
  306. [2022-04-30 14:40:34,187 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 798376f73f71e22cffffffffffffffffffffffff0100000001000000, aborting
  307. [2022-04-30 14:40:34,204 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 878b2330f6be64f9ffffffffffffffffffffffff0100000001000000, aborting
  308. [2022-04-30 14:40:34,217 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object aa5ec071964645d2ffffffffffffffffffffffff0100000001000000, aborting
  309. [2022-04-30 14:40:34,221 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 6980fa2d7d749926ffffffffffffffffffffffff0100000001000000, aborting
  310. [2022-04-30 14:40:34,227 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object b812cdadd8a780aaffffffffffffffffffffffff0100000001000000, aborting
  311. [2022-04-30 14:40:34,238 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c0ca37ae352c9211ffffffffffffffffffffffff0100000001000000, aborting
  312. [2022-04-30 14:40:34,239 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object e3a0a2b2c6761c99ffffffffffffffffffffffff0100000001000000, aborting
  313. [2022-04-30 14:40:34,241 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object a2646843e7781738ffffffffffffffffffffffff0100000001000000, aborting
  314. [2022-04-30 14:40:34,250 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 8615ef612312971cffffffffffffffffffffffff0100000001000000, aborting
  315. [2022-04-30 14:40:34,258 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object eb9952338cd40cb8ffffffffffffffffffffffff0100000001000000, aborting
  316. [2022-04-30 14:40:34,266 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 2f7fdef4e2df3f2bffffffffffffffffffffffff0100000001000000, aborting
  317. [2022-04-30 14:40:34,269 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 8c115ad83d29434cffffffffffffffffffffffff0100000001000000, aborting
  318. [2022-04-30 14:40:34,292 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 8c5fdac94fbfecf4ffffffffffffffffffffffff0100000001000000, aborting
  319. [2022-04-30 14:40:34,298 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 9a02fb89192a6825ffffffffffffffffffffffff0100000001000000, aborting
  320. [2022-04-30 14:40:34,308 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 28241d0dda191e41ffffffffffffffffffffffff0100000001000000, aborting
  321. [2022-04-30 14:40:34,308 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000008000000, aborting
  322. [2022-04-30 14:40:34,308 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 940f384be6ded156ffffffffffffffffffffffff0100000001000000, aborting
  323. [2022-04-30 14:40:34,310 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 04c1bfd18b2d268bffffffffffffffffffffffff0100000001000000, aborting
  324. [2022-04-30 14:40:34,350 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 058595f16dc6f278ffffffffffffffffffffffff0100000001000000, aborting
  325. [2022-04-30 14:40:34,350 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 0d0a4ad0afd5b5d6ffffffffffffffffffffffff0100000001000000, aborting
  326. [2022-04-30 14:40:34,353 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object c35e1f786ff20173ffffffffffffffffffffffff0100000001000000, aborting
  327. [2022-04-30 14:40:34,356 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 4481ec012324614bffffffffffffffffffffffff0100000001000000, aborting
  328. [2022-04-30 14:40:34,356 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 72482135a26f4e0fffffffffffffffffffffffff0100000001000000, aborting
  329. [2022-04-30 14:40:34,373 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object e52bcdf406bc5724ffffffffffffffffffffffff0100000001000000, aborting
  330. [2022-04-30 14:40:34,389 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object ae2ec164b227894dffffffffffffffffffffffff0100000001000000, aborting
  331. [2022-04-30 14:40:34,395 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 8d3cb6ca70334e93ffffffffffffffffffffffff0100000001000000, aborting
  332. [2022-04-30 14:40:34,397 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 9a667646e288b252ffffffffffffffffffffffff0100000001000000, aborting
  333. [2022-04-30 14:40:34,398 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 575dd43a3ded8278ffffffffffffffffffffffff0100000001000000, aborting
  334. [2022-04-30 14:40:34,398 I 13292 13512] (raylet.exe) object_buffer_pool.cc:153: Not enough memory to create requested object 00ffffffffffffffffffffffffffffffffffffff0100000006000000, aborting
  335. [2022-04-30 14:40:43,682 W 13292 13512] (raylet.exe) agent_manager.cc:94: Agent process with pid 14932 has not registered. ip 127.0.0.1, pid 16324
  336. [2022-04-30 14:40:43,683 W 13292 5940] (raylet.exe) agent_manager.cc:104: Agent process with pid 14932 exit, return value 1067. ip 127.0.0.1. pid 16324
  337. [2022-04-30 14:40:43,683 E 13292 5940] (raylet.exe) agent_manager.cc:107: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. See `dashboard_agent.log` for the root cause.
  338.  
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!