Python-Core-Driver Log

[2022-10-05 10:30:31,049 I 1590212 1590234] core_worker.cc:476: Event stats:


Global stats: 117739 total (71 active)
Queueing time: mean = 230.980 us, max = 663.959 ms, min = -0.014 s, total = 27.195 s
Execution time:  mean = 61.834 us, total = 7.280 s
Event stats:
	UNKNOWN - 91695 total (4 active, 1 running), CPU time: mean = 47.415 us, total = 4.348 s
	CoreWorker.deadline_timer.flush_profiling_events - 8338 total (1 active), CPU time: mean = 24.928 us, total = 207.852 ms
	NodeManagerService.grpc_client.ReportWorkerBacklog - 8337 total (0 active), CPU time: mean = 24.023 us, total = 200.279 ms
	CoreWorkerService.grpc_server.GetCoreWorkerStats - 4117 total (0 active), CPU time: mean = 113.823 us, total = 468.609 ms
	CoreWorkerService.grpc_client.PushTask - 1778 total (1 active), CPU time: mean = 157.508 us, total = 280.048 ms
	CoreWorkerDirectActorTaskSubmitter::SubmitTask - 1778 total (0 active), CPU time: mean = 69.505 us, total = 123.579 ms
	CoreWorkerService.grpc_client.PubsubLongPolling - 915 total (64 active), CPU time: mean = 163.451 us, total = 149.557 ms
	CoreWorkerService.grpc_client.PubsubCommandBatch - 127 total (0 active), CPU time: mean = 15.423 us, total = 1.959 ms
	Subscriber.HandlePublishedMessage_GCS_ACTOR_CHANNEL - 117 total (0 active), CPU time: mean = 158.988 us, total = 18.602 ms
	InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 116 total (1 active), CPU time: mean = 260.504 us, total = 30.219 ms
	StatsGcsService.grpc_client.AddProfileData - 77 total (0 active), CPU time: mean = 108.769 us, total = 8.375 ms
	CoreWorkerService.grpc_server.WaitForActorOutOfScope - 64 total (0 active), CPU time: mean = 13.366 us, total = 855.416 us
	ActorInfoGcsService.grpc_client.GetActorInfo - 64 total (0 active), CPU time: mean = 31.091 us, total = 1.990 ms
	ActorCreator.AsyncRegisterActor - 64 total (0 active), CPU time: mean = 306.524 us, total = 19.618 ms
	ActorInfoGcsService.grpc_client.CreateActor - 64 total (0 active), CPU time: mean = 364.153 us, total = 23.306 ms
	ActorInfoGcsService.grpc_client.RegisterActor - 64 total (0 active), CPU time: mean = 235.484 us, total = 15.071 ms
	CoreWorkerService.grpc_server.LocalGC - 13 total (0 active), CPU time: mean = 105.725 ms, total = 1.374 s
	PeriodicalRunner.RunFnPeriodically - 5 total (0 active), CPU time: mean = 70.456 us, total = 352.281 us
	InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 4 total (0 active), CPU time: mean = 1.848 ms, total = 7.391 ms
	NodeInfoGcsService.grpc_client.GetAllNodeInfo - 1 total (0 active), CPU time: mean = 516.477 us, total = 516.477 us
	WorkerInfoGcsService.grpc_client.AddWorkerInfo - 1 total (0 active), CPU time: mean = 12.829 us, total = 12.829 us


[2022-10-05 10:30:52,454 I 1590212 1590212] core_worker.cc:593: Disconnecting to the raylet.
[2022-10-05 10:30:52,454 I 1590212 1590212] raylet_client.cc:163: RayletClient::Disconnect, exit_type=INTENDED_USER_EXIT, exit_detail=Shutdown by ray.shutdown()., has creation_task_exception_pb_bytes=0
[2022-10-05 10:30:52,454 I 1590212 1590212] core_worker.cc:540: Shutting down a core worker.
[2022-10-05 10:30:52,454 I 1590212 1590212] core_worker.cc:564: Disconnecting a GCS client.
[2022-10-05 10:30:52,454 I 1590212 1590212] core_worker.cc:568: Waiting for joining a core worker io thread. If it hangs here, there might be deadlock or a high load in the core worker io service.
[2022-10-05 10:30:52,454 I 1590212 1590234] core_worker.cc:691: Core worker main io service stopped.
[2022-10-05 10:30:52,455 I 1590212 1590212] core_worker.cc:577: Core worker ready to be deallocated.
[2022-10-05 10:30:52,455 I 1590212 1590212] core_worker.cc:531: Core worker is destructed
[2022-10-05 10:30:52,735 I 1590212 1590212] core_worker_process.cc:144: Destructing CoreWorkerProcessImpl. pid: 1590212
[2022-10-05 10:30:52,735 I 1590212 1590212] io_service_pool.cc:47: IOServicePool is stopped.