API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 22nd, 2023
34
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 22.91 KB | None | 0 0
raw download clone embed print report
  1. 2023-12-20 11:47:01,095 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 20.06 GiB -- Worker memory limit: 30.73 GiB
  2. 2023-12-20 11:47:46,387 - distributed.worker.memory - WARNING - Worker is at 84% memory usage. Pausing worker. Process memory: 26.03 GiB -- Worker memory limit: 30.73 GiB
  3. 2023-12-20 11:47:46,388 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 26.03 GiB -- Worker memory limit: 30.73 GiB
  4. 2023-12-20 11:47:46,888 - distributed.worker.memory - WARNING - Worker is at 78% memory usage. Resuming worker. Process memory: 24.12 GiB -- Worker memory limit: 30.73 GiB
  5. 2023-12-20 11:47:47,088 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.68 GiB -- Worker memory limit: 30.73 GiB
  6. 2023-12-20 11:47:49,451 - distributed.worker.memory - WARNING - Worker is at 72% memory usage. Resuming worker. Process memory: 22.28 GiB -- Worker memory limit: 30.73 GiB
  7. 2023-12-20 11:48:55,659 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 21.23 GiB -- Worker memory limit: 30.73 GiB
  8. 2023-12-20 11:51:54,201 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:39439 (pid=10968) exceeded 95% memory budget. Restarting...
  9. 2023-12-20 11:51:56,371 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703101915.7858977
  10. 2023-12-20 11:51:56,406 - distributed.nanny - WARNING - Restarting worker
  11. 2023-12-20 12:08:52,413 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 20.10 GiB -- Worker memory limit: 30.73 GiB
  12. 2023-12-20 12:12:24,582 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 23.76 GiB -- Worker memory limit: 30.73 GiB
  13. 2023-12-20 12:12:25,193 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.59 GiB -- Worker memory limit: 30.73 GiB
  14. 2023-12-20 12:12:30,999 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:40955 (pid=11829) exceeded 95% memory budget. Restarting...
  15. 2023-12-20 12:12:37,168 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703103156.6875005
  16. 2023-12-20 12:12:37,198 - distributed.nanny - WARNING - Restarting worker
  17. 2023-12-20 12:12:40,797 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:34821 (pid=10963) exceeded 95% memory budget. Restarting...
  18. 2023-12-20 12:12:41,362 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703103161.2366807
  19. 2023-12-20 12:12:42,048 - distributed.nanny - WARNING - Restarting worker
  20. 2023-12-20 12:12:43,547 - distributed.worker - WARNING - Compute Failed
  21. Key: ('shuffle-transfer-68b183a3b10ac29176568e72a8d8e017', 1744)
  22. Function: shuffle_transfer
  23. args: ( ProductCode ... _partitions
  24. Time ...
  25. 2022-06-10 13:42:47.205000+00:00 ESU22 ... 1594
  26. 2022-06-10 13:42:47.205001+00:00 ESU22 ... 1594
  27. 2022-06-10 13:42:47.206000+00:00 ESM22 ... 273
  28. 2022-06-10 13:42:47.215000+00:00 ESU22 ... 1594
  29. 2022-06-10 13:42:47.224000+00:00 ESM22 ... 273
  30. ... ... ... ...
  31. 2022-06-10 19:13:25.164003+00:00 ESM22 ... 273
  32. 2022-06-10 19:13:25.164004+00:00 ESM22 ... 273
  33. 2022-06-10 19:13:25.164005+00:00 ESM22 ... 273
  34. 2022-06-10 19:13:25.249000+00:00 ESM22 ... 273
  35. 2022-06-10 19:13:25.372000+00:00 ESM22 ... 273
  36.  
  37. [1053010 rows x 13 columns], '68b183a3b10ac29176568e72a8d8e017', 1744, 2177, '_partitions', Empty DataFrame
  38. Columns: [ProductCode, Open, High, Low, Close, Trades, Volume, BidVolum
  39. kwargs: {}
  40. Exception: "RuntimeError('P2P shuffling 68b183a3b10ac29176568e72a8d8e017 failed during transfer phase')"
  41.  
  42. 2023-12-20 12:13:14,473 - distributed.shuffle._comms - ERROR - Worker tcp://127.0.0.1:34821 left during active SchedulerShuffleState<68b183a3b10ac29176568e72a8d8e017[3]>
  43. Traceback (most recent call last):
  44. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/utils.py", line 832, in wrapper
  45. return await func(*args, **kwargs)
  46. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
  47. await self.send(address, shards)
  48. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 179, in send
  49. return await retry(
  50. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/utils_comm.py", line 424, in retry
  51. return await coro()
  52. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 159, in _send
  53. self.raise_if_closed()
  54. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 220, in raise_if_closed
  55. raise self._exception
  56. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 428, in handle_transfer_errors
  57. yield
  58. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_shuffle.py", line 70, in shuffle_transfer
  59. return get_worker_plugin().add_partition(
  60. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_worker_plugin.py", line 338, in add_partition
  61. return shuffle_run.add_partition(
  62. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 292, in add_partition
  63. sync(self._loop, self._write_to_comm, shards)
  64. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/utils.py", line 434, in sync
  65. raise error
  66. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/utils.py", line 408, in f
  67. result = yield future
  68. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/tornado/gen.py", line 767, in run
  69. value = future.result()
  70. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 208, in _write_to_comm
  71. self.raise_if_closed()
  72. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 220, in raise_if_closed
  73. raise self._exception
  74. RuntimeError: Worker tcp://127.0.0.1:34821 left during active SchedulerShuffleState<68b183a3b10ac29176568e72a8d8e017[3]>
  75. 2023-12-20 12:29:44,797 - distributed.worker.memory - WARNING - Worker is at 85% memory usage. Pausing worker. Process memory: 26.35 GiB -- Worker memory limit: 30.73 GiB
  76. 2023-12-20 12:29:47,097 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:43341 (pid=12604) exceeded 95% memory budget. Restarting...
  77. 2023-12-20 12:29:47,530 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703104187.476938
  78. 2023-12-20 12:29:48,135 - distributed.nanny - WARNING - Restarting worker
  79. 2023-12-20 12:45:42,736 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 22.25 GiB -- Worker memory limit: 30.73 GiB
  80. 2023-12-20 12:45:53,616 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 22.06 GiB -- Worker memory limit: 30.73 GiB
  81. 2023-12-20 12:46:18,664 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 23.91 GiB -- Worker memory limit: 30.73 GiB
  82. 2023-12-20 12:46:19,222 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.70 GiB -- Worker memory limit: 30.73 GiB
  83. 2023-12-20 12:46:21,423 - distributed.worker.memory - WARNING - Worker is at 78% memory usage. Resuming worker. Process memory: 24.13 GiB -- Worker memory limit: 30.73 GiB
  84. 2023-12-20 12:46:21,623 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.64 GiB -- Worker memory limit: 30.73 GiB
  85. 2023-12-20 12:46:25,058 - distributed.worker.memory - WARNING - Worker is at 72% memory usage. Resuming worker. Process memory: 22.21 GiB -- Worker memory limit: 30.73 GiB
  86. 2023-12-20 12:46:28,725 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 22.98 GiB -- Worker memory limit: 30.73 GiB
  87. 2023-12-20 12:47:09,820 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 21.22 GiB -- Worker memory limit: 30.73 GiB
  88. 2023-12-20 12:48:45,697 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:44611 (pid=12583) exceeded 95% memory budget. Restarting...
  89. 2023-12-20 12:48:46,819 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703105326.7601912
  90. 2023-12-20 12:48:46,829 - distributed.nanny - WARNING - Restarting worker
  91. 2023-12-20 13:04:33,452 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.67 GiB -- Worker memory limit: 30.73 GiB
  92. 2023-12-20 13:04:38,635 - distributed.worker.memory - WARNING - Worker is at 65% memory usage. Resuming worker. Process memory: 20.14 GiB -- Worker memory limit: 30.73 GiB
  93. 2023-12-20 13:04:38,636 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 20.14 GiB -- Worker memory limit: 30.73 GiB
  94. 2023-12-20 13:05:57,786 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 24.00 GiB -- Worker memory limit: 30.73 GiB
  95. 2023-12-20 13:07:37,328 - distributed.worker.memory - WARNING - Worker is at 83% memory usage. Pausing worker. Process memory: 25.69 GiB -- Worker memory limit: 30.73 GiB
  96. 2023-12-20 13:07:37,332 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 25.69 GiB -- Worker memory limit: 30.73 GiB
  97. 2023-12-20 13:07:48,996 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 28.02 GiB -- Worker memory limit: 30.73 GiB
  98. 2023-12-20 13:07:50,994 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:34537 (pid=13227) exceeded 95% memory budget. Restarting...
  99. 2023-12-20 13:07:51,394 - distributed.worker.memory - WARNING - Worker is at 86% memory usage. Pausing worker. Process memory: 26.73 GiB -- Worker memory limit: 30.73 GiB
  100. 2023-12-20 13:07:51,395 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 26.73 GiB -- Worker memory limit: 30.73 GiB
  101. 2023-12-20 13:07:51,517 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703106471.4624138
  102. 2023-12-20 13:07:51,686 - distributed.nanny - WARNING - Restarting worker
  103. 2023-12-20 13:07:51,796 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:43647 (pid=13929) exceeded 95% memory budget. Restarting...
  104. 2023-12-20 13:07:52,683 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703106472.4071612
  105. 2023-12-20 13:07:52,691 - distributed.nanny - WARNING - Restarting worker
  106. 2023-12-20 13:07:53,025 - distributed.shuffle._comms - ERROR - Worker tcp://127.0.0.1:43647 left during active SchedulerShuffleState<68b183a3b10ac29176568e72a8d8e017[7]>
  107. Traceback (most recent call last):
  108. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/utils.py", line 832, in wrapper
  109. return await func(*args, **kwargs)
  110. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
  111. await self.send(address, shards)
  112. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 179, in send
  113. return await retry(
  114. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/utils_comm.py", line 424, in retry
  115. return await coro()
  116. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 159, in _send
  117. self.raise_if_closed()
  118. File "/home/igor/sierra/pyenv/lib/python3.10/site-packages/distributed/shuffle/_core.py", line 220, in raise_if_closed
  119. raise self._exception
  120. RuntimeError: Worker tcp://127.0.0.1:43647 left during active SchedulerShuffleState<68b183a3b10ac29176568e72a8d8e017[7]>
  121. 2023-12-20 13:23:48,242 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 19.93 GiB -- Worker memory limit: 30.73 GiB
  122. 2023-12-20 13:24:41,648 - distributed.worker.memory - WARNING - Worker is at 83% memory usage. Pausing worker. Process memory: 25.61 GiB -- Worker memory limit: 30.73 GiB
  123. 2023-12-20 13:24:41,648 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 25.61 GiB -- Worker memory limit: 30.73 GiB
  124. 2023-12-20 13:24:42,686 - distributed.worker.memory - WARNING - Worker is at 79% memory usage. Resuming worker. Process memory: 24.46 GiB -- Worker memory limit: 30.73 GiB
  125. 2023-12-20 13:24:42,785 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.86 GiB -- Worker memory limit: 30.73 GiB
  126. 2023-12-20 13:24:44,597 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:35109 (pid=10970) exceeded 95% memory budget. Restarting...
  127. 2023-12-20 13:24:45,323 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703107484.988867
  128. 2023-12-20 13:24:46,202 - distributed.nanny - WARNING - Restarting worker
  129. 2023-12-20 13:41:14,405 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 23.44 GiB -- Worker memory limit: 30.73 GiB
  130. 2023-12-20 13:41:15,209 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 23.90 GiB -- Worker memory limit: 30.73 GiB
  131. 2023-12-20 13:41:16,323 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.61 GiB -- Worker memory limit: 30.73 GiB
  132. 2023-12-20 13:41:20,304 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 25.23 GiB -- Worker memory limit: 30.73 GiB
  133. 2023-12-20 13:41:20,922 - distributed.worker.memory - WARNING - Worker is at 78% memory usage. Resuming worker. Process memory: 24.10 GiB -- Worker memory limit: 30.73 GiB
  134. 2023-12-20 13:41:21,623 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 24.63 GiB -- Worker memory limit: 30.73 GiB
  135. 2023-12-20 13:41:22,748 - distributed.worker.memory - WARNING - Worker is at 77% memory usage. Resuming worker. Process memory: 23.91 GiB -- Worker memory limit: 30.73 GiB
  136. 2023-12-20 13:41:26,143 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 25.82 GiB -- Worker memory limit: 30.73 GiB
  137. 2023-12-20 13:41:26,405 - distributed.worker.memory - WARNING - Worker is at 72% memory usage. Resuming worker. Process memory: 22.23 GiB -- Worker memory limit: 30.73 GiB
  138. 2023-12-20 13:41:27,490 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 21.53 GiB -- Worker memory limit: 30.73 GiB
  139. 2023-12-20 13:43:37,421 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 25.44 GiB -- Worker memory limit: 30.73 GiB
  140. 2023-12-20 13:43:37,421 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 25.44 GiB -- Worker memory limit: 30.73 GiB
  141. 2023-12-20 13:43:39,492 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:41463 (pid=14120) exceeded 95% memory budget. Restarting...
  142. 2023-12-20 13:43:39,763 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 68b183a3b10ac29176568e72a8d8e017 restarted due to stimulus 'handle-worker-cleanup-1703108619.7306645
  143. 2023-12-20 13:43:40,777 - distributed.nanny - WARNING - Restarting worker
  144. ---------------------------------------------------------------------------
  145. KilledWorker Traceback (most recent call last)
  146. Cell In[10], line 1
  147. ----> 1 merged.to_parquet('/mnt/d/sierra_data/ticks_flat/resampled_merged/', overwrite=True)
  148.  
  149. File ~/sierra/pyenv/lib/python3.10/site-packages/dask/dataframe/core.py:5733, in DataFrame.to_parquet(self, path, *args, **kwargs)
  150. 5730 """See dd.to_parquet docstring for more information"""
  151. 5731 from dask.dataframe.io import to_parquet
  152. -> 5733 return to_parquet(self, path, *args, **kwargs)
  153.  
  154. File ~/sierra/pyenv/lib/python3.10/site-packages/dask/dataframe/io/parquet/core.py:1057, in to_parquet(df, path, engine, compression, write_index, append, overwrite, ignore_divisions, partition_on, storage_options, custom_metadata, write_metadata_file, compute, compute_kwargs, schema, name_function, filesystem, **kwargs)
  155. 1054 out = Scalar(graph, final_name, "")
  156. 1056 if compute:
  157. -> 1057 out = out.compute(**compute_kwargs)
  158. 1059 # Invalidate the filesystem listing cache for the output path after write.
  159. 1060 # We do this before returning, even if `compute=False`. This helps ensure
  160. 1061 # that reading files that were just written succeeds.
  161. 1062 fs.invalidate_cache(path)
  162.  
  163. File ~/sierra/pyenv/lib/python3.10/site-packages/dask/base.py:342, in DaskMethodsMixin.compute(self, **kwargs)
  164. 318 def compute(self, **kwargs):
  165. 319 """Compute this dask collection
  166. 320
  167. 321 This turns a lazy Dask collection into its in-memory equivalent.
  168. (...)
  169. 340 dask.compute
  170. 341 """
  171. --> 342 (result,) = compute(self, traverse=False, **kwargs)
  172. 343 return result
  173.  
  174. File ~/sierra/pyenv/lib/python3.10/site-packages/dask/base.py:628, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
  175. 625 postcomputes.append(x.__dask_postcompute__())
  176. 627 with shorten_traceback():
  177. --> 628 results = schedule(dsk, keys, **kwargs)
  178. 630 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
  179.  
  180. File ~/sierra/pyenv/lib/python3.10/site-packages/distributed/client.py:2243, in Client._gather(self, futures, errors, direct, local_worker)
  181. 2241 exc = CancelledError(key)
  182. 2242 else:
  183. -> 2243 raise exception.with_traceback(traceback)
  184. 2244 raise exc
  185. 2245 if errors == "skip":
  186.  
  187. KilledWorker: Attempted to run task ('shuffle_p2p-4e91bef288d199c948c38110a3239403', 2114) on 4 different workers, but all those workers died while running it. The last worker that attempt to run the task was tcp://127.0.0.1:41463. Inspecting worker logs is often a good next step to diagnose what went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.
  188.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!