API tools faq
paste
subasah

SAC failure log

subasah
Oct 26th, 2021 (edited)
594
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 5.01 KB | None | 0 0
raw download clone embed print report
  1. 386 reward -32.62/-15.71/ -2.84 len 24.00 saved sac_training_26oct/NR_IES/checkpoint_1386/checkpoint-1386
  2. 2021-10-26 20:12:32,539 DEBUG memory_monitor.py:134 -- Memory usage is 29.408241271972656 / 31.058788299560547
  3. 2021-10-26 20:12:33,542 DEBUG memory_monitor.py:134 -- Memory usage is 29.3974609375 / 31.058788299560547
  4. 1387 reward -33.65/-15.27/ -2.84 len 24.00 saved sac_training_26oct/NR_IES/checkpoint_1387/checkpoint-1387
  5. 1388 reward -33.65/-15.27/ -2.84 len 24.00 saved sac_training_26oct/NR_IES/checkpoint_1388/checkpoint-1388
  6. 1389 reward -33.65/-15.27/ -2.84 len 24.00 saved sac_training_26oct/NR_IES/checkpoint_1389/checkpoint-1389
  7. 1390 reward -33.65/-15.27/ -2.84 len 24.00 saved sac_training_26oct/NR_IES/checkpoint_1390/checkpoint-1390
  8. Traceback (most recent call last):
  9. File "sac_train.py", line 62, in <module>
  10. main()
  11. File "sac_train.py", line 50, in main
  12. result = agent.train()
  13. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 526, in train
  14. raise e
  15. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 515, in train
  16. result = Trainable.train(self)
  17. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/tune/trainable.py", line 226, in train
  18. result = self.step()
  19. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 148, in step
  20. res = next(self.train_exec_impl)
  21. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 756, in __next__
  22. return next(self.built_iterator)
  23. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 783, in apply_foreach
  24. for item in it:
  25. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 843, in apply_filter
  26. for item in it:
  27. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 843, in apply_filter
  28. for item in it:
  29. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 783, in apply_foreach
  30. for item in it:
  31. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 843, in apply_filter
  32. for item in it:
  33. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 1075, in build_union
  34. item = next(it)
  35. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 756, in __next__
  36. return next(self.built_iterator)
  37. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 783, in apply_foreach
  38. for item in it:
  39. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 783, in apply_foreach
  40. for item in it:
  41. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 783, in apply_foreach
  42. for item in it:
  43. [Previous line repeated 1 more time]
  44. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/util/iter.py", line 471, in base_iterator
  45. yield ray.get(futures, timeout=timeout)
  46. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
  47. return func(*args, **kwargs)
  48. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/worker.py", line 1456, in get
  49. raise value.as_instanceof_cause()
  50. ray.exceptions.RayTaskError(RayOutOfMemoryError): ray::RolloutWorker.par_iter_next() (pid=616133, ip=131.183.21.110)
  51. File "python/ray/_raylet.pyx", line 440, in ray._raylet.execute_task
  52. File "/home/ssah3@UTAD/miniconda3/envs/rl/lib/python3.7/site-packages/ray/memory_monitor.py", line 132, in raise_if_low_memory
  53. self.error_threshold))
  54. ray.memory_monitor.RayOutOfMemoryError: More than 95% of the memory on node ENG86NTJD3 is used (29.54 / 31.06 GB). The top 10 memory consumers are:
  55.  
  56. PID MEM COMMAND
  57. 616133 17.1GiB python sac_train.py
  58. 540320 4.92GiB python sac_train.py
  59. 529027 0.69GiB /var/lib/snapd/snap/code/78/usr/share/code/code /home/ssah3@UTAD/.vscode/extensions/ms-python.vscode
  60. 34207 0.33GiB /usr/bin/gnome-shell
  61. 528707 0.28GiB /var/lib/snapd/snap/code/78/usr/share/code/code --type=renderer --disable-color-correct-rendering --
  62. 33518 0.25GiB /usr/libexec/packagekitd
  63. 35507 0.23GiB /snap/chromium/1781/usr/lib/chromium-browser/chrome --password-store=basic
  64. 528806 0.2GiB /var/lib/snapd/snap/code/78/usr/share/code/code /var/lib/snapd/snap/code/78/usr/share/code/resources
  65. 528688 0.2GiB /var/lib/snapd/snap/code/78/usr/share/code/code --type=gpu-process --field-trial-handle=883898412019
  66. 35923 0.13GiB /snap/chromium/1781/usr/lib/chromium-browser/chrome --type=gpu-process --field-trial-handle=12377216
  67.  
  68. In addition, up to 0.34 GiB of shared memory is currently being used by the Ray object store.
  69. ---
  70. --- Tip: Use the `ray memory` command to list active objects in the cluster.
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!