Guest User

Untitled

a guest
Jul 8th, 2020
9
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. 2020-07-07 03:09:15
  2. 2020-07-07 01:09:15,476 WARNING worker.py:1090 -- A worker died or was killed while executing task ffffffffffffffff4d81fd5d0100.
  3. 2020-07-07 03:09:15
  4.  
  5. 2020-07-07 03:09:15
  6.  
  7. 2020-07-07 03:09:15
  8. +-------------------------------+------------+-------+--------+------------------+-------+----------+
  9. 2020-07-07 03:09:15
  10. | PPO_procgen_env_wrapper_00000 | TERMINATED | | 3 | 7318.15 | 55296 | 3.8 |
  11. 2020-07-07 03:09:15
  12. |-------------------------------+------------+-------+--------+------------------+-------+----------|
  13. 2020-07-07 03:09:15
  14. | Trial name | status | loc | iter | total time (s) | ts | reward |
  15. 2020-07-07 03:09:15
  16. +-------------------------------+------------+-------+--------+------------------+-------+----------+
  17. 2020-07-07 03:09:15
  18. Number of trials: 1 (1 TERMINATED)
  19. 2020-07-07 03:09:15
  20. Result logdir: /outputs/ray-results/procgen-ppo
  21. 2020-07-07 03:09:15
  22. Resources requested: 0/16 CPUs, 0.0/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
  23. 2020-07-07 03:09:15
  24. Using FIFO scheduling algorithm.
  25. 2020-07-07 03:09:15
  26. Memory usage on this node: 7.1/59.0 GiB
  27. 2020-07-07 03:09:15
  28. == Status ==
  29. 2020-07-07 03:09:15
  30.  
  31. 2020-07-07 03:09:15
  32.  
  33. 2020-07-07 03:09:15
  34. +-------------------------------+----------+---------------+--------+------------------+-------+----------+
  35. 2020-07-07 03:09:15
  36. | PPO_procgen_env_wrapper_00000 | RUNNING | 10.60.3.12:99 | 3 | 7318.15 | 55296 | 3.8 |
  37. 2020-07-07 03:09:15
  38. |-------------------------------+----------+---------------+--------+------------------+-------+----------|
  39. 2020-07-07 03:09:15
  40. | Trial name | status | loc | iter | total time (s) | ts | reward |
  41. 2020-07-07 03:09:15
  42. +-------------------------------+----------+---------------+--------+------------------+-------+----------+
  43. 2020-07-07 03:09:15
  44. Number of trials: 1 (1 RUNNING)
  45. 2020-07-07 03:09:15
  46. Result logdir: /outputs/ray-results/procgen-ppo
  47. 2020-07-07 03:09:15
  48. Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
  49. 2020-07-07 03:09:15
  50. Using FIFO scheduling algorithm.
  51. 2020-07-07 03:09:15
  52. Memory usage on this node: 7.1/59.0 GiB
  53. 2020-07-07 03:09:15
  54. == Status ==
  55. 2020-07-07 03:09:15
  56. [Errno 2] No such file or directory: 'merged-videos/training.mp4'
  57. 2020-07-07 03:09:15
  58.  
  59. 2020-07-07 03:09:15
  60.  
  61. 2020-07-07 03:09:15
  62. +-------------------------------+----------+---------------+--------+------------------+-------+----------+
  63. 2020-07-07 03:09:15
  64. | PPO_procgen_env_wrapper_00000 | RUNNING | 10.60.3.12:99 | 2 | 4900.67 | 36864 | 3.75 |
  65. 2020-07-07 03:09:15
  66. |-------------------------------+----------+---------------+--------+------------------+-------+----------|
  67. 2020-07-07 03:09:15
  68. | Trial name | status | loc | iter | total time (s) | ts | reward |
  69. 2020-07-07 03:09:15
  70. +-------------------------------+----------+---------------+--------+------------------+-------+----------+
  71. 2020-07-07 03:09:15
  72. Number of trials: 1 (1 RUNNING)
  73. 2020-07-07 03:09:15
  74. Result logdir: /outputs/ray-results/procgen-ppo
  75. 2020-07-07 03:09:15
  76. Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
  77. 2020-07-07 03:09:15
  78. Using FIFO scheduling algorithm.
  79. 2020-07-07 03:09:15
  80. Memory usage on this node: 7.0/59.0 GiB
  81. 2020-07-07 03:09:15
  82. == Status ==
  83. 2020-07-07 03:09:15
  84. [Errno 2] No such file or directory: 'merged-videos/training.mp4'
  85. 2020-07-07 03:09:15
  86.  
  87. 2020-07-07 03:09:15
  88.  
  89. 2020-07-07 03:09:15
  90. +-------------------------------+----------+---------------+--------+------------------+-------+----------+
  91. 2020-07-07 03:09:15
  92. | PPO_procgen_env_wrapper_00000 | RUNNING | 10.60.3.12:99 | 1 | 2469.2 | 18432 | 4.16667 |
  93. 2020-07-07 03:09:15
  94. |-------------------------------+----------+---------------+--------+------------------+-------+----------|
  95. 2020-07-07 03:09:15
  96. | Trial name | status | loc | iter | total time (s) | ts | reward |
  97. 2020-07-07 03:09:15
  98. +-------------------------------+----------+---------------+--------+------------------+-------+----------+
  99. 2020-07-07 03:09:15
  100. Number of trials: 1 (1 RUNNING)
  101. 2020-07-07 03:09:15
  102. Result logdir: /outputs/ray-results/procgen-ppo
  103. 2020-07-07 03:09:15
  104. Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
  105. 2020-07-07 03:09:15
  106. Using FIFO scheduling algorithm.
  107. 2020-07-07 03:09:15
  108. Memory usage on this node: 6.9/59.0 GiB
  109. 2020-07-07 03:09:15
  110. == Status ==
  111. 2020-07-07 03:09:15
  112. [Errno 2] No such file or directory: 'merged-videos/training.mp4'
  113. 2020-07-07 03:09:15
  114. (pid=104) /pytorch/torch/csrc/utils/tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.
  115. 2020-07-07 03:09:15
  116. (pid=99) 2020-07-06 23:07:13,612 WARNING util.py:37 -- Install gputil for GPU system monitoring.
  117. 2020-07-07 03:09:15
  118. (pid=99) 2020-07-06 23:07:13,612 INFO trainable.py:217 -- Getting current IP.
  119. 2020-07-07 03:09:15
  120. (pid=99) 2020-07-06 23:07:11,633 INFO trainer.py:580 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
  121. 2020-07-07 03:09:15
  122. (pid=99) 2020-07-06 23:07:11,628 INFO trainer.py:421 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
  123. 2020-07-07 03:09:15
  124.  
  125. 2020-07-07 03:09:15
  126.  
  127. 2020-07-07 03:09:15
  128. +-------------------------------+----------+-------+
  129. 2020-07-07 03:09:15
  130. | PPO_procgen_env_wrapper_00000 | RUNNING | |
  131. 2020-07-07 03:09:15
  132. |-------------------------------+----------+-------|
  133. 2020-07-07 03:09:15
  134. | Trial name | status | loc |
  135. 2020-07-07 03:09:15
  136. +-------------------------------+----------+-------+
  137. 2020-07-07 03:09:15
  138. Number of trials: 1 (1 RUNNING)
  139. 2020-07-07 03:09:15
  140. Result logdir: /outputs/ray-results/procgen-ppo
  141. 2020-07-07 03:09:15
  142. Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
  143. 2020-07-07 03:09:15
  144. Using FIFO scheduling algorithm.
  145. 2020-07-07 03:09:15
  146. Memory usage on this node: 2.7/59.0 GiB
  147. 2020-07-07 03:09:15
  148. == Status ==
  149. 2020-07-07 03:09:15
  150. {'contrib/RandomAgent': <function _import_random_agent at 0x7f3a30feddd0>, 'contrib/MADDPG': <function _import_maddpg at 0x7f3a30f94a70>, 'contrib/AlphaZero': <function _import_alphazero at 0x7f3a30f94d40>, 'contrib/LinTS': <function _import_bandit_lints at 0x7f3a30f94dd0>, 'contrib/LinUCB': <function _import_bandit_linucb at 0x7f3a30f94e60>}
  151. 2020-07-07 03:09:15
  152. 2020-07-07 01:09:15,224 WARNING util.py:137 -- The `process_trial` operation took 1.1798417568206787 seconds to complete, which may be a performance bottleneck.
  153. 2020-07-07 02:28:56
  154. 2020-07-07 00:28:56,565 WARNING util.py:137 -- The `process_trial` operation took 0.9509642124176025 seconds to complete, which may be a performance bottleneck.
  155. 2020-07-07 01:48:24
  156. 2020-07-06 23:48:24,132 WARNING util.py:137 -- The `process_trial` operation took 1.3134164810180664 seconds to complete, which may be a performance bottleneck.
  157. 2020-07-07 01:48:24
  158. if isinstance(v, collections.MutableMapping):
  159. 2020-07-07 01:48:24
  160. train.py:117: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  161. 2020-07-07 01:07:09
  162. 2020-07-06 23:07:09,465 ERROR syncer.py:39 -- Log sync requires rsync to be installed.
  163. 2020-07-07 01:07:08
  164.  
  165. 2020-07-07 01:07:08
  166. OSError: [Errno 99] error while attempting to bind on address ('::1', 8265, 0, 0): cannot assign requested address
  167. 2020-07-07 01:07:08
  168. % (sa, err.strerror.lower())) from None
  169. 2020-07-07 01:07:08
  170. File "/home/aicrowd/.conda/lib/python3.7/asyncio/base_events.py", line 1385, in create_server
  171. 2020-07-07 01:07:08
  172. reuse_port=self._reuse_port)
  173. 2020-07-07 01:07:08
  174. File "/home/aicrowd/.conda/lib/python3.7/site-packages/aiohttp/web_runner.py", line 104, in start
  175. 2020-07-07 01:07:08
  176. await site.start()
  177. 2020-07-07 01:07:08
  178. File "/home/aicrowd/.conda/lib/python3.7/site-packages/aiohttp/web.py", line 359, in _run_app
  179. 2020-07-07 01:07:08
  180. return future.result()
  181. 2020-07-07 01:07:08
  182. File "/home/aicrowd/.conda/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
  183. 2020-07-07 01:07:08
  184. reuse_port=reuse_port))
  185. 2020-07-07 01:07:08
  186. File "/home/aicrowd/.conda/lib/python3.7/site-packages/aiohttp/web.py", line 433, in run_app
  187. 2020-07-07 01:07:08
  188. aiohttp.web.run_app(self.app, host=self.host, port=self.port)
  189. 2020-07-07 01:07:08
  190. File "/home/aicrowd/.conda/lib/python3.7/site-packages/ray/dashboard/dashboard.py", line 594, in run
  191. 2020-07-07 01:07:08
  192. dashboard.run()
  193. 2020-07-07 01:07:08
  194. File "/home/aicrowd/.conda/lib/python3.7/site-packages/ray/dashboard/dashboard.py", line 1220, in <module>
  195. 2020-07-07 01:07:08
  196. Traceback (most recent call last):
  197. 2020-07-07 01:07:08
  198. 2020-07-06 23:07:08,749 WARNING worker.py:1090 -- The dashboard on node submission-gsz5w-2446593925 failed with the following error:
  199. 2020-07-07 01:07:08
  200. ls: cannot access '/outputs/ray-results/procgen-ppo/*/': No such file or directory
  201. 2020-07-07 01:07:07
  202. 2020-07-06 23:07:07,698 WARNING services.py:1494 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This may slow down performance! You may be able to free up space by deleting files in /dev/shm or terminating any running plasma_store_server processes. If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size' to 'docker run'.
  203. 2020-07-07 01:07:07
  204. 2020-07-06 23:07:07,695 INFO services.py:1170 -- View the Ray dashboard at localhost:8265
  205. 2020-07-07 01:07:07
  206. 2020-07-06 23:07:07,316 INFO resource_spec.py:212 -- Starting Ray with 55.96 GiB memory available for workers and up to 27.94 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
  207. 2020-07-07 01:07:07
  208. import imp
  209. 2020-07-07 01:07:07
  210. /home/aicrowd/.conda/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  211. 2020-07-07 01:07:04
  212. Executing: python train.py -f experiments/impala-4x-gpu.yaml --ray-memory 60129542144 --ray-num-cpus 16 --ray-object-store-memory 30000000000
  213. 2020-07-07 01:07:04
  214.  
  215. 2020-07-07 01:07:04
  216. /_/ \_\_____\___|_| \___/ \_/\_/ \__,_|
  217. 2020-07-07 01:07:04
  218. / ____ \ _| || (__| | | (_) \ V V / (_| |
  219. 2020-07-07 01:07:04
  220. / /\ \ | | / __| '__/ _ \ \ /\ / / _ |
  221. 2020-07-07 01:07:04
  222. / \ | | ___ _ __ _____ ____| |
  223. 2020-07-07 01:07:04
  224. /\ |_ _| | |
  225. 2020-07-07 01:07:04
  226. _____ _
RAW Paste Data