2020-07-07 03:09:15
2020-07-07 01:09:15,476	WARNING worker.py:1090 -- A worker died or was killed while executing task ffffffffffffffff4d81fd5d0100.
2020-07-07 03:09:15

2020-07-07 03:09:15

2020-07-07 03:09:15
+-------------------------------+------------+-------+--------+------------------+-------+----------+
2020-07-07 03:09:15
| PPO_procgen_env_wrapper_00000 | TERMINATED |       |      3 |          7318.15 | 55296 |      3.8 |
2020-07-07 03:09:15
|-------------------------------+------------+-------+--------+------------------+-------+----------|
2020-07-07 03:09:15
| Trial name                    | status     | loc   |   iter |   total time (s) |    ts |   reward |
2020-07-07 03:09:15
+-------------------------------+------------+-------+--------+------------------+-------+----------+
2020-07-07 03:09:15
Number of trials: 1 (1 TERMINATED)
2020-07-07 03:09:15
Result logdir: /outputs/ray-results/procgen-ppo
2020-07-07 03:09:15
Resources requested: 0/16 CPUs, 0.0/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
2020-07-07 03:09:15
Using FIFO scheduling algorithm.
2020-07-07 03:09:15
Memory usage on this node: 7.1/59.0 GiB
2020-07-07 03:09:15
== Status ==
2020-07-07 03:09:15

2020-07-07 03:09:15

2020-07-07 03:09:15
+-------------------------------+----------+---------------+--------+------------------+-------+----------+
2020-07-07 03:09:15
| PPO_procgen_env_wrapper_00000 | RUNNING  | 10.60.3.12:99 |      3 |          7318.15 | 55296 |      3.8 |
2020-07-07 03:09:15
|-------------------------------+----------+---------------+--------+------------------+-------+----------|
2020-07-07 03:09:15
| Trial name                    | status   | loc           |   iter |   total time (s) |    ts |   reward |
2020-07-07 03:09:15
+-------------------------------+----------+---------------+--------+------------------+-------+----------+
2020-07-07 03:09:15
Number of trials: 1 (1 RUNNING)
2020-07-07 03:09:15
Result logdir: /outputs/ray-results/procgen-ppo
2020-07-07 03:09:15
Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
2020-07-07 03:09:15
Using FIFO scheduling algorithm.
2020-07-07 03:09:15
Memory usage on this node: 7.1/59.0 GiB
2020-07-07 03:09:15
== Status ==
2020-07-07 03:09:15
[Errno 2] No such file or directory: 'merged-videos/training.mp4'
2020-07-07 03:09:15

2020-07-07 03:09:15

2020-07-07 03:09:15
+-------------------------------+----------+---------------+--------+------------------+-------+----------+
2020-07-07 03:09:15
| PPO_procgen_env_wrapper_00000 | RUNNING  | 10.60.3.12:99 |      2 |          4900.67 | 36864 |     3.75 |
2020-07-07 03:09:15
|-------------------------------+----------+---------------+--------+------------------+-------+----------|
2020-07-07 03:09:15
| Trial name                    | status   | loc           |   iter |   total time (s) |    ts |   reward |
2020-07-07 03:09:15
+-------------------------------+----------+---------------+--------+------------------+-------+----------+
2020-07-07 03:09:15
Number of trials: 1 (1 RUNNING)
2020-07-07 03:09:15
Result logdir: /outputs/ray-results/procgen-ppo
2020-07-07 03:09:15
Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
2020-07-07 03:09:15
Using FIFO scheduling algorithm.
2020-07-07 03:09:15
Memory usage on this node: 7.0/59.0 GiB
2020-07-07 03:09:15
== Status ==
2020-07-07 03:09:15
[Errno 2] No such file or directory: 'merged-videos/training.mp4'
2020-07-07 03:09:15

2020-07-07 03:09:15

2020-07-07 03:09:15
+-------------------------------+----------+---------------+--------+------------------+-------+----------+
2020-07-07 03:09:15
| PPO_procgen_env_wrapper_00000 | RUNNING  | 10.60.3.12:99 |      1 |           2469.2 | 18432 |  4.16667 |
2020-07-07 03:09:15
|-------------------------------+----------+---------------+--------+------------------+-------+----------|
2020-07-07 03:09:15
| Trial name                    | status   | loc           |   iter |   total time (s) |    ts |   reward |
2020-07-07 03:09:15
+-------------------------------+----------+---------------+--------+------------------+-------+----------+
2020-07-07 03:09:15
Number of trials: 1 (1 RUNNING)
2020-07-07 03:09:15
Result logdir: /outputs/ray-results/procgen-ppo
2020-07-07 03:09:15
Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
2020-07-07 03:09:15
Using FIFO scheduling algorithm.
2020-07-07 03:09:15
Memory usage on this node: 6.9/59.0 GiB
2020-07-07 03:09:15
== Status ==
2020-07-07 03:09:15
[Errno 2] No such file or directory: 'merged-videos/training.mp4'
2020-07-07 03:09:15
(pid=104) /pytorch/torch/csrc/utils/tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.
2020-07-07 03:09:15
(pid=99) 2020-07-06 23:07:13,612	WARNING util.py:37 -- Install gputil for GPU system monitoring.
2020-07-07 03:09:15
(pid=99) 2020-07-06 23:07:13,612	INFO trainable.py:217 -- Getting current IP.
2020-07-07 03:09:15
(pid=99) 2020-07-06 23:07:11,633	INFO trainer.py:580 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
2020-07-07 03:09:15
(pid=99) 2020-07-06 23:07:11,628	INFO trainer.py:421 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2020-07-07 03:09:15

2020-07-07 03:09:15

2020-07-07 03:09:15
+-------------------------------+----------+-------+
2020-07-07 03:09:15
| PPO_procgen_env_wrapper_00000 | RUNNING  |       |
2020-07-07 03:09:15
|-------------------------------+----------+-------|
2020-07-07 03:09:15
| Trial name                    | status   | loc   |
2020-07-07 03:09:15
+-------------------------------+----------+-------+
2020-07-07 03:09:15
Number of trials: 1 (1 RUNNING)
2020-07-07 03:09:15
Result logdir: /outputs/ray-results/procgen-ppo
2020-07-07 03:09:15
Resources requested: 11/16 CPUs, 0.9/1 GPUs, 0.0/55.96 GiB heap, 0.0/19.24 GiB objects
2020-07-07 03:09:15
Using FIFO scheduling algorithm.
2020-07-07 03:09:15
Memory usage on this node: 2.7/59.0 GiB
2020-07-07 03:09:15
== Status ==
2020-07-07 03:09:15
{'contrib/RandomAgent': <function _import_random_agent at 0x7f3a30feddd0>, 'contrib/MADDPG': <function _import_maddpg at 0x7f3a30f94a70>, 'contrib/AlphaZero': <function _import_alphazero at 0x7f3a30f94d40>, 'contrib/LinTS': <function _import_bandit_lints at 0x7f3a30f94dd0>, 'contrib/LinUCB': <function _import_bandit_linucb at 0x7f3a30f94e60>}
2020-07-07 03:09:15
2020-07-07 01:09:15,224	WARNING util.py:137 -- The `process_trial` operation took 1.1798417568206787 seconds to complete, which may be a performance bottleneck.
2020-07-07 02:28:56
2020-07-07 00:28:56,565	WARNING util.py:137 -- The `process_trial` operation took 0.9509642124176025 seconds to complete, which may be a performance bottleneck.
2020-07-07 01:48:24
2020-07-06 23:48:24,132	WARNING util.py:137 -- The `process_trial` operation took 1.3134164810180664 seconds to complete, which may be a performance bottleneck.
2020-07-07 01:48:24
  if isinstance(v, collections.MutableMapping):
2020-07-07 01:48:24
train.py:117: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
2020-07-07 01:07:09
2020-07-06 23:07:09,465	ERROR syncer.py:39 -- Log sync requires rsync to be installed.
2020-07-07 01:07:08

2020-07-07 01:07:08
OSError: [Errno 99] error while attempting to bind on address ('::1', 8265, 0, 0): cannot assign requested address
2020-07-07 01:07:08
    % (sa, err.strerror.lower())) from None
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/asyncio/base_events.py", line 1385, in create_server
2020-07-07 01:07:08
    reuse_port=self._reuse_port)
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/site-packages/aiohttp/web_runner.py", line 104, in start
2020-07-07 01:07:08
    await site.start()
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/site-packages/aiohttp/web.py", line 359, in _run_app
2020-07-07 01:07:08
    return future.result()
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
2020-07-07 01:07:08
    reuse_port=reuse_port))
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/site-packages/aiohttp/web.py", line 433, in run_app
2020-07-07 01:07:08
    aiohttp.web.run_app(self.app, host=self.host, port=self.port)
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/site-packages/ray/dashboard/dashboard.py", line 594, in run
2020-07-07 01:07:08
    dashboard.run()
2020-07-07 01:07:08
  File "/home/aicrowd/.conda/lib/python3.7/site-packages/ray/dashboard/dashboard.py", line 1220, in <module>
2020-07-07 01:07:08
Traceback (most recent call last):
2020-07-07 01:07:08
2020-07-06 23:07:08,749	WARNING worker.py:1090 -- The dashboard on node submission-gsz5w-2446593925 failed with the following error:
2020-07-07 01:07:08
ls: cannot access '/outputs/ray-results/procgen-ppo/*/': No such file or directory
2020-07-07 01:07:07
2020-07-06 23:07:07,698	WARNING services.py:1494 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This may slow down performance! You may be able to free up space by deleting files in /dev/shm or terminating any running plasma_store_server processes. If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size' to 'docker run'.
2020-07-07 01:07:07
2020-07-06 23:07:07,695	INFO services.py:1170 -- View the Ray dashboard at localhost:8265
2020-07-07 01:07:07
2020-07-06 23:07:07,316	INFO resource_spec.py:212 -- Starting Ray with 55.96 GiB memory available for workers and up to 27.94 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-07-07 01:07:07
  import imp
2020-07-07 01:07:07
/home/aicrowd/.conda/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
2020-07-07 01:07:04
Executing: python train.py -f experiments/impala-4x-gpu.yaml --ray-memory 60129542144 --ray-num-cpus 16 --ray-object-store-memory 30000000000
2020-07-07 01:07:04

2020-07-07 01:07:04
 /_/    \_\_____\___|_|  \___/ \_/\_/ \__,_|
2020-07-07 01:07:04
  / ____ \ _| || (__| | | (_) \ V  V / (_| |
2020-07-07 01:07:04
   / /\ \   | | / __| '__/ _ \ \ /\ / / _  |
2020-07-07 01:07:04
    /  \    | |  ___ _ __ _____      ____| |
2020-07-07 01:07:04
     /\   |_   _|                        | |
2020-07-07 01:07:04
           _____                          _