Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- *****************************************
- Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
- *****************************************
- init pg, pid: 16652
- init pg, pid: 16651
- step : 0, pid: 16651
- Traceback (most recent call last):
- File "./trainer.py", line 153, in <module>
- main()
- File "./trainer.py", line 140, in main
- raise e
- File "./trainer.py", line 137, in main
- return train(env)
- File "./trainer.py", line 110, in train
- raise Exception("test exception")
- Exception: test exception
- step : 0, pid: 16652
- 16652 starting 0 batch
- WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 16652 closing signal SIGTERM
- ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 16651) of binary: /home/ubuntu/miniconda3/envs/aivanou-dev/bin/python
- Traceback (most recent call last):
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/runpy.py", line 194, in _run_module_as_main
- return _run_code(code, main_globals, None,
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/runpy.py", line 87, in _run_code
- exec(code, run_globals)
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/site-packages/torch/distributed/run.py", line 723, in <module>
- main()
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
- return f(*args, **kwargs)
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/site-packages/torch/distributed/run.py", line 719, in main
- run(args)
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
- elastic_launch(
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
- return launch_agent(self._config, self._entrypoint, list(args))
- File "/home/ubuntu/miniconda3/envs/aivanou-dev/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 276, in launch_agent
- raise ChildFailedError(
- torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
- ============================================================
- ./trainer.py FAILED
- ------------------------------------------------------------
- Failures:
- <NO_OTHER_FAILURES>
- ------------------------------------------------------------
- Root Cause (first observed failure):
- [0]:
- time : 2022-02-08_16:33:51
- host : ip-172-31-24-178.us-west-2.compute.internal
- rank : 0 (local_rank: 0)
- exitcode : 1 (pid: 16651)
- error_file: <N/A>
- traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
- ============================================================
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement