W1206 03:22:37.720000 1782551 site-packages/torch/distributed/run.py:803]
W1206 03:22:37.720000 1782551 site-packages/torch/distributed/run.py:803] *****************************************
W1206 03:22:37.720000 1782551 site-packages/torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1206 03:22:37.720000 1782551 site-packages/torch/distributed/run.py:803] *****************************************

Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]
Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]
Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead!
You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers

Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead!


Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 56.89it/s]

Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 55.74it/s]

Loading pipeline components...:  29%|██▊       | 2/7 [00:00<00:00,  5.90it/s]
Loading pipeline components...:  29%|██▊       | 2/7 [00:00<00:00,  6.13it/s]
Loading pipeline components...:  14%|█▍        | 1/7 [00:00<00:02,  2.93it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers


Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading pipeline components...:  14%|█▍        | 1/7 [00:00<00:02,  2.27it/s]`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 43.59it/s]

Loading pipeline components...:  43%|████▎     | 3/7 [00:00<00:00,  6.67it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers


Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading pipeline components...:  43%|████▎     | 3/7 [00:00<00:00,  5.42it/s]
Loading pipeline components...:  29%|██▊       | 2/7 [00:00<00:01,  3.83it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 57.52it/s]

Loading pipeline components...:  57%|█████▋    | 4/7 [00:00<00:00,  8.26it/s]

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 44.33it/s]

Loading pipeline components...:  57%|█████▋    | 4/7 [00:00<00:00,  6.05it/s]`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 42.07it/s]


Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading pipeline components...:  57%|█████▋    | 4/7 [00:00<00:00,  6.65it/s]
Loading pipeline components...:  71%|███████▏  | 5/7 [00:00<00:00,  7.13it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 43.73it/s]

Loading pipeline components...:  86%|████████▌ | 6/7 [00:00<00:00,  8.82it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00,  8.57it/s]

Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 57.05it/s]

Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00, 10.37it/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00,  8.36it/s]

Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00,  9.79it/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00,  8.45it/s]

Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00, 10.97it/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00,  8.17it/s]
Attention backends are an experimental feature and the API may be subject to change.
`enable_parallelism` is an experimental feature. The API may change in the future and breaking changes may be introduced at any time without warning.
Attention backends are an experimental feature and the API may be subject to change.
`enable_parallelism` is an experimental feature. The API may change in the future and breaking changes may be introduced at any time without warning.

  0%|          | 0/50 [00:00<?, ?it/s]
  0%|          | 0/50 [00:00<?, ?it/s]Attention backends are an experimental feature and the API may be subject to change.
`enable_parallelism` is an experimental feature. The API may change in the future and breaking changes may be introduced at any time without warning.
Attention backends are an experimental feature and the API may be subject to change.
`enable_parallelism` is an experimental feature. The API may change in the future and breaking changes may be introduced at any time without warning.

  0%|          | 0/50 [00:00<?, ?it/s]
  0%|          | 0/50 [00:00<?, ?it/s]
  0%|          | 0/50 [00:01<?, ?it/s]
  0%|          | 0/50 [00:01<?, ?it/s]


  0%|          | 0/50 [00:03<?, ?it/s]
  0%|          | 0/50 [00:02<?, ?it/s]

[rank3]: Traceback (most recent call last):
[rank3]:   File "/fsx/sayak/diffusers/check_unified_attention.py", line 30, in <module>
[rank3]:     image = pipeline(prompt, guidance_scale=3.5, num_inference_steps=50, generator=generator).images[0]
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank3]:     return func(*args, **kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 944, in __call__
[rank3]:     noise_pred = self.transformer(
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/hooks/hooks.py", line 189, in new_forward
[rank3]:     output = function_reference.forward(*args, **kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 743, in forward
[rank3]:     encoder_hidden_states, hidden_states = block(
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 453, in forward
[rank3]:     attention_outputs = self.attn(
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 352, in forward
[rank3]:     return self.processor(self, hidden_states, encoder_hidden_states, attention_mask, image_rotary_emb, **kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 118, in __call__
[rank3]:     hidden_states = dispatch_attention_fn(
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 351, in dispatch_attention_fn
[rank3]:     return backend_fn(**kwargs)
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 2012, in _native_cudnn_attention
[rank3]:     out = _templated_context_parallel_attention(
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1407, in _templated_context_parallel_attention
[rank3]:     return TemplatedUnifiedAttention(
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1349, in TemplatedUnifiedAttention
[rank3]:     out = TemplatedRingAttention.apply(
[rank3]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/autograd/function.py", line 581, in apply
[rank3]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank3]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1168, in forward
[rank3]:     out = prev_out - torch.nn.functional.sigmoid(lse - prev_lse) * (prev_out - out)
[rank3]: RuntimeError: The size of tensor a (12) must match the size of tensor b (2304) at non-singleton dimension 2
[rank2]: Traceback (most recent call last):
[rank2]:   File "/fsx/sayak/diffusers/check_unified_attention.py", line 30, in <module>
[rank2]:     image = pipeline(prompt, guidance_scale=3.5, num_inference_steps=50, generator=generator).images[0]
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank2]:     return func(*args, **kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 944, in __call__
[rank2]:     noise_pred = self.transformer(
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank2]:     return self._call_impl(*args, **kwargs)
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank2]:     return forward_call(*args, **kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/hooks/hooks.py", line 189, in new_forward
[rank2]:     output = function_reference.forward(*args, **kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 743, in forward
[rank2]:     encoder_hidden_states, hidden_states = block(
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank2]:     return self._call_impl(*args, **kwargs)
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank2]:     return forward_call(*args, **kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 453, in forward
[rank2]:     attention_outputs = self.attn(
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank2]:     return self._call_impl(*args, **kwargs)
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank2]:     return forward_call(*args, **kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 352, in forward
[rank2]:     return self.processor(self, hidden_states, encoder_hidden_states, attention_mask, image_rotary_emb, **kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 118, in __call__
[rank2]:     hidden_states = dispatch_attention_fn(
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 351, in dispatch_attention_fn
[rank2]:     return backend_fn(**kwargs)
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 2012, in _native_cudnn_attention
[rank2]:     out = _templated_context_parallel_attention(
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1407, in _templated_context_parallel_attention
[rank2]:     return TemplatedUnifiedAttention(
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1349, in TemplatedUnifiedAttention
[rank2]:     out = TemplatedRingAttention.apply(
[rank2]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/autograd/function.py", line 581, in apply
[rank2]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank2]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1168, in forward
[rank2]:     out = prev_out - torch.nn.functional.sigmoid(lse - prev_lse) * (prev_out - out)
[rank2]: RuntimeError: The size of tensor a (12) must match the size of tensor b (2304) at non-singleton dimension 2
[rank1]: Traceback (most recent call last):
[rank1]:   File "/fsx/sayak/diffusers/check_unified_attention.py", line 30, in <module>
[rank1]:     image = pipeline(prompt, guidance_scale=3.5, num_inference_steps=50, generator=generator).images[0]
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank1]:     return func(*args, **kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 944, in __call__
[rank1]:     noise_pred = self.transformer(
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/hooks/hooks.py", line 189, in new_forward
[rank1]:     output = function_reference.forward(*args, **kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 743, in forward
[rank1]:     encoder_hidden_states, hidden_states = block(
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 453, in forward
[rank1]:     attention_outputs = self.attn(
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 352, in forward
[rank1]:     return self.processor(self, hidden_states, encoder_hidden_states, attention_mask, image_rotary_emb, **kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 118, in __call__
[rank1]:     hidden_states = dispatch_attention_fn(
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 351, in dispatch_attention_fn
[rank1]:     return backend_fn(**kwargs)
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 2012, in _native_cudnn_attention
[rank1]:     out = _templated_context_parallel_attention(
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1407, in _templated_context_parallel_attention
[rank1]:     return TemplatedUnifiedAttention(
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1349, in TemplatedUnifiedAttention
[rank1]:     out = TemplatedRingAttention.apply(
[rank1]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/autograd/function.py", line 581, in apply
[rank1]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank1]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1168, in forward
[rank1]:     out = prev_out - torch.nn.functional.sigmoid(lse - prev_lse) * (prev_out - out)
[rank1]: RuntimeError: The size of tensor a (12) must match the size of tensor b (2304) at non-singleton dimension 2
[rank0]: Traceback (most recent call last):
[rank0]:   File "/fsx/sayak/diffusers/check_unified_attention.py", line 30, in <module>
[rank0]:     image = pipeline(prompt, guidance_scale=3.5, num_inference_steps=50, generator=generator).images[0]
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 944, in __call__
[rank0]:     noise_pred = self.transformer(
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/hooks/hooks.py", line 189, in new_forward
[rank0]:     output = function_reference.forward(*args, **kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 743, in forward
[rank0]:     encoder_hidden_states, hidden_states = block(
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 453, in forward
[rank0]:     attention_outputs = self.attn(
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 352, in forward
[rank0]:     return self.processor(self, hidden_states, encoder_hidden_states, attention_mask, image_rotary_emb, **kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/transformers/transformer_flux.py", line 118, in __call__
[rank0]:     hidden_states = dispatch_attention_fn(
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 351, in dispatch_attention_fn
[rank0]:     return backend_fn(**kwargs)
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 2012, in _native_cudnn_attention
[rank0]:     out = _templated_context_parallel_attention(
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1407, in _templated_context_parallel_attention
[rank0]:     return TemplatedUnifiedAttention(
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1349, in TemplatedUnifiedAttention
[rank0]:     out = TemplatedRingAttention.apply(
[rank0]:   File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/autograd/function.py", line 581, in apply
[rank0]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank0]:   File "/fsx/sayak/diffusers/src/diffusers/models/attention_dispatch.py", line 1168, in forward
[rank0]:     out = prev_out - torch.nn.functional.sigmoid(lse - prev_lse) * (prev_out - out)
[rank0]: RuntimeError: The size of tensor a (12) must match the size of tensor b (2304) at non-singleton dimension 2
[rank1]:[W1206 03:23:03.518618524 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank0]:[W1206 03:23:03.570948018 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank2]:[W1206 03:23:04.916631126 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
W1206 03:23:09.585000 1782551 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1782613 closing signal SIGTERM
W1206 03:23:09.589000 1782551 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1782614 closing signal SIGTERM
W1206 03:23:09.589000 1782551 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1782616 closing signal SIGTERM
E1206 03:23:10.668000 1782551 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 2 (pid: 1782615) of binary: /fsx/sayak/miniconda3/envs/diffusers/bin/python
Traceback (most recent call last):
  File "/fsx/sayak/miniconda3/envs/diffusers/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
check_unified_attention.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-12-06_03:23:09
  host      : ip-26-0-160-103.ec2.internal
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 1782615)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================