GTL_DEBUG: [2] hsa_amd_ipc_memory_attach: HSA_STATUS_ERROR_INVALID_ARGUMENT: One of the actual arguments does not meet a precondition stated in the documentation of the corresponding formal argument.
GTL_DEBUG: [3] hsa_amd_ipc_memory_attach: HSA_STATUS_ERROR_INVALID_ARGUMENT: One of the actual arguments does not meet a precondition stated in the documentation of the corresponding formal argument.
MPICH ERROR [Rank 2] [job id 3654896.0] [Thu Jun  8 19:05:54 2023] [nid007563] - Abort(742508546) (rank 2 in comm 0): Fatal error in PMPI_Recv: Invalid count, error stack:
PMPI_Recv(177)............................: MPI_Recv(buf=0x149dae200000, count=921600, MPI_DOUBLE, src=0, tag=4, MPI_COMM_WORLD, status=0x1) failed
MPIR_Wait_impl(41)........................:
MPID_Progress_wait(193)...................:
MPIDI_Progress_test(97)...................:
MPIDI_SHMI_progress(118)..................:
MPIDI_POSIX_progress(422).................:
MPIDI_CRAY_Common_lmt_ctrl_send_rts_cb(64):
MPIDI_CRAY_Common_lmt_handle_recv(44).....:
MPIDI_CRAY_Common_lmt_import_mem(218).....:
(unknown)(): Invalid count

aborting job:
Fatal error in PMPI_Recv: Invalid count, error stack:
PMPI_Recv(177)............................: MPI_Recv(buf=0x149dae200000, count=921600, MPI_DOUBLE, src=0, tag=4, MPI_COMM_WORLD, status=0x1) failed
MPIR_Wait_impl(41)........................:
MPID_Progress_wait(193)...................:
MPIDI_Progress_test(97)...................:
MPIDI_SHMI_progress(118)..................:
MPIDI_POSIX_progress(422).................:
MPIDI_CRAY_Common_lmt_ctrl_send_rts_cb(64):
MPIDI_CRAY_Common_lmt_handle_recv(44).....:
MPIDI_CRAY_Common_lmt_import_mem(218).....:
(unknown)(): Invalid count
MPICH ERROR [Rank 3] [job id 3654896.0] [Thu Jun  8 19:05:54 2023] [nid007563] - Abort(1010944002) (rank 3 in comm 0): Fatal error in PMPI_Recv: Invalid count, error stack:
PMPI_Recv(177)............................: MPI_Recv(buf=0x14f482c00000, count=921600, MPI_DOUBLE, src=1, tag=5, MPI_COMM_WORLD, status=0x1) failed
MPIR_Wait_impl(41)........................:
MPID_Progress_wait(193)...................:
MPIDI_Progress_test(97)...................:
MPIDI_SHMI_progress(118)..................:
MPIDI_POSIX_progress(422).................:
MPIDI_CRAY_Common_lmt_ctrl_send_rts_cb(64):
MPIDI_CRAY_Common_lmt_handle_recv(44).....:
MPIDI_CRAY_Common_lmt_import_mem(218).....:
(unknown)(): Invalid count

aborting job:
Fatal error in PMPI_Recv: Invalid count, error stack:
PMPI_Recv(177)............................: MPI_Recv(buf=0x14f482c00000, count=921600, MPI_DOUBLE, src=1, tag=5, MPI_COMM_WORLD, status=0x1) failed
MPIR_Wait_impl(41)........................:
MPID_Progress_wait(193)...................:
MPIDI_Progress_test(97)...................:
MPIDI_SHMI_progress(118)..................:
MPIDI_POSIX_progress(422).................:
MPIDI_CRAY_Common_lmt_ctrl_send_rts_cb(64):
MPIDI_CRAY_Common_lmt_handle_recv(44).....:
MPIDI_CRAY_Common_lmt_import_mem(218).....:
(unknown)(): Invalid count
srun: error: nid007563: tasks 2-3: Exited with exit code 255
srun: launch/slurm: _step_signal: Terminating StepId=3654896.0
slurmstepd: error: *** STEP 3654896.0 ON nid007563 CANCELLED AT 2023-06-08T19:05:55 ***
srun: error: nid007563: tasks 0-1,4-7: Terminated
srun: Force Terminated StepId=3654896.0