Hi all, I am encountering a silent hang involving MPI_Ssend and MPI_Irecv. The subroutine in question is called by each processor and is structured similar to the pseudo code below. The subroutine is successfully called several thousand times before the silent hang behavior manifests and never resolves. The hang will occur in nearly (but not exactly) the same spot for bit-wise identical tests. During the hang, all MPI ranks will be at the Line 18 Barrier except for two. One will be waiting at Line 17, waiting for its Irecv to complete, and the other at one of the Ssend Line 9 or 14. This suggests that a MPI_Irecv never completes and a processor is indefinitely blocked in the Ssend unable to complete the transfer.
I’ve found similar discussion of this kind of behavior on the OpenMPI mailing list: https://www.mail-archive.com/users@lists.open-mpi.org/msg19227.html ultimately resolving in setting the mca parameter btl_openib_flags to 304 or 305 (default 310): https://www.mail-archive.com/users@lists.open-mpi.org/msg19277.html. I have seen some promising behavior by doing the same. As the mailer suggests, this implies a problem with the RDMA protocols in infiniband for large messages. I wanted to breathe life back into this conversation as the silent hang issue is particularly debilitating and confusing to me. Increasing/decreasing the number of processors used does not seem to alleviate the issue, using MPI_Send results in the same behavior, perhaps a message has exceeded a memory limit? I am running a test now that reports the individual message sizes but I previously implemented a switch to check for buffer size discrepancies which is not triggered. In the meantime, has anyone run into similar issues or have thoughts as to remedies for this behavior? 1: call MPI_BARRIER(…) 2: do i = 1,nprocs 3: if(commatrix_recv(i) .gt. 0) then ! Identify which procs to receive from via predefined matrix 4: call Mpi_Irecv(…) 5: endif 6: enddo 7: do j = mype+1,nproc 8: if(commatrix_send(j) .gt. 0) then ! Identify which procs to send to via predefined matrix 9: MPI_Ssend(…) 10: endif 11: enddo 12: do j = 1,mype 13: if(commatrix_send(j) .gt. 0) then ! Identify which procs to send to via predefined matrix 14: MPI_Ssend(…) 15: endif 16: enddo 17: call MPI_Waitall(…) ! Wait for all Irecv to complete 18: call MPI_Barrier(…) Cluster information: 30 processors Managed by slurm OS: Red Hat v. 7.7 Thank you for help/advice you can provide, Sean Sean C. Lewis Doctoral Candidate Department of Physics Drexel University
