Hi all, I am currently working on a multi-threaded hybrid parallel simulation which uses both pthreads and OpenMPI. The simulation uses several pthreads per MPI node.
My code uses the nonblocking routines MPI_Isend/MPI_Irecv/MPI_Waitany quite successfully to implement the node-to-node communication. When I try to interleave other computations during this communication, however, bad things happen. I have two MPI nodes with two threads each: one thread (i) doing the nonblocking communication and the other (ii) doing other computations. At some point, the threads (ii) need to exchange data using MPI_Allreduce, which fails if the first thread (i) has not completed all the communication, i.e. if thread (i) is still in MPI_Waitany. Using the in-place MPI_Allreduce, I get a re-run of this bug: http://www.open-mpi.org/community/lists/users/2011/09/17432.php. If I don't use in-place, the call to MPI_Waitany (thread ii) on one of the MPI nodes waits forever. My guess is that when the thread (ii) calls MPI_Allreduce, it gets whatever the other node sent with MPI_Isend to thread (i), drops whatever it should have been getting from the other node's MPI_Allreduce, and the call to MPI_Waitall hangs. Is this a known issue? Is MPI_Allreduce not designed to work alongside the nonblocking routines? Is there a "safe" variant of MPI_Allreduce I should be using instead? I am using OpenMPI version 1.4.3 (version 1.4.3-1ubuntu3 of the package openmpi-bin in Ubuntu). Both MPI nodes are run on the same dual-core computer (Lenovo x201 laptop). If you need more information, please do let me know! I'll also try to cook-up a small program reproducing this problem... Cheers and kind regards, Pedro
