A process or rank is not allowed to participate multiple times in the same group (at least not in the current version of the MPI standard). The sentence about "dual membership" you pointed out makes sense only for inter-communicators (and the paragraph where the sentence is located clearly talks about local and remote groups). Moreover, and this is something the MPI standard is not clear about, such a sentence only makes sense when the local_leader of the 2 groups is not the same process.
The problem with your approach is that you have in the peer_comm the same process as being the leader of the 2 groups. In the implementation of the MPI_Intercomm_create there is a need for a handshake between the 2 leaders to exchange the sizes of their respective groups, and this handshake is using the TAG you provided. The problem (at least in Open MPI) is that the handshake is implemented as (Irecv + Send + Wait) with the same rank on the peer_comm. Obviously such a communication pattern will not successfully exchange the data between the peers in most cases (in the same communicator the matching is guaranteed to be in FIFO order by the MPI standard). So the complete answer is: It can be done but not in the context you are trying to achieve. George. On Mon, May 15, 2017 at 10:22 AM, Clune, Thomas L. (GSFC-6101) < thomas.l.cl...@nasa.gov> wrote: > I am trying to craft a client-server layer that needs to have 2 different > modes of operation. In the “remote server” mode, then the server runs on > distinct processes, and intercommunicator is a perfect fit for my design. > In the “local server” the server will actually run on a dedicate thread > within the same pool of processes that are running the client. What I > would like is some analog of an intercommunicator that would operate > between two comms that are essentially identical (but distinct due to use > of MPI_Comm_dup()), with one running on thread 0 and the other running on > thread 1. > > Now, I know the language in the standard sounds like it explicitly forbids > such overlapping groups: > > "Overlap of local and remote groups that are bound into an > inter-communicator is prohibited.” > > But there is then that hint of wiggle room with the parenthetical: > > “(If a process is multithreaded, and MPI calls block only a thread, > rather than a process, then “dual membership” can be supported. It is > then the user’s responsibility to make sure that calls on behalf of the two > “roles” of a process are executed by two independent threads.)" > > This sounds very much like what I want to do, but the following example > code crashes (apologies for Fortran source). It is intended to be run on > two threads - each of which creates a dup of MPI_COMM_WORLD. The > intercomm create would then have the “other” comm as the remote for each > thread. I’m not wed to the details here. Anything that would > support the notion of an MPI based server running on extra threads would be > a potential solution. I know that in the worst case, I could achieve the > same by oversubscribing processes on my nodes, but that pushes some issues > in our batch processing system that I’d prefer to avoid. A definitive “ > Can’t be done. Move along.” would also be useful. > > The following fails with OpenMPI 2.1 on OS X 10.12: > > program main > use mpi > implicit none > > integer, external :: omp_get_thread_num > integer :: ierror > integer :: comms(0:1) > integer :: comm > integer :: tag = 10 > integer :: intercomm > integer :: provided > integer :: i > > call MPI_Init_thread(MPI_THREAD_MULTIPLE, provided, ierror) > call MPI_Comm_dup(MPI_COMM_WORLD, comm, ierror) > > !$omp parallel num_threads(2) default(none), private(i, intercomm, > ierror), shared(tag, comms, comm) > i = omp_get_thread_num() > > if (i == 0) then > call MPI_Comm_dup(MPI_COMM_WORLD, comms(i), ierror) > end if > !$omp barrier > if (i == 1) then > call MPI_Comm_dup(MPI_COMM_WORLD, comms(i), ierror) > end if > !$omp barrier > call MPI_Intercomm_create(comms(i), 0, comm, 0, tag, intercomm, > ierror) > > !$omp end parallel > > stop > call MPI_Finalize(ierror) > end program main > > > Thanks in advance. > > - Tom > > > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users >
_______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
