The only MPI calls I am using are these (grep-ed from my code): MPI_Abort(MPI_COMM_WORLD, 1); MPI_Barrier(MPI_COMM_WORLD); MPI_Bcast(&bufarray[0].hdr, sizeof(BD_CHDR), MPI_CHAR, 0, MPI_COMM_WORLD); MPI_Comm_rank(MPI_COMM_WORLD,&myid); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Finalize(); MPI_Init(&argc, &argv); MPI_Irecv( MPI_Isend( MPI_Recv(buff, BUFSIZE, MPI_CHAR, 0, TAG, MPI_COMM_WORLD, &stat); MPI_Send(buff, BUFSIZE, MPI_CHAR, 0, TAG, MPI_COMM_WORLD); MPI_Test(&request, &complete, &status); MPI_Wait(&request, &status);
The big wait happens on receipt of a bcast call that would otherwise work. Its a bit mysterious really... I presume that bcast is implemented with multicast calls but does it use any actual broadcast calls at all? I know I'm scraping the edges here looking for something but I just cant get my head around why it should fail where it has. --- On Mon, 9/8/10, Ralph Castain <r...@open-mpi.org> wrote: From: Ralph Castain <r...@open-mpi.org> Subject: Re: [OMPI users] MPI_Bcast issue To: "Open MPI Users" <us...@open-mpi.org> Received: Monday, 9 August, 2010, 1:32 PM Hi Randolph Unless your code is doing a connect/accept between the copies, there is no way they can cross-communicate. As you note, mpirun instances are completely isolated from each other - no process in one instance can possibly receive information from a process in another instance because it lacks all knowledge of it -unless- they wireup into a greater communicator by performing connect/accept calls between them. I suspect you are inadvertently doing just that - perhaps by doing connect/accept in a tree-like manner, not realizing that the end result is one giant communicator that now links together all the N servers. Otherwise, there is no possible way an MPI_Bcast in one mpirun can collide or otherwise communicate with an MPI_Bcast between processes started by another mpirun. On Aug 8, 2010, at 7:13 PM, Randolph Pullen wrote: Thanks, although “An intercommunicator cannot be used for collective communication.” i.e , bcast calls., I can see how the MPI_Group_xx calls can be used to produce a useful group and then communicator; - thanks again but this is really the side issue to my main question about MPI_Bcast. I seem to have duplicate concurrent processes interfering with each other. This would appear to be a breach of the MPI safety dictum, ie MPI_COMM_WORD is supposed to only include the processes started by a single mpirun command and isolate these processes from other similar groups of processes safely. So, it would appear to be a bug. If so this has significant implications for environments such as mine, where it may often occur that the same program is run by different users simultaneously. It is really this issue that it concerning me, I can rewrite the code but if it can crash when 2 copies run at the same time, I have a much bigger problem. My suspicion is that a within the MPI_Bcast handshaking, a syncronising broadcast call may be colliding across the environments. My only evidence is an otherwise working program waits on broadcast reception forever when two or more copies are run at [exactly] the same time. Has anyone else seen similar behavior in concurrently running programs that perform lots of broadcasts perhaps? Randolph --- On Sun, 8/8/10, David Zhang <solarbik...@gmail.com> wrote: From: David Zhang <solarbik...@gmail.com> Subject: Re: [OMPI users] MPI_Bcast issue To: "Open MPI Users" <us...@open-mpi.org> Received: Sunday, 8 August, 2010, 12:34 PM In particular, intercommunicators On 8/7/10, Aurélien Bouteiller <boute...@eecs.utk.edu> wrote: > You should consider reading about communicators in MPI. > > Aurelien > -- > Aurelien Bouteiller, Ph.D. > Innovative Computing Laboratory, The University of Tennessee. > > Envoyé de mon iPad > > Le Aug 7, 2010 à 1:05, Randolph Pullen <randolph_pul...@yahoo.com.au> a > écrit : > >> I seem to be having a problem with MPI_Bcast. >> My massive I/O intensive data movement program must broadcast from n to n >> nodes. My problem starts because I require 2 processes per node, a sender >> and a receiver and I have implemented these using MPI processes rather >> than tackle the complexities of threads on MPI. >> >> Consequently, broadcast and calls like alltoall are not completely >> helpful. The dataset is huge and each node must end up with a complete >> copy built by the large number of contributing broadcasts from the sending >> nodes. Network efficiency and run time are paramount. >> >> As I don’t want to needlessly broadcast all this data to the sending nodes >> and I have a perfectly good MPI program that distributes globally from a >> single node (1 to N), I took the unusual decision to start N copies of >> this program by spawning the MPI system from the PVM system in an effort >> to get my N to N concurrent transfers. >> >> It seems that the broadcasts running on concurrent MPI environments >> collide and cause all but the first process to hang waiting for their >> broadcasts. This theory seems to be confirmed by introducing a sleep of >> n-1 seconds before the first MPI_Bcast call on each node, which results >> in the code working perfectly. (total run time 55 seconds, 3 nodes, >> standard TCP stack) >> >> My guess is that unlike PVM, OpenMPI implements broadcasts with broadcasts >> rather than multicasts. Can someone confirm this? Is this a bug? >> >> Is there any multicast or N to N broadcast where sender processes can >> avoid participating when they don’t need to? >> >> Thanks in advance >> Randolph >> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > -- Sent from my mobile device David Zhang University of California, San Diego _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -----Inline Attachment Follows----- _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
