The slaves send specific requests to the master and then waits for a reply to that request. For instance it might send a request to read a variable from the file. The master will read the variable and send it back with the same tag in response. Thus there is never more than one response at a time to a given slave. We do not use any broadcast functions in the code.
The fact that it run ok on one host but not more than one host seems to indicate something else is the problem. The code has been used for 13 years in parallel and runs with PVM and other MPI distros without any problems. The communication patterns are very simple and only require that message order be preserved. -----Original Message----- From: Jeff Squyres [mailto:jsquy...@cisco.com] Sent: Tuesday, January 30, 2007 8:44 AM To: Open MPI Users Subject: Re: [OMPI users] Scrambled communications using ssh starteronmultiple nodes. On Jan 30, 2007, at 9:35 AM, Fisher, Mark S wrote: > The master process uses both MPI_ANY_SOURCE and MPI_ANY_TAG while > waiting for requests from slave processes. The slaves sometimes use > MPI_ANY_TAG but the source is always specified. I think you said that you only had corruption issues on the slave, right? If so, the ANY_SOURCE/ANY_TAG on the master probably aren't the issue. But if you're doing ANY_TAG on the slaves, you might want to double check that that code is doing exactly what you think it's doing. Are there any race conditions such that a message could be received on that ANY_TAG that you did not intend to receive there? Look especially hard at non-blocking receives with ANY_TAG. > We have run the code through valgrid for a number of cases including > the one being used here. Excellent. > The code is Fortran 90 and we are using the FORTRAN 77 interface so I > do not believe this is a problem. Agreed; should not be an issue. > We are using Gigabit Ethernet. Ok, good. > I could look at LAM again to see if it would work. The code needs to > be in a specific working directory and we need some environment > variable set. This was not supported well in pre MPI 2. versions of > MPI. For > MPICH1 I actually launch a script for the slaves so that we have the > proper setup before running the executable. Note I had tried that with > OpenMPI and it had an internal error in orterun. This is not a problem Really? OMPI's mpirun does not depend on the executable being an MPI application -- indeed, you can "mpirun -np 2 uptime" with no problem. What problem did you run into here? > since the mpirun can setup everything we need. If you think it is > worth while I will download and try it. From what you describe, it sounds like order of messaging may be the issue, not necessarily MPI handle types. So let's hold off on that one for the moment (although LAM should be pretty straightforward to try -- you should be able to mpirun scripts with no problems; perhaps you can try it as a background effort when you have spare cycles / etc.), and look at your slave code for receiving. > -----Original Message----- > From: Jeff Squyres [mailto:jsquy...@cisco.com] > Sent: Monday, January 29, 2007 7:54 PM > To: Open MPI Users > Subject: Re: [OMPI users] Scrambled communications using ssh starter > onmultiple nodes. > > Without analyzing your source, it's hard to say. I will say that OMPI > may send fragments out of order, but we do, of course, provide the > same message ordering guarantees that MPI mandates. So let me ask a > few leading questions: > > - Are you using any wildcards in your receives, such as MPI_ANY_SOURCE > or MPI_ANY_TAG? > > - Have you run your code through a memory-checking debugger such as > valgrind? > > - I don't know what Scali MPI uses, but MPICH and Intel MPI use > integers for MPI handles. Have you tried LAM/MPI as well? It, like > Open MPI, uses pointers for MPI handles. I mention this because apps > that unintentionally have code that takes advantage of integer handles > can sometimes behave unpredictably when switching to a pointer-based > MPI implementation. > > - What network interconnect are you using between the two hosts? > > > > On Jan 25, 2007, at 4:22 PM, Fisher, Mark S wrote: > >> Recently I wanted to try OpenMPI for use with our CFD flow solver >> WINDUS. The code uses a master/slave methodology were the master >> handles I/O and issues tasks for the slaves to perform. The original >> parallel implementation was done in 1993 using PVM and in 1999 we >> added support for MPI. >> >> When testing the code with Openmpi 1.1.2 it ran fine when running on >> a > >> single machine. As soon as I ran on more than one machine I started >> getting random errors right away (the attached tar ball has a good >> and > >> bad output). It looked like either the messages were out of order or >> were for the other slave process. In the run mode used there is no >> slave to slave communication. In the file the code died near the >> beginning of the communication between master and slave. Sometimes it >> will run further before it fails. >> >> I have included a tar file with the build and configuration info. The >> two nodes are identical Xeon 2.8 GHZ machines running SLED 10. I am >> running real-time (no queue) using the ssh starter using the >> following > >> appt file. >> >> -x PVMTASK -x BCFD_PS_MODE --mca pls_rsh_agent /usr/bin/ssh --host >> skipper2 -wdir /opt/scratch/m209290/ol.scr.16348 -np 1 ./ >> __bcfdbeta.exe -x PVMTASK -x BCFD_PS_MODE --mca pls_rsh_agent >> /usr/bin/ssh --host copland -wdir /tmp/mpi.m209290 -np 2 >> ./__bcfdbeta.exe >> >> The above file fails but the following works: >> >> -x PVMTASK -x BCFD_PS_MODE --mca pls_rsh_agent /usr/bin/ssh --host >> skipper2 -wdir /opt/scratch/m209290/ol.scr.16348 -np 1 ./ >> __bcfdbeta.exe -x PVMTASK -x BCFD_PS_MODE --mca pls_rsh_agent >> /usr/bin/ssh --host >> skipper2 -wdir /tmp/mpi.m209290 -np 2 ./__bcfdbeta.exe >> >> The first process is the master and the second two are the slaves. >> I am >> not sure what is going wrong, the code runs fine with many other MPI >> distributions (MPICH1/2, Intel, Scali...). I assume that either I >> built it wrong or am not running it properly but I cannot see what I >> am doing wrong. Any help would be appreciated! >> >> <<mpipb.tgz>> >> <mpipb.tgz> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > Server Virtualization Business Unit > Cisco Systems > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Server Virtualization Business Unit Cisco Systems _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
