Just to be clear: what you are launching is an orted daemon, not your application process. Once the daemons are running, then we use them to launch the actual application process. So the issue here is with starting the daemons themselves. You might try adding "-mca plm_base_verbose 5 --debug-daemons" to watch the debug output from the daemons as they are launched.
The lifeline is a socket connection between the daemons and mpirun. For some reason, the socket from your remote daemon back to mpirun is being closed, which the remote daemon interprets as "lifeline lost" and terminates itself. You could try setting the verbosity on the OOB to get the debug output from it (see "ompi_info --param oob tcp" for the settings), though it's likely to just tell you that the socket closed. On Aug 20, 2012, at 5:11 AM, Yann RADENAC <yann.rade...@inria.fr> wrote: > > Hi, > > I'm developing MPI support for XtreemOS (www.xtreemos.eu) so that an MPI > program is managed as a single XtreemOS job. > To manage all processes as a single XtreemOS job, I've developed the program > xos-createProcess that plays the role of the rsh agent (replacing ssh/rsh) to > start a process on a remote machine that is part of the ones reserved for the > current job. > > I'm running a simple hello world MPI program where each processes sends a > string to the process 0 that itself prints them on standard output. > > When using OpenMPI with ssh, this program works perfectly on several machines. > > When using OpenMPI with my launcher xos-createProcess, it works with an MPI > program of 2 processes on 2 different machines. > > However I cannot pass through the following error that happens when running > an MPI program of 3 processes on 3 different machines (or any n processes on > n different machines with n >= 3). > > A process started by xos-createProcess on a remote machine ends with the > following error: > > [paradent-5.rennes.grid5000.fr:08191] [[50627,0],2] routed:binomial: > Connection to lifeline [[50627,0],0] lost > > But, process 0 is still running! lifeline should not have been lost! > Actually, process 0 is still waiting for remote process to terminate (checked > with gdb, the initial process is calling libc's poll()). > > > The run command is: > > -bash -c '(mpirun --mca orte_rsh_agent xos-createProcess > --leave-session-attached -np 2 -host `xreservation -a $XOS_RSVID` > mpi/hello_world_MPI < /dev/null > mpirun.out) >& mpirun.err' > > Same problem with or without option --leave-session-attached. > > > > So, how is the lifeline implemented? why does it work with 2 processes but > start failing when using 3 or more processes? > > > I'm using Open MPI 1.6. > > > Thanks for your help. > > -- > Yann Radenac > Research Engineer, INRIA > Myriads research team, INRIA Rennes - Bretagne Atlantique > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
