(sigh) let me clarify this to resolve some offlist chatter. It would be rather simple to implement an option that allowed an MPI job to continue executing after the failure of one or more processes. The problem is that OMPI's MPI layer does not yet know how to handle that situation. As Josh indicated in his posting, there are several research branches out there that are exploring solutions. However, these are -research- branches - they have not been tested nor vetted by the overall OMPI developer community.
So if I add the specified option to the OMPI code, it would (as of this moment) be up to the application to deal with the failure. While I could ensure that OMPI would clean up if/when the job finalized, it would be up to the application to ensure that the job actually reached that stage. For example, it would be up to the application to ensure that any collectives underway at the time of proc failure were properly exited so the job could finalize. As Josh said, at some point we expect these research branches to fold their results into the OMPI code base. Historically, we have found that this generally creates some period of instability - research code doesn't get exposed to the entire range of environments we support, and so there are usually problems that need to be ironed out. Using the code in a production environment before that has occurred is a "use at your own risk" venture. HTH Ralph On Sep 16, 2011, at 8:28 AM, Ralph Castain wrote: > Actually, I honestly don't remember even having that discussion. In looking > at it, this would be relatively easy to implement if someone really wanted it. > > Only issue: user would bear full responsibility for OMPI not cleaning up > failed jobs since we wouldn't terminate upon seeing a proc fail. Definitely > not something you'd want to do in production! > > > On Sep 16, 2011, at 6:55 AM, Josh Hursey wrote: > >> Though I do not share George's pessimism about acceptance to the Open >> MPI community, it has been slightly difficult to add such a >> non-standard feature to the code base for various reasons. >> >> At ORNL, I have been developing a prototype for the MPI Forum Fault >> Tolerance Working Group [1] of the Run-Through Stabilization proposal >> [2,3]. This would allow the application to continue running and using >> MPI functions even though processes fail during execution. We have >> been doing some limited alpha releases for some friendly application >> developers desiring to play with the prototype for a while now. We are >> hoping to do a more public beta release in the coming months. I'll >> likely post a message to the ompi-devel list once it is ready. >> >> -- Josh >> >> [1] http://svn.mpi-forum.org/trac/mpi-forum-web/wiki/FaultToleranceWikiPage >> [2] See PDF on >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilization >> [3] See PDF on >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilization_2 >> >> On Thu, Sep 15, 2011 at 4:14 PM, George Bosilca <bosi...@eecs.utk.edu> wrote: >>> Rob, >>> >>> The Open MPI community did consider such as option, but it deemed it as >>> uninteresting. However, we (UTK team) have a patched version supporting >>> several fault tolerant modes, including the one you described in your >>> email. If you are interested please contact me directly. >>> >>> Thanks, >>> george. >>> >>> >>> On Sep 12, 2011, at 20:43 , Ralph Castain wrote: >>> >>>> We don't have anything similar in OMPI. There are fault tolerance modes, >>>> but not like the one you describe. >>>> >>>> On Sep 12, 2011, at 5:52 PM, Rob Stewart wrote: >>>> >>>>> Hi, >>>>> >>>>> I have implemented a simple fault tolerant ping pong C program with MPI, >>>>> here: http://pastebin.com/7mtmQH2q >>>>> >>>>> MPICH2 offers a parameter with mpiexec: >>>>> $ mpiexec -disable-auto-cleanup >>>>> >>>>> .. as described here: http://trac.mcs.anl.gov/projects/mpich2/ticket/1421 >>>>> >>>>> It is fault tolerant in the respect that, when I ssh to one of the nodes >>>>> in the hosts file, and kill the relevant process, the MPI job is not >>>>> terminated. Simply, the ping will not prompt a pong from the dead node, >>>>> but the ping-pong runs forever on the remaining live nodes. >>>>> >>>>> Is such an feature available for openMPI, either via mpiexec or some >>>>> other means? >>>>> >>>>> >>>>> -- >>>>> Rob Stewart >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >
