On 02/03/2014 03:59 PM, Ralph Castain wrote:
Very strange - even if you kill the job with SIGTERM, or have processes that segfault, OMPI should clean itself up and remove those session directories. Granted, the 1.6 series isn't as good about doing so as the 1.7 series, but it at least to-date has done pretty well.
Ok, one more information here that may matter: All sequential tests are launched *without* mpiexec... I don't know if the "cleanup" phase is done by mpiexec or the binaries...
Best I can suggest for now is to do the following in your test script: (1) set TMPDIR=</tmp/regression> (2) run your tests (3) rm -rf /tmp/regression/* That will ensure you only blow away the session dirs from your regression tests. Hopefully, you'll find the directory empty more often than not...
Ok, I just added:
find /tmp/openmpi-sessions-${USER}* -maxdepth 1 -type f -exec rm {} \;
which should delete files that shouldn't exists... ;-)
But, IMHO, I still think OpenMPI should "choose" another directory name
if it can't create it because a poor file exists!
How can all users be aware that they have to cleanup such files?Maybe a good compromise would be to have the error message to tell there is a file with the same name of the directory chosen?
Or add a new entry to the FAQ to help users find the workaround you proposed... ;-)
thanks again! Eric
HTH Ralph On Feb 3, 2014, at 12:31 PM, Eric Chamberland <eric.chamberl...@giref.ulaval.ca> wrote:Hi, On 02/03/2014 03:09 PM, Ralph Castain wrote:OMPI will error out in that case, as you originally reported. What seems to be happening is that you have a bunch of stale session directories, but I'm puzzled because the creation dates are so current - for whatever reason, OMPI seems to be getting the same jobid much more often than it should. Can you tell me something about the environment - e.g., is it managed or just using hostfile?This computer is used about 11 times a day to launch about 1500 executions on our in-house (finite element) code. We do launch at most 12 single process executions at the same time, but we use PETSc, which always initialize the MPI environment... Also, we are launching some tests which use between 2 to 128 processes (on the same computer) just to ensure proper code testing. In fact, performance is not quit an issue in these 128 processes tests and we set the following environment variable: export OMPI_MCA_mpi_yield_when_idle=1 because we encountered timeout problems before... The whole testing lasts about 1 hour and the result is used to give a feed-back for users who "pushed" modifications to the code.... So I would add: sometime the tests may be interrupted by segfaults, "kill -TERM" or anything you can imagine... The problem now is that it won't even start if a mere file exists... I can flush those files right now, but I am almost sure they will reappear it the following days, leading to false "bad results" for the tests... and I will have to setup a cleanup procedure before launching all the tests... But that will not prevent the fact that those files may be created while running the firsts of the 1500 tests and have 1 or some of the rest to fail.... I hope this is the information you wanted... Is it? Thanks, EricOn Feb 3, 2014, at 12:00 PM, Eric Chamberland <eric.chamberl...@giref.ulaval.ca> wrote:On 02/03/2014 02:49 PM, Ralph Castain wrote:Seems rather odd - is your /tmp by any chance network mounted?No it is a "normal" /tmp: "cd /tmp; df -h ." gives: Filesystem Size Used Avail Use% Mounted on /dev/sda1 49G 17G 30G 37% / And there is plenty of disk space... I agree it is odd, but how should OpenMPI react when trying to create a directory over an existing file name? I mean what is it programmed to do? Thanks, Eric
