Re: [OMPI users] /dev/shm

2008-12-03 Thread Jeff Squyres
Another curiosity question (sorry for the delay; SC08 and Thankgiving have significantly increased the latency on replying to my INBOX): are we sure that these are Open MPI files? On Nov 19, 2008, at 2:17 PM, Ray Muno wrote: Ralph Castain wrote: Hi Ray Are the jobs that leave files behind

Re: [OMPI users] /dev/shm

2008-11-20 Thread John Hearns
2008/11/20 Ray Muno > J >> >> >> >> > OK, what should I be seeing when I run "ipcs -p"? > > Looks like I don't know my System V from my POSIX. I know what to do.

Re: [OMPI users] /dev/shm

2008-11-20 Thread Ray Muno
John Hearns wrote: 2008/11/19 Ray Muno Thought I would revisit this one. We are still having issues with this. It is not clear to me what is leaving the user files behind in /dev/shm. This is not something users are doing directly, they are just compiling their code directly with mpif90 (fro

Re: [OMPI users] /dev/shm

2008-11-19 Thread Ray Muno
Ralph Castain wrote: Hi Ray Are the jobs that leave files behind terminating normally or aborting? Are there any warnings/error messages out of mpirun? Just trying to determine if this is an abnormal termination issue or a bug in OMPI itself. Ralph As far as I know, they are from jobs t

Re: [OMPI users] /dev/shm

2008-11-19 Thread Ralph Castain
Hi Ray Are the jobs that leave files behind terminating normally or aborting? Are there any warnings/error messages out of mpirun? Just trying to determine if this is an abnormal termination issue or a bug in OMPI itself. Ralph On Nov 19, 2008, at 8:05 AM, Ray Muno wrote: Thought I wo

Re: [OMPI users] /dev/shm

2008-11-19 Thread John Hearns
2008/11/19 Ray Muno > Thought I would revisit this one. > > We are still having issues with this. It is not clear to me what is leaving > the user files behind in /dev/shm. > > This is not something users are doing directly, they are just compiling > their code directly with mpif90 (from OpenMPI)

Re: [OMPI users] /dev/shm

2008-11-19 Thread Ray Muno
Thought I would revisit this one. We are still having issues with this. It is not clear to me what is leaving the user files behind in /dev/shm. This is not something users are doing directly, they are just compiling their code directly with mpif90 (from OpenMPI), using various compilers. Co

Re: [OMPI users] /dev/shm

2008-11-10 Thread Ray Muno
Jeff Squyres wrote: That is odd. Is your user's app crashing or being forcibly killed? The ORTE daemon that is silently launched in v1.2 jobs should ensure that files under /tmp/openmpi-sessions-@ are removed. It looks like I see orphaned directories under /tmp/openmpi* as well. -- Ra

Re: [OMPI users] /dev/shm

2008-11-10 Thread Brock Palen
yeah if that gets full it is not going to work, We use /dev/shm for some FEA apps that have bad IO patters, I tend to keep it to just the most educated users. It just impacts others to much if not treated with respect. Brock Palen www.umich.edu/~brockp Center for Advanced Computing bro...@um

Re: [OMPI users] /dev/shm

2008-11-10 Thread Jeff Squyres
That is odd. Is your user's app crashing or being forcibly killed? The ORTE daemon that is silently launched in v1.2 jobs should ensure that files under /tmp/openmpi-sessions-@ are removed. On Nov 10, 2008, at 2:14 PM, Ray Muno wrote: Brock Palen wrote: on most systems /dev/shm is limite

Re: [OMPI users] /dev/shm

2008-11-10 Thread Ray Muno
Brock Palen wrote: on most systems /dev/shm is limited to half the physical ram. Was the user someone filling up /dev/shm so there was no space? The problem is there is a large collection of stale files left in there by the users that have run on that node (Rocks based cluster). I am tryi

Re: [OMPI users] /dev/shm

2008-11-10 Thread Brock Palen
on most systems /dev/shm is limited to half the physical ram. Was the user someone filling up /dev/shm so there was no space? Brock Palen www.umich.edu/~brockp Center for Advanced Computing bro...@umich.edu (734)936-1985 On Nov 10, 2008, at 1:25 PM, Ray Muno wrote: We are running OpenMPI 1