Hi Jeff, hi all,
I can't think of what OMPI would be doing related to the predefined stack size -- I am not aware of anywhere in the code where we look up the predefine stack size and then do something with it.
I do not know OMPI code at all - but what I see is the consumption of virtual memory according to the twice stack size defaults by new login..
That being said, I don't know what the OS and resource consumption effects are of setting 1GB+ stack size on *any* application...
we defenitely have applications which *need* stack size of 500+MB.Users who use such codes, may trend to hard-code a *huge* stack size in their profile (you do not wanna to lose a day ot two of computing time just by forgitting to set a ulimit, right?). (Currently, I see *one* such user, but who knows how many there are...)
nevertheless, also if the users do not use a huge stack size, the default stack size is some 20 MB. That's not much, but does this allocation-and-never-use of twice of the stack size really needed?
Best wishes, PK Have you
tried non-MPI examples, potentially with applications as large as MPI applications but without the complexity of MPI?On Nov 19, 2009, at 3:13 PM, David Singleton wrote:Depending on the setup, threads often get allocated a thread local stack with size equal to the stacksize rlimit. Two threads maybe? David Terry Dontje wrote: > A couple things to note. First Sun MPI 8.2.1 is effectively OMPI> 1.3.4. I also reproduced the below issue using a C code so I think this> is a general issue with OMPI and not Fortran based. > > I did a pmap of a process and there were two anon spaces equal to the > stack space set by ulimit. > > In one case (setting 102400) the anon spaces were next to each other > prior to all the loadable libraries. In another case (setting 1024000) > one anon space was locate in the same area as the first case but the > second space was deep into some memory used by ompi. > > Is any of this possibly related to the predefined handles? Though I am > not sure why it would expand based on stack size?. > > --td >> Date: Thu, 19 Nov 2009 19:21:46 +0100 >> From: Paul Kapinos <kapi...@rz.rwth-aachen.de> >> Subject: [OMPI users] exceedingly virtual memory consumption of MPI >> environment if higher-setting "ulimit -s" >> To: Open MPI Users <us...@open-mpi.org> >> Message-ID: <4b058cba.3000...@rz.rwth-aachen.de> >> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" >> >> Hi volks, >> >> we see an exeedingly *virtual* memory consumtion through MPI processes >> if "ulimit -s" (stack size)in profile configuration was setted higher. >> >> Furthermore we believe, every mpi process started, wastes about the >> double size of `ulimit -s` value which will be set in a fresh console >> (that is, the value is configurated in e.g. .zshenv, *not* the value >> actually setted in the console from which the mpiexec runs). >> >> Sun MPI 8.2.1, an empty mpi-HelloWorld program >> ! either if running both processes on the same host.. >> >> .zshenv: ulimit -s 10240 --> VmPeak: 180072 kB >> .zshenv: ulimit -s 102400 --> VmPeak: 364392 kB >> .zshenv: ulimit -s 1024000 --> VmPeak: 2207592 kB >> .zshenv: ulimit -s 2024000 --> VmPeak: 4207592 kB >> .zshenv: ulimit -s 20240000 --> VmPeak: 39.7 GB!!!! >> (see the attached files; the a.out binary is a mpi helloworld program >> running an never ending loop). >> >> >> >> Normally, the stack size ulimit is set to some 10 MB by us, but we see >> a lot of codes which needs *a lot* of stack space, e.g. Fortran codes, >> OpenMP codes (and especially fortran OpenMP codes). Users tends to >> hard-code the setting-up the higher value for stack size ulimit. >> >> Normally, the using of a lot of virtual memory is no problem, because >> there is a lot of this thing :-) But... If more than one person is >> allowed to work on a computer, you have to divide the ressources in >> such a way that nobody can crash the box. We do not know how to limit >> the real RAM used so we need to divide the RAM by means of setting >> virtual memory ulimit (in our batch system e.g.. That is, for us >> "virtual memory consumption" = "real memory consumption". >> And real memory is not that way cheap than virtual memory. >> >> >> So, why consuming the *twice* amount of stack size for each process? >> >> And, why consuming the virtual memory at all? We guess this virtual >> memory is allocated for the stack (why else it will be related to the >> stack size ulimit). But, is such allocation really needed? Is there a >> way to avoid the vaste of virtual memory? >> >> best regards, >> Paul Kapinos >> >> >> >> >> _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
-- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, Center for Computing and Communication Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915
smime.p7s
Description: S/MIME Cryptographic Signature
