On Jul 14, 2008, at 7:07 AM, Ashley Pittman wrote:
The default in v1.3 is that ptmalloc2 is *not* built into libopen-
pal. This is different than v1.2, where ptmalloc2 *was* included in
libopen-pal unless you specified --disable-memory-manager.
Thank you for clearing that ip Jeff, what is the cost of using this
option. The comments in the code led me to believe this was more to
do
with pinning memory than anything else?
Would it be advisable to add a mpicc option to enable and disable
linking this library, with 1.2.6 I was sucesfully able to compile and
run a application without it by simply changing the gcc compile line.
FWIW, we have an open ticket to document this stuff before the v1.3
release. :-)
Here's the scoop:
- setting "mpi_leave_pinned" to true does a few things; see http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned
for details
- we decided that ptmalloc2 is more trouble than it's worth; it causes
real problems in some applications
- but some applications definitely do benefit from leave_pinned behavior
- some kind of memory manager is *required* in OMPI for leave_pinned
behavior to work (i.e., OMPI *has* to have visibility to when memory
is returned to the OS so that we know to invalidate relevant entries
in our internal registration cache)
- there's another option, however: simply just don't return memory to
the OS (via a mallopt hint). This avoids the problem altogether.
- it is important to note that OMPI's ptmalloc2 was configured
essentially to do this anyway (rarely to return memory to the OS), and
we haven't seen complaints about this effect.
- so for v1.3, we have changed the default behavior:
- when transports that benefits from leave_pinned behavior are
used in a job, mpi_leave_pinned will automatically enable itself
- OMPI's internal ptmalloc2 library will now be a separate
library (-lopenmpi-malloc) and it will *NOT* be linked in by default.
If you want to use it, you need to add -lopenmpi-malloc to your
wrapper compiler command line.
- if ptmalloc2 is not available and a transport that would
benefit from leave_pinned is used, we'll use the mallopt hint to
disable memory from being returned to the OS
- if mallopt fails for some reason, we'll disable leave_pinned
Note that the "automatically enable itself" part isn't enabled yet;
that's still on my to-do list. It'll be done by v1.3. ptmalloc2 as a
separate library is already done.
What this means is quite simple:
- if you run with openfabrics networks, mpi_leave_pinned will now be
the default, but through the mallopt mechanism, not ptmalloc2
- if you run with openfabrics and want to use ptmalloc2, add -lopenmpi-
malloc to your link command line
- if you do not run with openfabrics networks, you're not really
affected (aside from now returning memory to the OS via the normal
libc memory allocator)
--
Jeff Squyres
Cisco Systems
