The mailing list snipped off the end of my mail -- here's the rest of
what I said:
The meanings of the 3 phases are explained in this pager: http://
www.open-mpi.org/papers/euro-pvmmpi-2006-hpc-protocols.
If you use the mpi_leave_pinned parameter and Open MPI is able to
leave your entire buffer registered, it should give the best
performance in OMPI v1.2.4 because it should be able to do a single
RDMA PUT to transfer the majority of the message (after the first
time, of course).
On Oct 12, 2007, at 11:31 AM, Jeff Squyres wrote:
On Oct 12, 2007, at 8:38 AM, Neeraj Chourasia wrote:
Yes, the buffer was being re-used. No we didnt try to benchmark it
with netpipe and other stuffs. But the program was pretty simple.
Do you think, I need to test it with bigger chunks (>8MB) for
communication.?
We also tried manipulating eager_limit and min_rdma_sze, but no
success.
It really depends on exactly what your application is doing. If
you re-use your buffers, then mpi_leave_pinned should help at least
some. But if the cost of communication is dwarfed by the rest of
the app, then all this tuning won't make much of a difference.
Here's a picture of Open MPI's MCA parameters w.r.t. the openib BTL
and the ob1 PML (we changed some names and ordering between the
v1.2 series and what will become the v1.3 series -- both are shown
below) -- hopefully this will help you understand what those MCA
parameters are for:
<ob1-params.pdf>
<mime-attachment.txt>
<mime-attachment.txt>
--
Jeff Squyres
Cisco Systems
