On Oct 18, 2007, at 7:56 AM, Gleb Natapov wrote:
Open MPI v1.2.4 (and newer) will get around 1.5us latency with 0 byte
ping-pong benchmarks on Mellanox ConnectX HCAs. Prior versions of
Open MPI can also achieve this low latency by setting the
btl_openib_use_eager_rdma MCA parameter to 1.
Actually setting btl_openib_use_eager_rdma to 1 will not help. The
reason is that it is 1 by default anyway, but Open MPI disables eager
rdma because it can't find HCA description in the ini file and cannot
distinguish between default value and value that user set explicitly.
Arrgh; that's a fun (read: annoying) bug. Well, it's not a total
loss -- you can still get the same performance in older Open MPI
versions by adding the following to the end of the $prefix/share/
openmpi/mca-btl-openib-hca-params.ini file:
[Mellanox Hermon]
vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708
vendor_part_id = 25408,25418,25428
use_eager_rdma = 1
mtu = 2048
--
Jeff Squyres
Cisco Systems
