Thanks Jeff for your answer. It is sad that the approach I mentioned of having all memory registered for user process on cluster nodes didn't become more popular.
I still believe that such an approach would shorten the executed code path in MPI libraries, reduce message latency, increase the communication/computation overlap potential and allows communication progress more naturally. But now since we have to live with memory registration issues, what changes should be done to standard Linux distro so that Open MPI can best use a recent Mellanox Infiniband network ? I guess that installing the ummunotify kernel module is a good idea ? Maybe also removing the limits on the "max locked memory" (ulimit -l) is also good ? Beside that, I guess that installing the latest OFED (to have the latest middleware) instead of using the default one coming with the Linux distro is a good idea ? Also does the XPMEM kernel module for more efficient intra node transfer of large message worth installing since kernels now include the CMA API ? Thanks, Martin Audet
