> On Jun 20, 2016, at 4:15 PM, Audet, Martin <martin.au...@cnrc-nrc.gc.ca>
> wrote:
>
> But now since we have to live with memory registration issues, what changes
> should be done to standard Linux distro so that Open MPI can best use a
> recent Mellanox Infiniband network ?
>
> I guess that installing the ummunotify kernel module is a good idea ?
There may be some bit rot in our ummunotify support in Open MPI. Ah yes, here
it is:
https://github.com/open-mpi/ompi/issues/429
We haven't dug into the problem because ummunotify isn't upstream, and no one
has apparently been using it. If someone had some cycles to look into it, that
would be great.
> Maybe also removing the limits on the “max locked memory” (ulimit -l) is also
> good ?
That's what most OS-bypass vendors recommend.
> Beside that, I guess that installing the latest OFED (to have the latest
> middleware) instead of using the default one coming with the Linux distro is
> a good idea ?
There's different religion surrounding that -- I believe there was some good
discussion about that topic on the user's list recently.
> Also does the XPMEM kernel module for more efficient intra node transfer of
> large message worth installing since kernels now include the CMA API ?
Yes, XPMEM == goodness. I know that Vader supports both XPMEM and CMA; I don't
know offhand the tradeoffs between the two.
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
