I am having a trouble running simple benchmarks like osu bidirectional
bandwidth tests with recent OMPI (> version 1.8.1)over MLX.
All versions including 1.8.1 seem to work.
The issue is that FDR will hang frequently and will complain about physical
memory available for user run is very low.
The bug starts in v1.8.2.
I searched the src code for differences, but no luck.
I get the message below and hangs...
--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory. This can cause MPI jobs to
run with erratic performance, hang, and/or crash.
This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered. You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.
See this Open MPI FAQ item for more information on these Linux kernel module
parameters:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Local host: sb-cn16
Registerable memory: 24576 MiB
Total memory: 65457 MiB
Your MPI job will continue, but may be behave poorly and/or hang.
---------------
Daniel Faraj
