Dear OpenMPI experts
I am confused about the role and values of priorities set to
the OpenMPI collective components, and why the shared memory (sm)
collective component disqualifies itself (apparently by default).
I describe the problem, then ask a some questions, if you have the
patience to read.
I am using OpenMPI 1.4.2 with default MCA settings.
I am running the job in a single node.
I use "mpirun -mca btl sm,self", since this is a single-node job.
In my naive view, I expected all MPI functions to use shared memory.
However, here are selected lines from the
verbose output of my mpirun command,
which tell what happens better than I could do:
*************
btl_self_priority=0 (default value)
btl_sm_priority=0 (default value)
[Note that 'sm' and 'self' 'btl' priorities
are tied at zero (by default).]
coll_basic_priority=10 (default value)
coll_hierarch_priority=0 (default value)
coll_inter_priority=40 (default value)
coll_self_priority=75 (default value)
coll_sm_priority=0 (default value)
coll_sync_priority=50 (default value)
coll_tuned_priority=30 (default value)
[Note that 'coll' priorities are *not* tied,
'self' is maximum (75), and 'sm' is minimum (0).]
coll:sm:comm_query (0/MPI_COMM_WORLD): priority too low; disqualifying
myself
coll:sm:comm_query (3/MPI_COMMUNICATOR 3): priority too low;
disqualifying myself
[Therefore, 'sm' seems to give up working in collectives ... :( ]
coll:base:comm_select: Checking all available modules
coll:base:comm_select: component available: basic, priority: 10
coll:base:comm_select: component not available: hierarch
coll:base:comm_select: component not available: inter
coll:base:comm_select: component not available: self
coll:base:comm_select: component not available: sm
coll:base:comm_select: component available: sync, priority: 50
coll:base:comm_select: component available: tuned, priority: 30
coll:base:comm_select: Checking all available modules
coll:base:comm_select: component available: basic, priority: 10
coll:base:comm_select: component not available: hierarch
coll:base:comm_select: component not available: inter
coll:base:comm_select: component available: self, priority: 75
[Eventually 'sm', 'inter', and 'hierarch' seem to go out of business,
whereas 'basic', 'sync' and 'tuned' hang in there.
It is awkward that 'self' claims both to
be available and not available!]
So, who's in first?
Self, basic, other?
*************
I confess I am in complete confusion.
My naive questions:
1) Are the "coll" priorities above (default values) the best choices
when I run in a single node, or were they chosen for a general
situation when the job runs across node boundaries?
2) Why does "self" have the largest value (75)?
3) Does it mean that all collectives will use the
self/loopback mechanism for communication?
How about 'basic' and the rest of the gang with smaller priorities?
4) Is it a good idea to set the 'sm' priority to a value
larger than 75 (to beat "self" and take over the collective functions)?
(I am trying it right now.)
5) In this case, will the collectives only use "sm"?
6) Will this improve or degrade performance ?
7) Is there any literature where I can learn
more about these OpenMPI collective priorities?
(I couldn't find anything about it on the FAQs.
Actually, a group of FAQ about collectives would be very helpful.)
Thank you,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
