I did some code-digging and I found the answer.
If the MCA parameter btl_openib_receive_queues is not spec'd on the mpirun
command line and not specified in
MPI_HOME/share/openmpi/mca-btl-openib-device-params.ini (via receive_queues
parameter), then OpenMPI derives the default setting from the code in
ompi/mca/btl/openib/btl_openib_mca.c. If you look in that code you will see:
snprintf(default_qps, 100,
"P,128,256,192,128:S,%u,1024,1008,64:S,%u,1024,1008,64:S,%u,1024,1008,64",
mid_qp_size, // For S,%u,1024,1008,64 where %u == 2048
(uint32_t)mca_btl_openib_module.super.btl_eager_limit,
(uint32_t)mca_btl_openib_module.super.btl_max_send_size);
So for OpenMPI 1.6.5 we have:
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64
So you can examine the code and see how the %u's are filled in to get the
resultant default btl_openib_receive_queues.
Nice if this was documented!
--john
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Shamis, Pavel
Sent: Thursday, November 05, 2015 2:41 PM
To: Open MPI Users
Subject: Re: [OMPI users] btl_openib_receive_queues &
mca-btl-openib-device-params.ini
Below is from ompi_info:
$ ompi_info --all | grep btl_openib_receive
MCA btl: parameter "btl_openib_receive_queues" (current value:
<P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64>,
data source: default value)
This tunning does make sense for me.
# receive_queues = P,128,256,192,128:S,65536,256,192,128
Most likely it was not updated for a long time.
I'm pretty sure the ompi_info version is the one that actually used.
So comparing the two above, they don't coincide, in particular for the SRQ
settings. Thus, I gather the settings reported by ompi_info above are those
that are actually used, and if so, I am wondering where/how they are derived?
The idea behind this parameters is coming from here:
http://www.open-mpi.org/papers/euro-pvmmpi-2007-ib/euro-pvmmpi-2007-ib.pdf<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.open-2Dmpi.org_papers_euro-2Dpvmmpi-2D2007-2Dib_euro-2Dpvmmpi-2D2007-2Dib.pdf&d=CwMF-g&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=tqKZ2vRCLufSSXPvzNxBrKr01YPimBPnb-JT-Js0Fmk&m=ful34HOXqlPDIxbufExLGHQRNlgb5Oy60k_TKVG3VmI&s=YOBOBrSC7_mvy-6Z2gqNRLZGDJTQ9UoTa1pZpzfxIcY&e=>
And later on XRC support was introduced:
http://www.open-mpi.org/papers/euro-pvmmpi-2008-xrc/euro-pvmmpi-2008-xrc.pdf<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.open-2Dmpi.org_papers_euro-2Dpvmmpi-2D2008-2Dxrc_euro-2Dpvmmpi-2D2008-2Dxrc.pdf&d=CwMF-g&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=tqKZ2vRCLufSSXPvzNxBrKr01YPimBPnb-JT-Js0Fmk&m=ful34HOXqlPDIxbufExLGHQRNlgb5Oy60k_TKVG3VmI&s=wKpbgirfQeTb_SW5gKhPep2khno1MHxnc-LdWVkk7Og&e=>
Best,
Pasha
