[OMPI users] Limiting IP addresses used by OpenMPI

[Charles Doland via users]

Is there a way to limit the IP addresses or network interfaces used for 
communication by OpenMPI? I am looking for something similar to the 
I_MPI_TCP_NETMASK or I_MPI_NETMASK environment variables for Intel MPI.

The OpenMPI documentation mentions the btl_tcp_if_include and 
btl_tcp_if_exclude MCA options. These do not  appear to be present, at least in 
OpenMPI v3.1.2. Is there another way to do this? Or are these options supported 
in a different version?

Charles Doland
charles.dol...@ansys.com
(408) 627-6621  [x6621]

Re: [OMPI users] Limiting IP addresses used by OpenMPI

[John Hearns via users]

Charles, I recall using the I_MPI_NETMASK to choose which interface for MPI
to use.
I guess you are asking the same question for OpenMPI?

On Tue, 1 Sep 2020 at 17:03, Charles Doland via users <
users@lists.open-mpi.org> wrote:

> Is there a way to limit the IP addresses or network interfaces used for
> communication by OpenMPI? I am looking for something similar to the
> I_MPI_TCP_NETMASK or I_MPI_NETMASK environment variables for Intel MPI.
>
> The OpenMPI documentation mentions the btl_tcp_if_include
> and btl_tcp_if_exclude MCA options. These do not  appear to be present, at
> least in OpenMPI v3.1.2. Is there another way to do this? Or are these
> options supported in a different version?
>
> Charles Doland
> charles.dol...@ansys.com
> (408) 627-6621  [x6621]
>

Re: [OMPI users] Limiting IP addresses used by OpenMPI

[Charles Doland via users]

Yes. It is not unusual to have multiple network interfaces on each host of a 
cluster. Usually there is a preference to use only one network interface on 
each host due to higher speed or throughput, or other considerations. It would 
be useful to be able to explicitly specify the interface to use for cases in 
which the MPI code does not select the preferred interface.

Charles Doland
charles.dol...@ansys.com
(408) 627-6621  [x6621]

From: users  on behalf of John Hearns via 
users 
Sent: Tuesday, September 1, 2020 12:22 PM
To: Open MPI Users 
Cc: John Hearns 
Subject: Re: [OMPI users] Limiting IP addresses used by OpenMPI


[External Sender]

Charles, I recall using the I_MPI_NETMASK to choose which interface for MPI to 
use.
I guess you are asking the same question for OpenMPI?

On Tue, 1 Sep 2020 at 17:03, Charles Doland via users 
mailto:users@lists.open-mpi.org>> wrote:
Is there a way to limit the IP addresses or network interfaces used for 
communication by OpenMPI? I am looking for something similar to the 
I_MPI_TCP_NETMASK or I_MPI_NETMASK environment variables for Intel MPI.

The OpenMPI documentation mentions the btl_tcp_if_include and 
btl_tcp_if_exclude MCA options. These do not  appear to be present, at least in 
OpenMPI v3.1.2. Is there another way to do this? Or are these options supported 
in a different version?

Charles Doland
charles.dol...@ansys.com
(408) 627-6621  [x6621]

Re: [OMPI users] Limiting IP addresses used by OpenMPI

[Jeff Squyres (jsquyres) via users]

3.1.2 was a long time ago, but I'm pretty sure that Open MPI v3.1.2 has 
btl_tcp_if_include / btl_tcp_if_exclude.

Try running: "ompi_info --all --parsable | grep btl_tcp_if_"

I believe that those options will both take a CIDR notation of which network(s) 
to use/not use.  Note: the _if_include option is mutually exclusive with the 
_if_exclude option.


On Sep 1, 2020, at 3:35 PM, Charles Doland via users 
mailto:users@lists.open-mpi.org>> wrote:

Yes. It is not unusual to have multiple network interfaces on each host of a 
cluster. Usually there is a preference to use only one network interface on 
each host due to higher speed or throughput, or other considerations. It would 
be useful to be able to explicitly specify the interface to use for cases in 
which the MPI code does not select the preferred interface.

Charles Doland
charles.dol...@ansys.com
(408) 627-6621  [x6621]

From: users 
mailto:users-boun...@lists.open-mpi.org>> on 
behalf of John Hearns via users 
mailto:users@lists.open-mpi.org>>
Sent: Tuesday, September 1, 2020 12:22 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Cc: John Hearns mailto:hear...@gmail.com>>
Subject: Re: [OMPI users] Limiting IP addresses used by OpenMPI

[External Sender]
Charles, I recall using the I_MPI_NETMASK to choose which interface for MPI to 
use.
I guess you are asking the same question for OpenMPI?

On Tue, 1 Sep 2020 at 17:03, Charles Doland via users 
mailto:users@lists.open-mpi.org>> wrote:
Is there a way to limit the IP addresses or network interfaces used for 
communication by OpenMPI? I am looking for something similar to the 
I_MPI_TCP_NETMASK or I_MPI_NETMASK environment variables for Intel MPI.

The OpenMPI documentation mentions the btl_tcp_if_include and 
btl_tcp_if_exclude MCA options. These do not  appear to be present, at least in 
OpenMPI v3.1.2. Is there another way to do this? Or are these options supported 
in a different version?

Charles Doland
charles.dol...@ansys.com
(408) 627-6621  [x6621]


--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Limiting IP addresses used by OpenMPI

[Joseph Schuchart via users]

Charles,

What is the machine configuration you're running on? It seems that there 
are two MCA parameter for the tcp btl: btl_tcp_if_include and 
btl_tcp_if_exclude (see ompi_info for details). There may be other knobs 
I'm not aware of. If you're using UCX then my guess is that UCX has its 
own way to choose the network interface to be used...


Cheers
Joseph

On 9/1/20 9:35 PM, Charles Doland via users wrote:
Yes. It is not unusual to have multiple network interfaces on each host 
of a cluster. Usually there is a preference to use only one network 
interface on each host due to higher speed or throughput, or other 
considerations. It would be useful to be able to explicitly specify the 
interface to use for cases in which the MPI code does not select the 
preferred interface.


Charles Doland
charles.dol...@ansys.com 
(408) 627-6621  [x6621]

*From:* users  on behalf of John 
Hearns via users 

*Sent:* Tuesday, September 1, 2020 12:22 PM
*To:* Open MPI Users 
*Cc:* John Hearns 
*Subject:* Re: [OMPI users] Limiting IP addresses used by OpenMPI

*[External Sender]*

Charles, I recall using the I_MPI_NETMASK to choose which interface for 
MPI to use.

I guess you are asking the same question for OpenMPI?

On Tue, 1 Sep 2020 at 17:03, Charles Doland via users 
mailto:users@lists.open-mpi.org>> wrote:


Is there a way to limit the IP addresses or network interfaces used
for communication by OpenMPI? I am looking for something similar to
the I_MPI_TCP_NETMASK or I_MPI_NETMASK environment variables for
Intel MPI.

The OpenMPI documentation mentions the btl_tcp_if_include
and btl_tcp_if_exclude MCA options. These do not  appear to be
present, at least in OpenMPI v3.1.2. Is there another way to do
this? Or are these options supported in a different version?

Charles Doland
charles.dol...@ansys.com 
(408) 627-6621  [x6621]

Re: [OMPI users] Problem in starting openmpi job - no output just hangs - SOLVED

[Tony Ladd via users]

Jeff

I found the solution - rdma needs significant memory so the limits on 
the shell have to be increased. I needed to add the lines


* soft memlock unlimited
* hard memlock unlimited

to the end of the file /etc/security/limits.conf. After that the openib 
driver loads and everything is fine - proper IB latency again.


I see that # 16 of the tuning FAQ discusses the same issue, but in my 
case there was no error or warning message. I am posting this in case 
anyone else runs into this issue.


The Mellanox OFED install adds those lines automatically, so I had not 
run into this before.


Tony


On 8/25/20 10:42 AM, Jeff Squyres (jsquyres) wrote:

[External Email]

On Aug 24, 2020, at 9:44 PM, Tony Ladd  wrote:

I appreciate your help (and John's as well). At this point I don't think is an 
OMPI problem - my mistake. I think the communication with RDMA is somehow 
disabled (perhaps its the verbs layer - I am not very knowledgeable with this). 
It used to work like a dream but Mellanox has apparently disabled some of the 
Connect X2 components, because neither ompi or ucx (with/without ompi) could 
connect with the RDMA layer. Some of the infiniband functions are also not 
working on the X2 (mstflint, mstconfig).

If the IB stack itself is not functioning, then you're right: Open MPI won't 
work, either (with openib or UCX).

You can try to keep poking with the low-layer diagnostic tools like ibv_devinfo 
and ibv_rc_pingpong.  If those don't work, Open MPI won't work over IB, either.


In fact ompi always tries to access the openib module. I have to explicitly 
disable it even to run on 1 node.

Yes, that makes sense: Open MPI will aggressively try to use every possible 
mechanism.


So I think it is in initialization not communication that the problem lies.

I'm not sure that's correct.

 From your initial emails, it looks like openib thinks it initialized properly.


This is why (I think) ibv_obj returns NULL.

I'm not sure if that's a problem or not.  That section of output is where Open 
MPI is measuring the distance from the current process to the PCI bus where the 
device lives.  I don't remember offhand if returning NULL in that area is 
actually a problem or just an indication of some kind of non-error condition.

Specifically: if returning NULL there was a problem, we *probably* would have 
aborted at that point.  I have not looked at the code to verify that, though.


The better news is that with the tcp stack everything works fine (ompi, ucx, 1 
node, many nodes) - the bandwidth is similar to rdma so for large messages its 
semi OK. Its a partial solution - not all I wanted of course. The direct rdma 
functions ib_read_lat etc also work fine with expected results. I am suspicious 
this disabling of the driver is a commercial more than a technical decision.
I am going to try going back to Ubuntu 16.04 - there is a version of OFED that 
still supports the X2. But I think it may still get messed up by kernel 
upgrades (it does for 18.04 I found). So its not an easy path.


I can't speak for Nvidia here, sorry.

--
Jeff Squyres
jsquy...@cisco.com


--
Tony Ladd

Chemical Engineering Department
University of Florida
Gainesville, Florida 32611-6005
USA

Email: tladd-"(AT)"-che.ufl.edu
Webhttp://ladd.che.ufl.edu

Tel:   (352)-392-6509
FAX:   (352)-392-9514