I'm using RoCE (or rather, attempting to) and need to select a
non-default GID to get my traffic properly classified. Both 1.4.4rc2
and 1.5.4 support the btl_openib_ipaddr_include option, but only 1.5.4
causes my traffic to use the proper GID and VLAN.
Is there something broken with ipaddr_includ
Alternatively, if OpenMPI is really trying to use both ports, you
could force it to use just one port with --mca btl_openib_if_include
mlx4_0:1 (probably)
--
Mike Shuey
On Tue, Mar 1, 2011 at 1:02 PM, Jeff Squyres wrote:
> On Feb 28, 2011, at 12:49 PM, Jagga Soorma wrote:
>
>> -bash-3.2$ mpiex
?
>
>
> On Mar 1, 2011, at 7:35 AM, Michael Shuey wrote:
>
>> So, since RoCE has no SM, and setting an SL is required to get
>> lossless ethernet on Cisco switches (and possibly others), does this
>> mean that RoCE will never work correctly with OpenMPI on Cisco
>> ha
squy...@cisco.com]
> Sent: Thursday, February 24, 2011 3:45 PM
> To: Michael Shuey
> Cc: Open MPI Users , Mike Dubman
> Subject: Re: [OMPI users] RoCE (IBoE) & OpenMPI
>
> On Feb 24, 2011, at 8:00 AM, Michael Shuey wrote:
>
>> Late yesterday I did have a chance to test t
Late yesterday I did have a chance to test the patch Jeff provided
(against 1.4.3 - testing 1.5.x is on the docket for today). While it
works, in that I can specify a gid_index, it doesn't do everything
required - my traffic won't match a lossless CoS on the ethernet
switch. Specifying a GID is o
means that in order to select network X or Y, you
>> may use ip/netmask (btl_openib_ipaddr_include) .
>>
>> Pavel (Pasha) Shamis
>> ---
>> Application Performance Tools Group
>> Computer Science and Math Division
>> Oak Ridge National Laboratory
>>
Fri, Feb 18, 2011 at 3:44 PM, Jeff Squyres wrote:
> On Feb 18, 2011, at 1:39 PM, Michael Shuey wrote:
>
>> RoCE HCAs keep a GID table, like normal HCAs. Every time you bring up
>> a vlan interface, another entry gets automatically added to the table.
>> If I select one o
essing you need to do)
> should be somewhere in the root-level setup for RoCE. Once you set a
> different subnet ID, Open MPI should just use it.
>
>
> On Feb 18, 2011, at 8:17 AM, Michael Shuey wrote:
>
>> I've been looking into OpenMPI's support for RoCE (M
I've been looking into OpenMPI's support for RoCE (Mellanox's recent
Infiniband-over-Ethernet) lately. While it's promising, I've hit a
snag: RoCE requires lossless ethernet, and on my switches the only way
to guarantee this is with CoS. RoCE adapters cannot emit CoS priority
tags unless the clie
