On Feb 21, 2008, at 12:36 PM, George Bosilca wrote:
Here are the MCA parameters that you can use:
MCA btl: parameter "btl_openib_if_include" (current value:
<none>)
Comma-delimited list of HCAs/ports to be used (e.g.
"mthca0,mthca1:2"; empty value means to use all
ports found).
Mutually exclusive with btl_openib_if_exclude.
MCA btl: parameter "btl_openib_if_exclude" (current value:
<none>)
Comma-delimited list of HCAs/ports to be excluded
(empty value
means to not exclude any ports). Mutually
exclusive with
btl_openib_if_include.
These parameters are [upcoming] v1.3 only -- they do not exist in the
v1.2 series.
(more below)
On Feb 21, 2008, at 2:45 PM, Bill Wichser wrote:
In trying to get openmpi up and running on a new cluster, I came
across
this error about having both of my IB switches set to the same
subnet-gid. Snooping around on my hosts which run the opensm
daemon, I
indeed found this to be the case in the /var/log/osm-ib[0-1].log
files,
giving up finding it with ibstat which showed these values to be
different, at least the second part of the GID.
Before I try and pursue how to actually change this value for the
opensm
daemon, I do have a question.
Since both of my hosts are connected to each switch, how am I to
instruct openmpi to use port0? I'm trying to use port0 as the MPI
network and port1 as the storage network. Is there something that I
need to add someplace forcing connections only to some default-
subnet-gid?
The v1.3 series will have the parameters that George mentioned above;
those give you fine-grained control about which HCAs and ports you are
using.
In the v1.2 series, you cannot explicitly control which hcas/ports you
are using. Instead, you can only limit the *number* of active ports
that Open MPI will use:
MCA btl: parameter "btl_openib_max_btls" (current value:
"-1")
Maximum number of HCA ports to use (-1 = use all
available, otherwise must be >= 1)
Open MPI starts with the first port on the first interface and goes
upward until it finds max_btls of active ports. This is admittedly
imperfect, but it was only somewhat recently that someone asked for
explicit control over which hcas/ports to use. Sorry... :-\
--
Jeff Squyres
Cisco Systems
