Dear all,
I am reinstalling a cluster where nodes are connected by
- 1 Gb ethernet interfaces
- 40 Gb infinband adapters
I installed OFED 3.12 on CentOS 6.5
I would like to be able to tell mpirun to use either the Gb ethernet
interfaces or the infinband adapters.
But, when I launch osu_bw with the following commands I always obtain
bandwidths of the infinibands links
(hostether is a file containing the IPs of the ethernet interfaces
IPs of ib adapters: 192.168.1.2,192.168.1.3
IPs of ethernet adapters: 192.168.70.12,192.168.70.13 )
mpirun --map-by node --mca btl ^openib --hostfile ./hostether -np 2
./osu_bw
mpirun --map-by node --mca btl_tcp_if_include eth0,eth1 --mca btl
tcp,self --hostfile ./hostether -np 2 ./osu_bw
mpirun --map-by node --mca btl_tcp_if_exclude ib0,ib1 --mca btl tcp,self
--hostfile ./hostether -np 2 ./osu_bw
mpirun --map-by node --mca btl_tcp_if_exclude
192.168.1.2,192.168.1.3,127.0.0.1/8,sppp --mca btl tcp,self --host
192.168.70.12,192.168.70.13 -np 2 ./osu_bw
All the previous commands lead to the following results with
either Open MPI 1.5.5, 1.7.5 or 1.8.1:
# OSU MPI Bandwidth Test (Version 2.3)
# Size Bandwidth (MB/s)
1 2.14
... ...
2097152 3015.64
4194304 3017.91
If I stop openibd I logically obtain these results:
# OSU MPI Bandwidth Test (Version 2.3)
# Size Bandwidth (MB/s)
1 0.19
... ...
2097152 117.63
4194304 117.66
Could someone give me some idea to investigate?
PS. I previouslly used this cluster with older versions
of CentOS and OFED and I was able to tell mpirun not to use
the infiniband adapters.
Best regards.
