I am sorry for the delay in replying; this week got a bit crazy on me.
I'm guessing that Open MPI is striping across both your eth0 and ib0 interfaces.
You can limit which interfaces it uses with the btl_tcp_if_include MCA param.
For example:
# Just use eth0
mpirun --mca btl tcp,sm,self --mca btl_tcp_if_include eth0 ...
# Just use ib0
mpirun --mca btl tcp,sm,self --mca btl_tcp_if_include ib0 ...
Note that IPoIB is nowhere near as efficient as native verbs, so you won't get
nearly as good performance as you do with OMPI's openib transport.
Note, too, that I specifically included "--mca btl tcp,sm,self" in the above
examples to force the use of the TCP MPI transport. Otherwise, OMPI may well
automatically choose the native IB (openib) transport. I see you mentioned
this in your first mail, too, but I am listing it here just to be
specific/pedantic.
On May 22, 2014, at 3:30 AM, Bibrak Qamar <bibr...@gmail.com> wrote:
> Hi,
>
> I am facing problem in running Open MPI using TCP (on 1G Ethernet). In
> practice the bandwidth must not exceed 1000 Mbps but for some data points
> (for point-to-point ping pong) it exceeds this limit. I checked with MPICH it
> works as desired.
>
> Following is the command I issue to run my program on TCP. Am I missing
> something?
>
> -bash-3.2$ mpirun -np 2 -machinefile machines -N 1 --mca btl tcp,self
> ./bandwidth.ompi
> --------------------------------------------------------------------------
> The following command line options and corresponding MCA parameter have
> been deprecated and replaced as follows:
>
> Command line options:
> Deprecated: --npernode, -npernode
> Replacement: --map-by ppr:N:node
>
> Equivalent MCA parameter:
> Deprecated: rmaps_base_n_pernode, rmaps_ppr_n_pernode
> Replacement: rmaps_base_mapping_policy=ppr:N:node
>
> The deprecated forms *will* disappear in a future version of Open MPI.
> Please update to the new syntax.
> --------------------------------------------------------------------------
> Hello, world. I am 1 on compute-0-16.local
> Hello, world. I am 0 on compute-0-15.local
> 1 25.66 0.30
> 2 25.54 0.60
> 4 25.34 1.20
> 8 25.27 2.42
> 16 25.24 4.84
> 32 25.49 9.58
> 64 26.44 18.47
> 128 26.85 36.37
> 256 29.43 66.37
> 512 36.02 108.44
> 1024 42.03 185.86
> 2048 194.30 80.42
> 4096 255.21 122.45
> 8192 258.85 241.45
> 16384 307.96 405.90
> 32768 422.78 591.32
> 65536 790.11 632.83
> 131072 1054.08 948.70
> 262144 1618.20 1235.94
> 524288 3126.65 1279.33
>
> -Bibrak
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
