Strange - it looks like a classic oversubscription behavior. Another
possibility is that it isn't using IB for some reason when extended to the
other nodes. What does your cmd line look like? Have you tried adding "-mca btl
openib,sm,self" just to ensure it doesn't use TCP for some reason?
On
Correct. 20 nodes, 8 cores per dual-socket on each node = 360.
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf
Of Ralph Castain
Sent: Sunday, June 09, 2013 6:18 PM
To: Open MPI Users
Subject: Re: [OMPI users] EXTERNAL: Re: Need advice on performance problem
So, ju
So, just to be sure - when you run 320 "cores", you are running across 20 nodes?
Just want to ensure we are using "core" the same way - some people confuse
cores with hyperthreads.
On Jun 9, 2013, at 3:50 PM, "Blosch, Edwin L" wrote:
> 16. dual-socket Xeon, E5-2670.
>
> I am trying a larger
16. dual-socket Xeon, E5-2670.
I am trying a larger model to see if the performance drop-off happens at a
different number of cores.
Also I'm running some intermediate core-count sizes to refine the curve a bit.
I also added mpi_show_mca_params all, and at the same time,
btl_openib_use_eager_rd
Looks to me like things are okay thru 160, and then things fall apart after
that point. How many cores are on a node?
On Jun 9, 2013, at 1:59 PM, "Blosch, Edwin L" wrote:
> I’m having some trouble getting good scaling with OpenMPI 1.6.4 and I don’t
> know where to start looking. This is an In
I'm having some trouble getting good scaling with OpenMPI 1.6.4 and I don't
know where to start looking. This is an Infiniband FDR network with Sandy
Bridge nodes. I am using affinity (--bind-to-core) but no other options. As
the number of cores goes up, the message sizes are typically going do
