Hi Roman
I googled out and found that CPMD is a molecular dynamics program.
(What would be of civilization without Google?)
Unfortunately I kind of wiped off from my mind
Schrodinger's equation, Quantum Mechanics,
and the Born approximation,
which I learned probably before you were born.
I couldn't find any short description of the CPMD algorithm,
or a good diagram with the mesh, problem size, whatever that might
clarify how the algorithm works.
Hence, it is still hard for me to tell what type of scaling of CPMD
to expect.
There must be something in the manual, if you read it.
However, they do mention mesh size, problem size, etc,
several times, and if you dig out, you'll find the parameters
that control scaling.
The topmost parameter that should control scaling is probably the
problem size or mesh size, but there may be other things,
such as their controlling variables "taskgroups",
"blocksize states", etc.
Not knowing the algorithm I can only guess.
Anyway, the CPMD manual has
some recommendations on how to divide the task
across processors in a meaningful (and efficient) way,
which you should read:
CPMD on Parallel Computers
http://www.cpmd.org/manual/node52.html
Paralellizing CPMD with MPI:
http://www.cpmd.org/manual/node53.html
For some reason, all the "CPMD 32 water" benchmarks on their
web page stop at 32 processors (except for one with 128 processors ran
on a Cray-TE-600,
which is a different beast than a beowulf cluster).
I suggest that you read the "Parallel Performance" section and figures,
as some sentences clearly indicate that some problem sizes are not
large enough to require more than a few tens of processors:
http://www.theochem.ruhr-uni-bochum.de/~axel.kohlmeyer/cpmd-bench.html
http://www.theochem.ruhr-uni-bochum.de/~axel.kohlmeyer/cpmd-bench.html#parallel
Is this perhaps because the size of the problem doesn't justify using
more than 32 processors?
What is the meaning of the "32" on "CPMD 32 water"?
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
1=
Gus Correa wrote:
Hi Roman
Just a guess.
Is this a domain decomposition code?
(I never heard about "cpmd 32 waters" before, sorry.)
Is it based on finite differences, finite volume, finite element?
If it is, once the size of the subdomains becomes too small compared to
the size of the halo around them, the overhead required to calculate
your solution for the halo swamps the whole calculation,
and scaling degrades.
This is not an MPI scaling problem, this is intrinsic to the domain
decomposition technique.
Typically this happens as the number of processors reach some high
number (which depends on the size of the problem).
So, what you are seeing may not be a problem with OpenMPI scaling,
but just that your problem is not large enough to require the use of,
say, 48 or 64 processors.
For instance, imagine a 1D problem with a grid with 1024 points,
that require a 2 grid point overlap (halo) on the left and right
of any subdomain to be calculated in parallel (i.e. decomposing the
domain in parts).
If you divide the domain across two processors only, each processor
has to work not on 1024/2=512 points, but on 512+2+2=516 points.
The calculation on the two processors gets an overhead of 2*(2+2)=8 grid
points,w.r.t. the same calculation done on a single processor.
This is an overhead of 8/1024=0.8% only, so using 2 processors
will speedup the calculation by a factor close to 2 (but slightly lower).
However, if you divide the same problem across 64 subdomains (i.e 64
processors), the size of each subdomain is 1024/64=16,
plus 2 halo grid point on each side, i.e. 20 grid points.
So the overhead is much higher now, 4/16=25%.
Dividing the problem across 64 processors will not speed it up by
a factor of 64, but by much less.
Every domain decomposition program that we have here shows this
effect. If we give them more processors they scale well, up
to a point (say 16 or 32 processors, for a reasonably sized problem).
However, beyond that point the scaling slowly flattens out.
When you go and look at the grid size and the
large number of processors,
you realize that most of the effort is being done to calculate halos,
i.e. on overhead.
On top of that, there is the overhead due to MPI communication, of
course, but it is likely that the halo overhead is the dominant factor.
I would guess other classes of problems and parallel methods of solution
also have the same problem that domain decomposition shows.
Is this perhaps what is going on with your test code?
Take a look at the code to see what it is doing,
and in particularly, what is the problem size.
See if it really makes sense to distribute it over 64 processors,
of if a smaller number would be the right choice.
Also, if the program allows you to change the problem size,
try the test again with a larger problem size
(say, two or four times bigger),
and then go up to a large number of processors also.
With a larger problem size the scaling may be better too
(but the runtimes will grow as well).
Finally, since you are using Infiniband, and I wonder if all the
nodes connect to each other with the same latency, or if some
pairs of nodes have higher latency to communicate.
On a single switch hopefully the latency is the same for all pairs of
nodes.
However, if you connect two switches, for instance, nodes that
are on switch A will probably have a larger latency to talk
to nodes on switch B, I suppose.
I hope it helps.
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Roman Martonak wrote:
Hello,
I observe very poor scaling with openmpi on HP blade system consisting
of 8 blades (each having 2 quad-core AMD Barcelona 2.2 GHz CPU) and
interconnected with Infiniband fabric. When running the standard cpmd
32 waters test, I observe the following scaling (the numbers are
elapsed time)
openmpi-1.2.6:
using full blades (8 cores per blade)
np8 7 MINUTES 26.40 SECONDS
np16 4 MINUTES 19.91 SECONDS
np32 2 MINUTES 55.51 SECONDS
np48 2 MINUTES 38.18 SECONDS
np64 3 MINUTES 19.78 SECONDS
I tried also openmpi-1.2.8 and openmpi-1.3 and it is about the same,
openmpi-1.3 is somewhat better for 32 cores but in all cases there is
practically no scaling beyond 4 blades (32 cores) and running on 64
cores is a disaster. With Intel MPI, however, I get the following
numbers
Intel MPI-3.2.1.009
using full blades (8 cores per blade)
np8 7 MINUTES 23.19 SECONDS
np16 4 MINUTES 22.17 SECONDS
np32 2 MINUTES 50.07 SECONDS
np48 1 MINUTES 42.87 SECONDS
np64 1 MINUTES 23.76 SECONDS
so there is reasonably good scaling up to 64 cores. I am running with
the option
--mca mpi_paffinity_alone 1, I have tried also -mca btl_openib_use_srq
1 but it had only marginal effect. With mvapich I get similar scaling
as with Intel MPI. The system is running the Rocksclusters
distribution 5.1 with the mellanox ofed-1.4 roll. I would be grateful
if somebody could suggest me what could be the origin of the problem
and how to tune openmpi to get better scaling.
Many thanks in advance.
Best regards
Roman
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
