RightCFD wrote:
Date:
Thu, 29 Oct 2009 15:45:06 -0400
From: Brock Palen <bro...@umich.edu>
Subject: Re: [OMPI users] mpi functions are slow when first called and
become normal afterwards
To: Open MPI Users <us...@open-mpi.org>
Message-ID: <890cc430-68b0-4307-8260-24a6fadae...@umich.edu>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
> When MPI_Bcast and MPI_Reduce are called for the first time, they
> are very slow. But after that, they run at normal and stable speed.
> Is there anybody out there who have encountered such problem? If
you
> need any other information, please let me know and I'll provide.
> Thanks in advance.
This is expected, and I think you can dig though the message archive
to find the answer. OMPI does not wireup all the communication at
startup, thus the first time you communicate with a host the
connection is made, but after that it is fast because it is already
open. This behavior is expected, and is needed for very large systems
where you could run out of sockets for some types of communication
with so many hosts.
Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
bro...@umich.edu
(734)936-1985
Thanks for your reply. I am surprised to know this is an expected
behavior of OMPI. I searched the archival but did not find many
relevant messages. I am wondering why other users of OMPI do not
complain this. Is there a way to avoid this when timing an MPI program?
An example of this is the NAS Parallel Benchmarks, which have been
around nearly 20 years. They:
*) turn timers on after MPI_Init and off before MPI_Finalize
*) execute at least one iteration before starting timers
Even so, with at least one of the NPB tests and with at least one MPI
implementation, I've seen more than one iteration needed to warm things
up. That is, if you timed each iteration, you could see that multiple
iterations were needed to warm everything up. In performance analysis,
it is reasonably common to expect to have to run multiple iterations
and correct data set size to get representative behavior.
|
