On 8/23/2011 1:24 PM, Dick Kachuma wrote:
I have used gprof to profile a program that uses openmpi. The result
shows that the code spends a long time in poll (37% on 8 cores, 50% on
16 and 85% on 32). I was wondering if there is anything I can do to
reduce the time spent in poll.
In serial performance optimization, if you spend a lot of time in a
function, you try to speed that function up. In parallel programming,
if you spend a lot of time in a function that waits, speeding the
function up probably will not help. You need to speed up the thing
you're waiting on. In the case of poll, it is quite likely that the
issue is not that poll is slow, but you're waiting on someone else.
Other performance tools might help here. Check
http://www.open-mpi.org/faq/?category=perftools E.g., a timeline view
of your run might be able to show you what other processes are doing
while the long-poll process is idling.
I cannot determine the number of calls
made to poll and exactly where they are. The bulk of my code uses
exclusively MPI_Ssend for the send and MPI_Irecv and MPI_Wait for the
receive. For instance, would there be any gain expected if I switch
from MPI_Ssend to MPI_Send?
It depends. Try to find out which MPI calls are taking a lot of time.
How long are the messages you're sending? If they're short and you're
spending a lot of time in MPI_Ssend, then switching to MPI_Send could help.
Alternatively would there be any gain in
switching to MPI_Isend/MPI_Recv instead of MPI_Ssend/MPI_Irecv?
Using Isend and Irecv can help if you can do useful work during the time
you're waiting for a non-blocking operation to complete.
Before considering too many strategies, however, it may make most sense
to get more performance information on your application. Which MPI
calls are taking the most time? What message patterns characterize your
slowdown? Are all processes spending lots of time in MPI, or is there
one process that is busy in computation and upon whom everyone else is
waiting?
