Ioannis Papadopoulos wrote:
Has anyone observed similar behaviour? Is it something that I'll have
to deal with it in my code or does it indeed qualify as an issue to be
looked into?
I would say this is NOT an issue that merits much attention. There are
too many potential performance anomalies that you might be encountering
and they aren't worth "fixing" (or even understanding) unless they
impact your application's performance in a meaningful way.
E.g., try timing "nothing". Here is a sample test program:
#include <stdio.h>
#include <mpi.h>
#define N 1000000
int main(int argc, char **argv) {
int i;
double t[N], tavg = 0, tmin = 1.e20, tmax = 0;
MPI_Init(&argc,&argv);
for ( i = 0; i < N; i++ ) {
t[i] = MPI_Wtime();
t[i] = MPI_Wtime() - t[i];
}
for ( i = 0; i < N; i++ ) {
tavg += t[i];
if ( tmin > t[i] ) tmin = t[i];
if ( tmax < t[i] ) tmax = t[i];
}
tavg /= N;
printf("avg %12.3lf\n", tavg * 1.e6);
printf("min %12.3lf\n", tmin * 1.e6);
printf("max %12.3lf\n", tmax * 1.e6);
MPI_Finalize();
return 0;
}
I find that the minimum is 0 (indicating non-infinitesimal granularity
of the timer), the average is small (some overhead of the timer call),
and the maximum is very large. Why? Because something will happen now
and then. What it is doesn't matter unless your application's
performance is suffering.
You report that the overall time is about the same. That is, it takes
just over a second to receive the message, which is expected if the
sender delays a second before sending.
One of the things you could do is look at total time to receive the
message and total time spent in MPI_Test. Then, vary TIMEOUT more
smoothly (0.000001, 0.000002, 0.000005, 0.00001, 0.00002, 0.00005,
0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02). You may also
have to run many times to see how reproducible the results are. As
TIMEOUT increases, the total time to get the message will roughly
increase, but not by much until TIMEOUT gets pretty large. The total
time spent in MPI_Test should fall as TIMEOUT increases. So, the idea
is that by increasing TIMEOUT, you decrease the responsiveness of the
receiver while you make more CPU time available for other tasks. With
any luck, there will be a broad range of TIMEOUT values that degrade
responsiveness negligibly while freeing a meaningful amount of time up
for other computational tasks.
The performance of MPI_Test() -- and of a particular MPI_Test() call --
is probably not very meaningful.
Note that your MPI_Irecv calls should strictly speaking have
MPI_ANY_SOURCE rather than MPI_ANY_TAG.
