Gary,
Looking at the trace it become clear that the culprit is the receiving process
and not the sending one.
Let me walk you over. I split the trace in several groups. The first one will
look at the sender (proc39) and the receiver (proc28) for the first message.
PROC39: 59.2268409810; SEND_R
Talked to one of our developers (afraid everyone is swamped with SC12 right
now), and we have a few suggestions:
1. to reduce OS-jitter, have you tried --bind-to-core or --bind-to-socket? This
would avoid context switches throwing you onto different memory segments
2. we recall you had an issue
Answering my own question, I have downloaded openMPI 1.6.2 and still get the
delay in the MPI_Send. Previously, I was using openMPI 1.4.1
I configured 1.6.2 with -enable-peruse and have implemented PERUSE callbacks.
Here is a trace of gb2 (PROC39) sending messages to ob (PROC28)
PROC39: 59.2268
