[OMPI users] Debugging memory use of Open MPI

[Shaun Jackman]

When running my Open MPI application, I'm seeing three processors that 
are using five times as much memory as the others when they should all 
use the same amount of memory. To start the debugging process, I would 
like to know if it's my application or the Open MPI library that's 
using the additional memory. Does anyone have any tips on calculating 
the amount of memory that Open MPI is using at a given moment in time?

My Open MPI application makes significant use of MPI_Irecv and 
MPI_Send. Every process has exactly one MPI_Irecv request active at 
any time. When it receives a message, it handles it, possibly 
transmits a response packet using MPI_Send, and starts a new MPI_Irecv 
request. What happens if one process is slow and lags behind? Will the 
messages be buffered at the sender or the receiver? Will the messages 
be buffered at the Open MPI level or at the OS level, say in a TCP 
transmit buffer or receive buffer?

Thanks,
Shaun