Re: [OMPI users] MPI_Probe succeeds, but subsequent MPI_Recv gets stuck

[Daniel Rozenbaum]

Yes, a memory bug has been my primary focus due to the not entirely 
consistent nature of this problem; I valgrind'ed the app a number of 
times, to no avail though. Will post again if anything new comes up... 
Thanks!

Jeff Squyres wrote:
Yes, that's the normal progression.  For some reason, OMPI appears to  
have decided that it had not yet received the message.  Perhaps a  
memory bug in your application...?  Have you run it through valgrind,  
or some other memory-checking debugger, perchance?

On Oct 18, 2007, at 12:35 PM, Daniel Rozenbaum wrote:

  
Unfortunately, so far I haven't even been able to reproduce it on a
different cluster. Since I had no success getting to the bottom of  
this
problem, I've been concentrating my efforts on changing the app so  
that
there's no need to send very large messages; I might be able to find
time later to create a short example that shows the problem.

FWIW, when I was debugging it, I peeked a little into Open MPI  
code, and
found that the client's MPI_Recv gets stuck in mca_pml_ob1_recv(),  
after
it determines that "recvreq- 
    
req_recv.req_base.req_ompi.req_complete ==
      
false" and calls opal_condition_wait().

Jeff Squyres wrote:
    
Can you send a short test program that shows this problem, perchance?


On Oct 3, 2007, at 1:41 PM, Daniel Rozenbaum wrote:


      
Hi again,

I'm trying to debug the problem I posted on several times recently;
I thought I'd try asking a more focused question:

I have the following sequence in the client code:
MPI_Status stat;
ret = MPI_Probe(0, MPI_ANY_TAG, MPI_COMM_WORLD, &stat);
assert(ret == MPI_SUCCESS);
ret = MPI_Get_elements(&stat, MPI_BYTE, &count);
assert(ret == MPI_SUCCESS);
char *buffer = malloc(count);
assert(buffer != NULL);
ret = MPI_Recv((void *)buffer, count, MPI_BYTE, 0, stat.MPI_TAG,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
assert(ret == MPI_SUCCESS);
fprintf(stderr, "MPI_Recv done\n");
<proceed to taking action on the received buffer, send response to
server>
Each MPI_ call in the lines above is surrounded by debug prints
that print out the client's rank, current time, the action about to
be taken with all its parameters' values, and the action's result.
After the first cycle (receive message from server -- process it --
send response -- wait for next message) works out as expected, the
next cycle get stuck in MPI_Recv. What I get in my debug prints is
more or less the following:
MPI_Probe(source= 0, tag= MPI_ANY_TAG, comm= MPI_COMM_WORKD,
status= <address1>)
MPI_Probe done, source= 0, tag= 2, error= 0
MPI_Get_elements(status= <address1>, dtype= MPI_BYTE, count=
<address2>)
MPI_Get_elements done, count= 2731776
MPI_Recv(buf= <address3>, count= 2731776, dtype= MPI_BYTE, src= 0,
tag= 2, comm= MPI_COMM_WORLD, stat= MPI_STATUS_IGNORE)
<nothing beyond this point. Some time afterwards there're "readv
failed" errors in server's stderr>
My question then is this - what would cause MPI_Recv to not return,
after the immediately preceding MPI_Probe and MPI_Get_elements
return properly?

Thanks,
Daniel