[OMPI users] mca_oob_tcp_peer_try_connect error when checkpoint and restart.

[Hiep Bui Hoang]

Hi,
I had setup Open MPI "trunk_16171" for 3 computers with Lan connection, and
set environment parameters, ssh without typing password for each node. I use
Red Hat Enterprise Linux 5. The program I tried is 'send_recv'. I run
successful my 'send_recv' program in those 3 nodes. And checkpoint/restart
successful on 1 node. But I had error when try to checkpoint/restart that
program on 3 nodes.

    $ mpirun -np 4 -host 172.28.11.40,172.28.11.28,172.28.11.18 -am
ft-enable-cr send_recv

....
Send 32 from rank 0
 Receive 32 at rank 1
Send 33 from rank 0
 Receive 33 at rank 1
[HNP:05700] [1,2]-[1,3] mca_oob_tcp_peer_try_connect: connect to
172.28.11.40:3680 failed: Software caused connection abort (103)
[HNP:05700] [1,2]-[1,3] mca_oob_tcp_peer_try_connect: connect to
172.28.11.40:3680 failed, connecting over all interfaces failed!
[node2:04837] [1,1]-[1,3] mca_oob_tcp_peer_try_connect: connect to
172.28.11.40:3680 failed: Software caused connection abort (103)
[node2:04837] [1,1]-[1,3] mca_oob_tcp_peer_try_connect: connect to
172.28.11.40:3680 failed, connecting over all interfaces failed!
 Receive 34 at rank 1
Send 34 from rank 0
.....

PID of above mpirun is 5693.
    $ ompi-checkpoint 5693
--------------------------------------------------------------------------
Error: The application (PID = 5693) failed to checkpoint properly.
       Returned -1.

--------------------------------------------------------------------------

Somebody know about this error?
Thanks.

This is my 'send_recv' program:

main(int argc, char **argv)
{
   int node;
   int MAX = 1000;
   MPI_Status status;
   MPI_Init(&argc,&argv);
   MPI_Comm_rank(MPI_COMM_WORLD, &node);

   int    i = 0;
   while( i <= MAX){
    if( 0 == node){
        MPI_Send(&i, 1, MPI_INT, 1, 10, MPI_COMM_WORLD);
        printf("Send %d from rank %d \n",i, node);
        sleep(1);
    }
    if( 1 == node ){
        MPI_Recv(&i, 1, MPI_INT, 0, 10, MPI_COMM_WORLD,
                     &status);
        printf(" Receive %d at rank %d \n",i,node);
        sleep(1);
    }
    i++;
   }
   MPI_Finalize();
}