Hi all,
I have a program that executes a communication loop similar to this one:
1: for(int p1=0; p1<np; ++p1) {
2: for(int p2=0; p2<np; ++p2) {
3: if(me==p1) {
4: if(sendSize(p2))
MPI_Ssend(sendBuffer[p2],sendSize(p2),MPI_FLOAT,p2,0,myw);
5: if(recvSize(p2))
MPI_Recv(recvBuffer[p2],recvSize(p2),MPI_FLOAT,p2,0,myw,&status);
6: } else if(yo==p2) {
7: if(recvSize(p1))
MPI_Recv(recvBuffer[p1],recvSize(p1),MPI_FLOAT,p2,0,myw,&status);
8: if(sendSize(p1))
MPI_Ssend(sendBuffer[p1],sendSize(p1),MPI_FLOAT,p2,0,myw);
9: }
10: MPI_Barrier(myw);
11: }
12: }
The program is an iterative process that makes some calculations,
communicates and then continues with the next iteration. The problem is
that after making 30 successful iterations the program hangs. With padb
I have seen that one of the processors waits at line 5 for the reception
of data that was already sent and the rest of the processors are waiting
at the barrier in line 10. The size of the messages and buffers is the
same for all the iterations.
My real program makes use of asynchronous communications for obvious
performance reasons and it worked without problems when the case to
solve was smaller (lower number of processors and memory), but I found
that for this case the program hanged and that is why a changed the
communication routine using synchronous communications to see where is
the problem. Now I know where the program hangs, but I don't understand
what I am doing wrong.
Any suggestions?
More specific data of the case and cluster:
Number of processors: 320
Max size of the message: 6800 floats (27200 bytes)
Number of cores by node: 32
File system: lustre
Resource manager: slurm
OMPI version: 1.4.4
Operative system: Ubuntu 10.04.4 LTS
Kernel: RHEL 6.2 2.6.32-220.4.2
Infiniband: OFED 1.4.2
InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s -
IB QDR / 10GigE] (rev b0)
Thank you for your time,
Jorge
--
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que est� net.
