Thanks Jeff.
>>1. Can you create a small example to reproduce the problem?
>>2. The TCP and verbs-based transports use different thresholds and
protocols, and can sometimes bring to light errors in the application
(e.g., the application is making assumptions that just happen to be true
for TCP, but not necessarily for other transports).
>>3. Is your program multi-threaded? If so, MPI_THREAD_MULTIPLE support in
the v1.8 and v1.10 series is not fully baked.
>>4. Additionally, if you have buffering / matching / progression
assumptions in your application, you might accidentally block. An
experiment to try to is to convert all MPI_SEND and MPI_ISEND to MPI_SSEND
and MPI_ISSEND, respectively, and see if your program still functions
properly on TCP.
1. I will try to create a mall example to reproduce the problem.
2. I didn't get your point. I didn't make any assumptions for TCP. Is there
any difference in MPI for TCP and RDMA?
3. My program doesn't enable MPI_THREAD_MULTIPLE
4. what do you mean by buffering / matching / progression assumptions in
your application?
My program communicates like this:
4 processes: process0, 1, 2, 3
process1/process3:
foreach to_id in process0, process2:
MPI_Send(send_buf, sendlen, to_id, TAG);
MPI_Recv(recv_buf, recvlen, to_id, TAG);
process0/process2:
while(true):
MPI_recv(recv_buf, any_source, TAG);
MPI_Send(send_buf, source_id, TAG);
2016-01-21 17:49 GMT+08:00 Eva <wuzh...@gmail.com>:
> Gilles,
> Actually, there are some more strange things.
> With the same environment and MPI version, I write a simple program by
> using the same communication logic with my hang program.
> The simple program can work without hang.
> So is there any possible reason? I can try them one by one.
> Or can I debug into the openib source code to find the root cause with
> your instructions or guide?
>
> 2016-01-21 17:03 GMT+08:00 Eva <wuzh...@gmail.com>:
>
>> Gilles,
>> >>Can you try to
>> >>mpirun --mca btl tcp,self --mca btl_tcp_eager_limit 56 ...
>> >>and confirm it works fine with TCP *and* without eager ?
>>
>> I have tried this and it works.
>> So what should I do next?
>>
>>
>> 2016-01-21 16:25 GMT+08:00 Eva <wuzh...@gmail.com>:
>>
>>> Thanks Gilles.
>>> it works fine on tcp
>>> So I use this to disable eager:
>>> -mca btl_openib_use_eager_rdma 0 -mca btl_openib_max_eager_rdma 0
>>>
>>> 2016-01-21 13:10 GMT+08:00 Eva <wuzh...@gmail.com>:
>>>
>>>> I run with two machines, 2 process per node: process0, process1,
>>>> process2, process3.
>>>> After some random rounds of communications, the communication hangs.
>>>> When I debug into the program, I found:
>>>> process1 sent a message to process2;
>>>> process2 received the message from process1 and then start to receive
>>>> messages from other processes.
>>>> But process1 doesn't get notice: process2 has received its message and
>>>> then hang on MPI_Send->...->poll_device() of rdmav2.
>>>>
>>>> #0 0x00007f6ba95f03e5 in ?? () from /usr/lib64/libmlx4-rdmav2.so
>>>> #1 0x00007f6bacf1ed93 in poll_device () from
>>>> /home/openmpi-1.8.5-gcc4.8/lib/openmpi/mca_btl_openib.so
>>>> #2 0x00007f6bacf1f7ed in btl_openib_component_progress () from
>>>> /home/openmpi-1.8.5-gcc4.8/lib/openmpi/mca_btl_openib.so
>>>> #3 0x00007f6bb06539da in opal_progress () from
>>>> /home/openmpi-1.8.5-gcc4.8/lib/libopen-pal.so.6
>>>> #4 0x00007f6bab831f55 in mca_pml_ob1_send () from
>>>> /home/openmpi-1.8.5-gcc4.8/lib/openmpi/mca_pml_ob1.so
>>>> #5 0x00007f6bb0df33c2 in PMPI_Send () from
>>>> /home/openmpi-1.8.5-gcc4.8/lib/libmpi.so.1
>>>>
>>>> Some experiments I have tried:
>>>> 1. compile openmpi without multi-thread enable
>>>> 2. --mca pml_ob1_use_early_completion 0
>>>> 3. disable eager mode
>>>> 4. ssend, Bsend
>>>>
>>>> but it still hangs.
>>>>
>>>> The same program works fine on TCP for more than one year. After I move
>>>> it onto rdma, it starts to hang. And I can't debug into any rdma details
>>>>
>>>> 2016-01-21 11:24 GMT+08:00 Eva <wuzh...@gmail.com>:
>>>>
>>>>> Run MPI_Send on MPI1.8.5 without multithread enabled:
>>>>> it hangs on mca_pml_ob1_send() -> opal_progreses() ->
>>>>> btl_openib_component_progress() -> poll_device() -> libmlx4-rdmav2.so ->
>>>>> cq
>>>>> -> phread_spin_unlock
>>>>> The program can run on TCP with no error.
>>>>>
>>>>
>>>>
>>>
>>
>
