When I use openmpi5, I get the same behavior even with a very small number
of processes per node. However, when I use mpich-ofi it runs fine (see
below). That gives me confidence that the network is setup correctly. The
nodes are connected via infiniband ConnectX-3 adapters, and all ib tests
show no problems.
I found an older post about ucx1.18 having possible issues with openmpi5.
I have assumed that ucx1.18 is now fully compatible with openmpi5. Could
this be the cause? Does anyone use ucx1.8 with openmpi5? If not ucx1.18,
what version is confirmed to work with openmpi5?
My test code:
----------------------------------------------------------------------
[av@c12 test]$ cat mpi_hello_world.c
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name, world_rank, world_size);
// Finalize the MPI environment.
MPI_Finalize();
}
-------------------------------------------------------------------------
[av@c12 test]$ which mpirun
/opt/ohpc/pub/mpi/openmpi5-gnu14/5.0.7/bin/mpirun
[av@sms test]$ mpicc -o openmpi5_hello_world mpi_hello_world.c
[av@sms test]$ salloc -n 4 -N 2
salloc: Granted job allocation 63
salloc: Nodes c[12-13] are ready for job
[av@c12 test]$ mpirun ./openmpi5_hello_world
Hello world from processor c12, rank 0 out of 4 processors
Hello world from processor c12, rank 1 out of 4 processors
Hello world from processor c13, rank 3 out of 4 processors
Hello world from processor c13, rank 2 out of 4 processors
[c12:1709 :0:1709] ud_ep.c:278 Fatal: UD endpoint 0x117ae80 to <no
debug data>: unhandled timeout error
==== backtrace (tid: 1709) ====
0
/opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(ucs_handle_error+0x294)
[0x7f200b4f3ee4]
................
-----------------------------------------------------------------------
[av@sms test]$ which mpicc
/opt/ohpc/pub/mpi/mpich-ofi-gnu14-ohpc/3.4.3/bin/mpicc
[av@sms test]$ which mpirun
/opt/ohpc/pub/mpi/mpich-ofi-gnu14-ohpc/3.4.3/bin/mpirun
[av@sms test]$ mpicc -o mpich-ofi_hello_world mpi_hello_world.c
[av@sms test]$ salloc -n 4 -N 2
salloc: Granted job allocation 66
salloc: Nodes c[12-13] are ready for job
[av@c12 test]$ mpirun ./mpich-ofi_hello_world
Hello world from processor c13, rank 2 out of 4 processors
Hello world from processor c13, rank 3 out of 4 processors
Hello world from processor c12, rank 0 out of 4 processors
Hello world from processor c12, rank 1 out of 4 processors
[av@c12 test]$
------------------------------------------------------------------------
Achilles
On Tuesday, July 1, 2025 at 7:14:06 AM UTC-4 George Bosilca wrote:
> This error message is usually due to a misconfiguration of the network.
> However, I don't think this is the case here because the output contains
> messages from both odd and even ranks (which according to your binding
> policy were placed on different nodes) suggesting at least some of the
> processes were able to connect (and thus the network configuration is
> correct).
>
> So I'm thinking about some timing issues during network setup due to the
> fact that you have many processes per node, and an application that does
> nothing except creating and then shutting down the network layer. Does this
> happen if you have less processes per node ? Does it happen if you add
> anything else in the application (such as an `MPI_Barrier(MPI_COMM_WORLD)`)
> ?
>
> George.
>
>
> On Mon, Jun 30, 2025 at 10:00 PM Achilles Vassilicos <avas...@gmail.com>
> wrote:
>
>> Hello all, new to the list.
>> While testing my openmpi5.0.7 installation using the simple
>> mpi_hello_world.c code, I am experiencing an unexpected behavior where the
>> execution on the last processor rank hangs with a "fatal unhandled timeout
>> error", which leads to core dumps. It confirmed that it happens regardless
>> of the compiler I use, i.e., gnu14 or intel2024.0. Moreover, it does not
>> happen when I use mpich3.4.3-ofi. Below I am including the setting I am
>> using and the runtime error. You will notice that the error happened on
>> node c11, which may suggest that there is something wrong with this node.
>> However, it turns out that any other other node that happens to execute the
>> last processor rank leads to the same error. I must be missing something.
>> Any Thoughts?
>> Sorry about the length of the post.
>>
>> -----------------------------------------------------
>> ]$ module list
>> Currently Loaded Modules:
>> 1) cmake/4.0.0 6) spack/0.23.1 11) mkl/2024.0
>> 16) ifort/2024.0.0 21) EasyBuild/5.0.0
>> 2) autotools 7) oclfpga/2024.0.0 12) intel/2024.0.0
>> 17) inspector/2024.2 22) valgrind/3.24.0
>> 3) hwloc/2.12.0 8) tbb/2021.11 13) debugger/2024.0.0
>> 18) intel_ipp_intel64/2021.10 23) openmpi5/5.0.7
>> 4) libfabric/1.18.0 9) compiler-rt/2024.0.0 14) dpl/2022.3
>> 19) intel_ippcp_intel64/2021.9 24) ucx/1.18.0
>> 5) prun/2.2 10) compiler/2024.0.0 15) icc/2023.2.1
>> 20) vtune/2025.3
>> ----------------------------------------------------------
>> $ sinfo
>> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
>> normal* up infinite 10 idle* c[2-10,12]
>> normal* up infinite 3 idle c[1,11,13]
>> [av@sms test]$ salloc -n 24 -N 2
>> salloc: Granted job allocation 61
>> salloc: Nodes c[1,11] are ready for job
>> [av@c1 test]$ mpirun --display-map --map-by node -x
>> MXM_RDMA_PORTS=mlx4_0:1 -mca btl_openib_if_include mlx4_0:1 mpi_hello_world
>>
>> ======================== JOB MAP ========================
>> Data for JOB prterun-c1-1575@1 offset 0 Total slots allocated 24
>> Mapping policy: BYNODE:NOOVERSUBSCRIBE Ranking policy: NODE Binding
>> policy: NUMA:IF-SUPPORTED
>> Cpu set: N/A PPR: N/A Cpus-per-rank: N/A Cpu Type: CORE
>>
>>
>> Data for node: c1 Num slots: 12 Max slots: 0 Num procs: 12
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 0 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 2 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 4 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 6 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 8 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 10 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 12 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 14 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 16 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 18 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 20 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 22 Bound:
>> package[0][core:0-17]
>>
>> Data for node: c11 Num slots: 12 Max slots: 0 Num procs: 12
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 1 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 3 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 5 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 7 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 9 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 11 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 13 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 15 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 17 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 19 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 21 Bound:
>> package[0][core:0-17]
>> Process jobid: prterun-c1-1575@1 App: 0 Process rank: 23 Bound:
>> package[0][core:0-17]
>>
>> =============================================================
>> Hello world from processor c1, rank 6 out of 24 processors
>> Hello world from processor c1, rank 20 out of 24 processors
>> Hello world from processor c1, rank 16 out of 24 processors
>> Hello world from processor c1, rank 12 out of 24 processors
>> Hello world from processor c1, rank 0 out of 24 processors
>> Hello world from processor c1, rank 2 out of 24 processors
>> Hello world from processor c1, rank 14 out of 24 processors
>> Hello world from processor c1, rank 10 out of 24 processors
>> Hello world from processor c1, rank 4 out of 24 processors
>> Hello world from processor c1, rank 22 out of 24 processors
>> Hello world from processor c1, rank 18 out of 24 processors
>> Hello world from processor c1, rank 8 out of 24 processors
>> Hello world from processor c11, rank 11 out of 24 processors
>> Hello world from processor c11, rank 1 out of 24 processors
>> Hello world from processor c11, rank 3 out of 24 processors
>> Hello world from processor c11, rank 13 out of 24 processors
>> Hello world from processor c11, rank 19 out of 24 processors
>> Hello world from processor c11, rank 7 out of 24 processors
>> Hello world from processor c11, rank 17 out of 24 processors
>> Hello world from processor c11, rank 21 out of 24 processors
>> Hello world from processor c11, rank 15 out of 24 processors
>> Hello world from processor c11, rank 23 out of 24 processors
>> Hello world from processor c11, rank 9 out of 24 processors
>> Hello world from processor c11, rank 5 out of 24 processors
>> [c11:2028 :0:2028] ud_ep.c:278 Fatal: UD endpoint 0x1c8da90 to <no
>> debug data>: unhandled timeout error
>> [c11:2035 :0:2035] ud_ep.c:278 Fatal: UD endpoint 0x722a90 to <no
>> debug data>: unhandled timeout error
>> [c11:2025 :0:2025] ud_ep.c:278 Fatal: UD endpoint 0xc52a90 to <no
>> debug data>: unhandled timeout error
>> ==== backtrace (tid: 2028) ====
>> 0
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(ucs_handle_error+0x294)
>> [0x7fade4326ee4]
>> 1
>>
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(ucs_fatal_error_message+0xb2)
>>
>> [0x7fade4324292]
>> 2 /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x2f369)
>> [0x7fade4324369]
>> 3 /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/ucx/libuct_ib.so.0(+0x263f0)
>> [0x7fade110d3f0]
>> 4 /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x24987)
>> [0x7fade4319987]
>> 5
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucp.so.0(ucp_worker_progress+0x2a)
>> [0x7fade43abc9a]
>> 6
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(+0xa09bc)
>> [0x7fade471b9bc]
>> 7
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs_nofence+0x6a)
>>
>> [0x7fade471b79a]
>> 8
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs+0x20)
>>
>> [0x7fade471baf0]
>> 9
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(mca_pml_ucx_del_procs+0x140)
>>
>> [0x7fade4d1cd70]
>> 10 /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(+0xac837)
>> [0x7fade4b27837]
>> 11
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_finalize_cleanup_domain+0x53)
>>
>> [0x7fade46aebd3]
>> 12
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_finalize+0x2e)
>>
>> [0x7fade46a22be]
>> 13
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(ompi_rte_finalize+0x1f9)
>>
>> [0x7fade4b21909]
>> 14 /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(+0xab304)
>> [0x7fade4b26304]
>> 15
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(ompi_mpi_instance_finalize+0xe5)
>>
>> [0x7fade4b26935]
>> 16
>>
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(ompi_mpi_finalize+0x3d1)
>>
>> [0x7fade4b1e091]
>> 17 mpi_hello_world() [0x40258f]
>> 18 /lib64/libc.so.6(+0x295d0) [0x7fade47b95d0]
>> 19 /lib64/libc.so.6(__libc_start_main+0x80) [0x7fade47b9680]
>> 20 mpi_hello_world() [0x402455]
>> =================================
>> [c11:02028] *** Process received signal ***
>> [c11:02028] Signal: Aborted (6)
>> [c11:02028] Signal code: (-6)
>> [c11:02028] [ 0] /lib64/libc.so.6(+0x3ebf0)[0x7fade47cebf0]
>> [c11:02028] [ 1] /lib64/libc.so.6(+0x8bedc)[0x7fade481bedc]
>> [c11:02028] [ 2] /lib64/libc.so.6(raise+0x16)[0x7fade47ceb46]
>> [c11:02028] [ 3] /lib64/libc.so.6(abort+0xd3)[0x7fade47b8833]
>> [c11:02028] [ 4]
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x2f297)[0x7fade4324297]
>> [c11:02028] [ 5]
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x2f369)[0x7fade4324369]
>> [c11:02028] [ 6]
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/ucx/libuct_ib.so.0(+0x263f0)[0x7fade110d3f0]
>> [c11:02028] [ 7]
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x24987)[0x7fade4319987]
>> [c11:02028] [ 8]
>> /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucp.so.0(ucp_worker_progress+0x2a)[0x7fade43abc9a]
>> [c11:02028] [ 9]
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(+0xa09bc)[0x7fade471b9bc]
>> [c11:02028] [10]
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs_nofence+0x6a)[0x7fade471b79a]
>> [c11:02028] [11]
>> /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs+0x20)[0x7fade471baf0]
>> c11:02028] [12] ==== backtrace (tid: 2035) ====
>> ..................
>>
>> --------------------------------------------------------------------------------
>> Achilles
>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to users+un...@lists.open-mpi.org.
>>
>
To unsubscribe from this group and stop receiving emails from it, send an email
to users+unsubscr...@lists.open-mpi.org.
