Hi Gilles, I created the ticket (#12296). The crash happened with either 1 or 2 MPI ranks (have not tried with more but I doubt that it would make any difference). Thanks, Arturo
Gilles Gouaillardet via users wrote: Hi, please open an issue on GitHub at https://github.com/open-mpi/ompi/issues <https://github.com/open-mpi/ompi/issues> and provide the requested information. If the compilation failed when configured with --enable-debug, please share the logs. the name of the WRF subroutine suggests the crash might occur in MPI_Comm_split(), if so, are you able to craft a reproducer that causes the crash? How many nodes and MPI tasks are needed in order to evidence the crash? Cheers, Gilles On Wed, Jan 31, 2024 at 10:09 PM afernandez via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: Hello Joseph, Sorry for the delay but I didn't know if I was missing something yesterday evening and wanted to double check everything this morning. This is for WRF but other apps exhibit the same behavior. * I had no problem with the serial version (and gdb obviously didn't report any issue). * I tried compiling with the --enable-debug flag but it was generating errors during the compilation and never completed. * I went back to my standard flags for debugging: -g -fbacktrace -ggdb -fcheck=bounds,do,mem,pointer -ffpe-trap=invalid,zero,overflow. WRF is still crashing with little extra info vs yesterday: Backtrace for this error: #0 0x7f5a4e54451f in ??? at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 #1 0x7f5a4e5a73fe in __GI___libc_free at ./malloc/malloc.c:3368 #2 0x7f5a4c7aa5c3 in ??? #3 0x7f5a4e83b048 in ??? #4 0x7f5a4e7d3ef1 in ??? #5 0x7f5a4e8dab7b in ??? #6 0x8f6bbf in __module_dm_MOD_split_communicator at /home/ubuntu/WRF-4.5.2/frame/module_dm.f90:5734 #7 0x1879ebd in init_modules_ at /home/ubuntu/WRF-4.5.2/share/init_modules.f90:63 #8 0x406fe4 in __module_wrf_top_MOD_wrf_init at ../main/module_wrf_top.f90:130 #9 0x405ff3 in wrf at /home/ubuntu/WRF-4.5.2/main/wrf.f90:22 #10 0x40605c in main at /home/ubuntu/WRF-4.5.2/main/wrf.f90:6 -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 0 on node ip-172-31-31-163 exited on signal 11 (Segmentation fault). -------------------------------------------------------------------------- Any pointers on what might be going on here as this never happened with OMPIv4. Thanks. Joseph Schuchart via users wrote: Hello, This looks like memory corruption. Do you have more details on what your app is doing? I don't see any MPI calls inside the call stack. Could you rebuild Open MPI with debug information enabled (by adding `--enable-debug` to configure)? If this error occurs on singleton runs (1 process) then you can easily attach gdb to it to get a better stack trace. Also, valgrind may help pin down the problem by telling you which memory block is being free'd here. Thanks Joseph On 1/30/24 07:41, afernandez via users wrote: quote class="gmail_quote" type="cite" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hello, I upgraded one of the systems to v5.0.1 and have compiled everything > exactly as dozens of previous times with v4. I wasn't expecting any > issue (and the compilations didn't report anything out of ordinary) > but running several apps has resulted in error messages such as: /Backtrace for this error:/ /#0 0x7f7c9571f51f in ???/ / at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0/ /#1 0x7f7c957823fe in __GI___libc_free/ / at ./malloc/malloc.c:3368/ /#2 0x7f7c93a635c3 in ???/ /#3 0x7f7c95f84048 in ???/ /#4 0x7f7c95f1cef1 in ???/ /#5 0x7f7c95e34b7b in ???/ /#6 0x6e05be in ???/ /#7 0x6e58d7 in ???/ /#8 0x405d2c in ???/ /#9 0x7f7c95706d8f in __libc_start_call_main/ / at ../sysdeps/nptl/libc_start_call_main.h:58/ /#10 0x7f7c95706e3f in __libc_start_main_impl/ / at ../csu/libc-start.c:392/ /#11 0x405d64 in ???/ /#12 0xffffffffffffffff in ???/ OS is Ubuntu 22.04, OpenMPI was built with GCC13.2, and before > building OpenMPI, I had previously built the hwloc (2.10.0) library at > /usr/lib/x86_64-linux-gnu. Maybe I'm missing something pretty basic, > but the problem seems to be related to memory allocation. Thanks.
