Barry, I configured PETSC with --with-debugging=yes. I think this is enough to block any optimizations, right? I tried both Intel and GNU compilers. The error persists. I tried to change the preconditioner and use PILUT instead of BoomerAMG from Hypre. Still, the error appears. I noticed that I do not have any problem when only one ksp is present. I am afraid that this is the culprit. I will try your suggestion and setup an X Windows DISPLAY variable and send back the results.
Thanks for your time, Pantelis ________________________________ Από: Barry Smith <[email protected]> Στάλθηκε: Πέμπτη, 9 Νοεμβρίου 2023 5:53 μμ Προς: Pantelis Moschopoulos <[email protected]> Κοιν.: [email protected] <[email protected]> Θέμα: Re: [petsc-users] PETSC breaks when using HYPRE preconditioner Pantelis If you can set an X Windows DISPLAY variable that works you can run with -on_error_attach_debugger and gdb should pop up in an Xterm on MPI rank 16 showing the code where it is crashing (based on Valgrind Address 0x0 is not stack'd, malloc'd or (recently) free'd there will be pointer of 0 that should not be). Or if the computer system has some parallel debugger you can use that directly. For lldb use -on_error_attach_debugger lldb If you have some compiler optimizations set when you ./configure PETSc you might try making another PETSC_ARCH without optimizations (this is PETSc's default when you do not use --with-debugging=0). Does it still crash with no optimizations? Perhaps also try with a different compiler? Barry On Nov 9, 2023, at 6:57 AM, Pantelis Moschopoulos <[email protected]> wrote: Hello everyone, I am trying to use Petsc coupled with Hypre BoomerAMG as preconditioner in our in-house code to simulate the transient motion of complex fluid with finite elements. The problem is that after a random number of iterations, an error arises when the Hypre is called. The error that I get in the terminal is the following: [16]PETSC ERROR: ------------------------------------------------------------------------ [16]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [16]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ [16]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [16]PETSC ERROR: The line numbers in the error traceback are not always exact. [16]PETSC ERROR: #1 Hypre solve [16]PETSC ERROR: #2 PCApply_HYPRE() at /home/pmosx/Libraries/petsc/src/ksp/pc/impls/hypre/hypre.c:451 [16]PETSC ERROR: #3 PCApply() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:486 [16]PETSC ERROR: #4 PCApplyBAorAB() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:756 [16]PETSC ERROR: #5 KSP_PCApplyBAorAB() at /home/pmosx/Libraries/petsc/include/petsc/private/kspimpl.h:443 [16]PETSC ERROR: #6 KSPGMRESCycle() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:146 [16]PETSC ERROR: #7 KSPSolve_GMRES() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:227 [16]PETSC ERROR: #8 KSPSolve_Private() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:910 [16]PETSC ERROR: #9 KSPSolve() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:1082 On the same time, I use valgrind and when the program stops, it reports the following: ==1261647== Invalid read of size 8 ==1261647== at 0x4841C74: _intel_fast_memcpy (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so<http://vgpreload_memcheck-amd64-linux.so/>) ==1261647== by 0x16231F73: hypre_GaussElimSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so<http://libhypre-2.29.0.so/>) ==1261647== by 0x1622DB4F: hypre_BoomerAMGCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so<http://libhypre-2.29.0.so/>) ==1261647== by 0x1620002E: hypre_BoomerAMGSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so<http://libhypre-2.29.0.so/>) ==1261647== by 0x12B6F8F8: PCApply_HYPRE (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x12C38785: PCApply (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x12C36A39: PCApplyBAorAB (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x126299E1: KSPGMRESCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x12628051: KSPSolve_GMRES (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x127A532E: KSPSolve_Private (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x127A3C8A: KSPSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== by 0x12C50AF1: kspsolve_ (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so<http://libpetsc.so/>.3.20.1) ==1261647== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==1261647== This is indeed a very peculiar error. I cannot understand why it happens. In our solution procedure, we split the equations and we solve them segregated. I create two different ksp (1_ksp and 2_ksp) using KSPSetOptionsPrefix. Might this choice create a confusion and results in this error? Any help is much appreciated. Pantelis
