Re: [OMPI users] OpenMPI v4.0.0 signal 11 (Segmentation fault)
On Wed, 20 Feb 2019 10:46:10 -0500 Adam LeBlanc wrote: > Hello, > > When I do a run with OpenMPI v4.0.0 on Infiniband with this command: > mpirun --mca btl_openib_warn_no_device_params_found 0 --map-by node > --mca orte_base_help_aggregate 0 --mca btl openib,vader,self --mca > pml ob1 --mca btl_openib_allow_ib 1 -np 6 > -hostfile /home/aleblanc/ib-mpi-hosts IMB-MPI1 > > I get this error: ... > # Benchmarking Reduce_scatter ... > 2097152 20 8738.08 9340.50 9147.89 > [pandora:04500] *** Process received signal *** > [pandora:04500] Signal: Segmentation fault (11) This is very likely a bug in IMB not in OpenMPI. It's been discussed on the list before, thread name: MPI_Reduce_Scatter Segmentation Fault with Intel 2019 Update 1 Compilers on OPA-1... You can work around by using an older IMB version (the bug is in the newer/est version). /Peter K ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] Do idle MPI threads consume clock cycles?
I have the following, rather unusual, scenario... I have a program running with OpenMP on a multicore computer. At one point in the program, I want to use an external package that is written to exploit MPI, not OpenMP, parallelism. So a (rather awkward) solution could be to launch the program in MPI, but most of the time, everything is being done in a single MPI process, which is using OpenMP (ie, run my current program in a single MPI process). Then, when I get to the part where I need to use the external package, distribute out the information to all the MPI processes, run it across all, and then pull them back to the master process. This is awkward, but probably better than my current approach, which is running the external package on a single processor (ie, not exploiting parallelism in this time-consuming part of the code). If I use this strategy, I fear that the idle MPI processes may be consuming clock cycles while I am running the rest of the program on the master process with OpenMP. Thus, they may compete with the OpenMP threads. OpenMP does not close threads between every pragma, but OMP_WAIT_POLICY can be set to sleep idle threads (actually, this is the default behavior). I have not been able to find any equivalent documentation regarding the behavior of idle threads in MPI. Best regards, Mark ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3
> On Feb 20, 2019, at 7:14 PM, Gilles Gouaillardet wrote: > > Ryan, > > as Edgar explained, that could be a compiler issue (fwiw, I am unable to > reproduce the bug) Same thing, OpenMPI 3.1.3, GCC 4.8.5, and HDF5 1.10.4 make check? Just making sure — that makes it seem like there’s something else going on for me here. Just for comparison’s sake: [novosirj@amarel-test2 testpar]$ rpm -qa gcc gcc-4.8.5-28.el7_5.1.x86_64 > You can build Open MPI again and pass --disable-builtin-atomics to the > configure command line. Thanks, I’ll look into that (din’t know the implications). > That being said, the "Alarm clock" message looks a bit suspicious. > > Does it always occur at 20+ minutes elapsed ? Aha! Yes, you are right: every time it fails, it’s at the 20 minute and a couple of seconds mark. For comparison, every time it runs, it runs for 2-3 seconds total. So it seems like what might actually be happening here is a hang, and not a failure of the test per se. > Is there some mechanism that automatically kills a job if it does not write > anything to stdout for some time ? > > A quick way to rule that out is to > > srun -- mpi=pmi2 -p main -t 1:00:00 -n6 -N1 sleep 1800 > > and see if that completes or get killed with the same error message. I was not aware of anything like that, but I’ll look into it now (running your suggestion). I guess we don’t run across this sort of thing very often — most stuff at least prints output when it starts. > You can also run use mpirun instead of srun, and even run mpirun outside of > slurm > > (if your cluster policy allows it, you can for example use mpirun and run on > the frontend node) I’m on the team that manages the cluster, so we can try various things. Every piece of software we ever run, though, runs via srun — we don’t provide mpirun as a matter of course, except in some corner cases. > On 2/21/2019 3:01 AM, Ryan Novosielski wrote: >> Does it make any sense that it seems to work fine when OpenMPI and HDF5 are >> built with GCC 7.4 and GCC 8.2, but /not/ when they are built with >> RHEL-supplied GCC 4.8.5? That appears to be the scenario. For the GCC 4.8.5 >> build, I did try an XFS filesystem and it didn’t help. GPFS works fine for >> either of the 7.4 and 8.2 builds. >> >> Just as a reminder, since it was reasonably far back in the thread, what I’m >> doing is running the “make check” tests in HDF5 1.10.4, in part because >> users use it, but also because it seems to have a good test suite and I can >> therefore verify the compiler and MPI stack installs. I get very little >> information, apart from it not working and getting that “Alarm clock” >> message. >> >> I originally suspected I’d somehow built some component of this with a >> host-specific optimization that wasn’t working on some compute nodes. But I >> controlled for that and it didn’t seem to make any difference. >> >> -- >> >> || \\UTGERS, >> |---*O*--- >> ||_// the State | Ryan Novosielski - novos...@rutgers.edu >> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus >> || \\of NJ | Office of Advanced Research Computing - MSB C630, >> Newark >> `' >> >>> On Feb 18, 2019, at 1:34 PM, Ryan Novosielski wrote: >>> >>> It didn’t work any better with XFS, as it happens. Must be something else. >>> I’m going to test some more and see if I can narrow it down any, as it >>> seems to me that it did work with a different compiler. >>> >>> -- >>> >>> || \\UTGERS, >>> |---*O*--- >>> ||_// the State | Ryan Novosielski - novos...@rutgers.edu >>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus >>> || \\of NJ | Office of Advanced Research Computing - MSB C630, >>> Newark >>> `' >>> On Feb 18, 2019, at 12:23 PM, Gabriel, Edgar wrote: While I was working on something else, I let the tests run with Open MPI master (which is for parallel I/O equivalent to the upcoming v4.0.1 release), and here is what I found for the HDF5 1.10.4 tests on my local desktop: In the testpar directory, there is in fact one test that fails for both ompio and romio321 in exactly the same manner. I used 6 processes as you did (although I used mpirun directly instead of srun...) From the 13 tests in the testpar directory, 12 pass correctly (t_bigio, t_cache, t_cache_image, testphdf5, t_filters_parallel, t_init_term, t_mpi, t_pflush2, t_pread, t_prestart, t_pshutdown, t_shapesame). The one tests that officially fails ( t_pflush1) actually reports that it passed, but then throws message that indicates that MPI_Abort has been called, for both ompio and romio. I will try to investigate this test to see what is going on. That being said, your repor
Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3
> -Original Message- > > Does it always occur at 20+ minutes elapsed ? > > Aha! Yes, you are right: every time it fails, it’s at the 20 minute and a > couple > of seconds mark. For comparison, every time it runs, it runs for 2-3 seconds > total. So it seems like what might actually be happening here is a hang, and > not a failure of the test per se. > I *think* I can confirm that. I compiled 3.1.3 yesterday with gcc 4.8 (although this was OpenSuSE, not Redhat), and it looked to me like one of tests were hanging, but I didn't have time to investigate it further. Thanks Edgar > > Is there some mechanism that automatically kills a job if it does not write > anything to stdout for some time ? > > > > A quick way to rule that out is to > > > > srun -- mpi=pmi2 -p main -t 1:00:00 -n6 -N1 sleep 1800 > > > > and see if that completes or get killed with the same error message. > > I was not aware of anything like that, but I’ll look into it now (running your > suggestion). I guess we don’t run across this sort of thing very often — most > stuff at least prints output when it starts. > > > You can also run use mpirun instead of srun, and even run mpirun > > outside of slurm > > > > (if your cluster policy allows it, you can for example use mpirun and > > run on the frontend node) > > I’m on the team that manages the cluster, so we can try various things. Every > piece of software we ever run, though, runs via srun — we don’t provide > mpirun as a matter of course, except in some corner cases. > > > On 2/21/2019 3:01 AM, Ryan Novosielski wrote: > >> Does it make any sense that it seems to work fine when OpenMPI and > HDF5 are built with GCC 7.4 and GCC 8.2, but /not/ when they are built with > RHEL-supplied GCC 4.8.5? That appears to be the scenario. For the GCC 4.8.5 > build, I did try an XFS filesystem and it didn’t help. GPFS works fine for > either > of the 7.4 and 8.2 builds. > >> > >> Just as a reminder, since it was reasonably far back in the thread, what > I’m doing is running the “make check” tests in HDF5 1.10.4, in part because > users use it, but also because it seems to have a good test suite and I can > therefore verify the compiler and MPI stack installs. I get very little > information, apart from it not working and getting that “Alarm clock” > message. > >> > >> I originally suspected I’d somehow built some component of this with a > host-specific optimization that wasn’t working on some compute nodes. But I > controlled for that and it didn’t seem to make any difference. > >> > >> -- > >> > >> || \\UTGERS, > >> |---*O*--- > >> ||_// the State | Ryan Novosielski - novos...@rutgers.edu > >> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS > Campus > >> || \\of NJ | Office of Advanced Research Computing - MSB C630, > Newark > >> `' > >> > >>> On Feb 18, 2019, at 1:34 PM, Ryan Novosielski > wrote: > >>> > >>> It didn’t work any better with XFS, as it happens. Must be something > else. I’m going to test some more and see if I can narrow it down any, as it > seems to me that it did work with a different compiler. > >>> > >>> -- > >>> > >>> || \\UTGERS, > >>> |---*O*--- > >>> ||_// the State| Ryan Novosielski - novos...@rutgers.edu > >>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS > Campus > >>> || \\of NJ| Office of Advanced Research Computing - MSB > C630, Newark > >>> `' > >>> > On Feb 18, 2019, at 12:23 PM, Gabriel, Edgar > wrote: > > While I was working on something else, I let the tests run with Open > MPI master (which is for parallel I/O equivalent to the upcoming v4.0.1 > release), and here is what I found for the HDF5 1.10.4 tests on my local > desktop: > > In the testpar directory, there is in fact one test that fails for both > ompio and romio321 in exactly the same manner. > I used 6 processes as you did (although I used mpirun directly instead > of srun...) From the 13 tests in the testpar directory, 12 pass correctly > (t_bigio, t_cache, t_cache_image, testphdf5, t_filters_parallel, t_init_term, > t_mpi, t_pflush2, t_pread, t_prestart, t_pshutdown, t_shapesame). > > The one tests that officially fails ( t_pflush1) actually reports that it > passed, but then throws message that indicates that MPI_Abort has been > called, for both ompio and romio. I will try to investigate this test to see > what > is going on. > > That being said, your report shows an issue in t_mpi, which passes > without problems for me. This is however not GPFS, this was an XFS local file > system. Running the tests on GPFS are on my todo list as well. > > Thanks > Edgar > > > > > -Original Message- > > From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of > > Gabriel, Edgar
Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3
> On Feb 21, 2019, at 2:52 PM, Gabriel, Edgar wrote: > >> -Original Message- >>> Does it always occur at 20+ minutes elapsed ? >> >> Aha! Yes, you are right: every time it fails, it’s at the 20 minute and a >> couple >> of seconds mark. For comparison, every time it runs, it runs for 2-3 seconds >> total. So it seems like what might actually be happening here is a hang, and >> not a failure of the test per se. >> > > I *think* I can confirm that. I compiled 3.1.3 yesterday with gcc 4.8 > (although this was OpenSuSE, not Redhat), and it looked to me like one of > tests were hanging, but I didn't have time to investigate it further. Just to be clear, the hanging test I have is t_mpi from HDF5 1.10.4. The OpenMPI 3.1.3 make check passes just fine on all of our builds. But I don’t believe it ever launches any jobs or anything like that. -- || \\UTGERS, |---*O*--- ||_// the State | Ryan Novosielski - novos...@rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\of NJ | Office of Advanced Research Computing - MSB C630, Newark `' signature.asc Description: Message signed with OpenPGP ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3
Yes, I was talking about the same thing, although for me it was not t_mpi, but t_shapesame that was hanging. It might be an indication of the same issue however. > -Original Message- > From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Ryan > Novosielski > Sent: Thursday, February 21, 2019 1:59 PM > To: Open MPI Users > Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI > 3.1.3 > > > > On Feb 21, 2019, at 2:52 PM, Gabriel, Edgar > wrote: > > > >> -Original Message- > >>> Does it always occur at 20+ minutes elapsed ? > >> > >> Aha! Yes, you are right: every time it fails, it’s at the 20 minute > >> and a couple of seconds mark. For comparison, every time it runs, it > >> runs for 2-3 seconds total. So it seems like what might actually be > >> happening here is a hang, and not a failure of the test per se. > >> > > > > I *think* I can confirm that. I compiled 3.1.3 yesterday with gcc 4.8 > (although this was OpenSuSE, not Redhat), and it looked to me like one of > tests were hanging, but I didn't have time to investigate it further. > > Just to be clear, the hanging test I have is t_mpi from HDF5 1.10.4. The > OpenMPI 3.1.3 make check passes just fine on all of our builds. But I don’t > believe it ever launches any jobs or anything like that. > > -- > > || \\UTGERS, > |---*O*--- > ||_// the State| Ryan Novosielski - novos...@rutgers.edu > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus > || \\of NJ| Office of Advanced Research Computing - MSB C630, > Newark > `' ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3
Related to this or not, I also get a hang on MVAPICH2 2.3 compiled with GCC 8.2, but on t_filters_parallel, not t_mpi. With that combo, though, I get a segfault, or at least a message about one. It’s only “Alarm clock” on the GCC 4.8 with OpenMPI 3.1.3 combo. It also happens at the ~20 minute mark, FWIW. Testing t_filters_parallel t_filters_parallel Test Log srun: job 84117363 queued and waiting for resources srun: job 84117363 has been allocated resources [slepner063.amarel.rutgers.edu:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11) srun: error: slepner063: task 0: Segmentation fault srun: error: slepner063: tasks 1-3: Alarm clock 0.01user 0.01system 20:01.44elapsed 0%CPU (0avgtext+0avgdata 5144maxresident)k 0inputs+0outputs (0major+1524minor)pagefaults 0swaps make[4]: *** [t_filters_parallel.chkexe_] Error 1 make[4]: Leaving directory `/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-mvapich2-2.3/testpar' make[3]: *** [build-check-p] Error 1 make[3]: Leaving directory `/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-mvapich2-2.3/testpar' make[2]: *** [test] Error 2 make[2]: Leaving directory `/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-mvapich2-2.3/testpar' make[1]: *** [check-am] Error 2 make[1]: Leaving directory `/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-mvapich2-2.3/testpar' make: *** [check-recursive] Error 1 > On Feb 21, 2019, at 3:03 PM, Gabriel, Edgar wrote: > > Yes, I was talking about the same thing, although for me it was not t_mpi, > but t_shapesame that was hanging. It might be an indication of the same issue > however. > >> -Original Message- >> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Ryan >> Novosielski >> Sent: Thursday, February 21, 2019 1:59 PM >> To: Open MPI Users >> Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI >> 3.1.3 >> >> >>> On Feb 21, 2019, at 2:52 PM, Gabriel, Edgar >> wrote: >>> -Original Message- > Does it always occur at 20+ minutes elapsed ? Aha! Yes, you are right: every time it fails, it’s at the 20 minute and a couple of seconds mark. For comparison, every time it runs, it runs for 2-3 seconds total. So it seems like what might actually be happening here is a hang, and not a failure of the test per se. >>> >>> I *think* I can confirm that. I compiled 3.1.3 yesterday with gcc 4.8 >> (although this was OpenSuSE, not Redhat), and it looked to me like one of >> tests were hanging, but I didn't have time to investigate it further. >> >> Just to be clear, the hanging test I have is t_mpi from HDF5 1.10.4. The >> OpenMPI 3.1.3 make check passes just fine on all of our builds. But I don’t >> believe it ever launches any jobs or anything like that. >> >> -- >> >> || \\UTGERS, >> |---*O*--- >> ||_// the State | Ryan Novosielski - novos...@rutgers.edu >> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus >> || \\of NJ | Office of Advanced Research Computing - MSB C630, >> Newark >> `' > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users signature.asc Description: Message signed with OpenPGP ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3
> On Feb 20, 2019, at 7:14 PM, Gilles Gouaillardet wrote: > > Ryan, > > That being said, the "Alarm clock" message looks a bit suspicious. > > Does it always occur at 20+ minutes elapsed ? > > Is there some mechanism that automatically kills a job if it does not write > anything to stdout for some time ? > > A quick way to rule that out is to > > srun -- mpi=pmi2 -p main -t 1:00:00 -n6 -N1 sleep 1800 > > and see if that completes or get killed with the same error message. FWIW, the “sleep” completes just fine: [novosirj@amarel-test2 testpar]$ sacct -j 84173276 -M perceval -o jobid,jobname,start,end,node,state JobIDJobName Start EndNodeList State -- --- --- --- -- 84173276 sleep 2019-02-21T14:46:03 2019-02-21T15:16:03 node077 COMPLETED 84173276.ex+ extern 2019-02-21T14:46:03 2019-02-21T15:16:03 node077 COMPLETED 84173276.0sleep 2019-02-21T14:46:03 2019-02-21T15:16:03 node077 COMPLETED -- || \\UTGERS, |---*O*--- ||_// the State | Ryan Novosielski - novos...@rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\of NJ | Office of Advanced Research Computing - MSB C630, Newark `' signature.asc Description: Message signed with OpenPGP ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
