the reason for potential performance issues on NFS are very different from Lustre. Basically, depending on your use-case and the NFS configuration, you have to enforce different locking policy to ensure correct output files. The default value for chosen for ompio is the most conservative setting, since this was the only setting that we found that would result in a correct output file for all of our tests. You can change settings to see whether other options would work you.
The parameter that you need to work with is fs_ufs_lock_algorithm. Setting it to 1 will completely disable it (and most likely lead to the best performance), setting it to 3 is a middle ground (lock specific ranges) and similar to what ROMIO does. So e.g. mpiexec -n 16 --mca fs_ufs_lock_algorihtm 1 ./mytests That being said, if you google NFS + MPI I/O, you will find a ton of document and reasons for potential problems, so using MPI I/O on top of NFS (whether OMPIO or ROMIO) is always at your own risk. Thanks Edgar -----Original Message----- From: users <users-boun...@lists.open-mpi.org> On Behalf Of Gilles Gouaillardet via users Sent: Thursday, December 3, 2020 4:46 AM To: Open MPI Users <users@lists.open-mpi.org> Cc: Gilles Gouaillardet <gilles.gouaillar...@gmail.com> Subject: Re: [OMPI users] Parallel HDF5 low performance Patrick, glad to hear you will upgrade Open MPI thanks to this workaround! ompio has known performance issues on Lustre (this is why ROMIO is still the default on this filesystem) but I do not remember such performance issues have been reported on a NFS filesystem. Sharing a reproducer will be very much appreciated in order to improve ompio Cheers, Gilles On Thu, Dec 3, 2020 at 6:05 PM Patrick Bégou via users <users@lists.open-mpi.org> wrote: > > Thanks Gilles, > > this is the solution. > I will set OMPI_MCA_io=^ompio automatically when loading the parallel > hdf5 module on the cluster. > > I was tracking this problem for several weeks but not looking in the > right direction (testing NFS server I/O, network bandwidth.....) > > I think we will now move definitively to modern OpenMPI implementations. > > Patrick > > Le 03/12/2020 à 09:06, Gilles Gouaillardet via users a écrit : > > Patrick, > > > > > > In recent Open MPI releases, the default component for MPI-IO is > > ompio (and no more romio) > > > > unless the file is on a Lustre filesystem. > > > > > > You can force romio with > > > > mpirun --mca io ^ompio ... > > > > > > Cheers, > > > > > > Gilles > > > > On 12/3/2020 4:20 PM, Patrick Bégou via users wrote: > >> Hi, > >> > >> I'm using an old (but required by the codes) version of hdf5 > >> (1.8.12) in parallel mode in 2 fortran applications. It relies on > >> MPI/IO. The storage is NFS mounted on the nodes of a small cluster. > >> > >> With OpenMPI 1.7 it runs fine but using modern OpenMPI 3.1 or 4.0.5 > >> the I/Os are 10x to 100x slower. Are there fundamentals changes in > >> MPI/IO for these new releases of OpenMPI and a solution to get back > >> to the IO performances with this parallel HDF5 release ? > >> > >> Thanks for your advices > >> > >> Patrick > >> >
