On Mon, 2018-10-15 at 12:21 +0100, Dave Love wrote: > For what it's worth, I found the following from running ROMIO's tests > with OMPIO on Lustre mounted without flock (or localflock). I used > 48 > processes on two nodes with Lustre for tests which don't require a > specific number. > > OMPIO fails tests atomicity, misc, and error on ext4; it additionally > fails noncontig_coll2, fp, shared_fp, and ordered_fp on > Lustre/noflock. > > On Lustre/noflock, ROMIO fails on atomicity, i_noncontig, noncontig, > shared_fp, ordered_fp, and error. > > Please can OMPIO be changed to fail in the same way as ROMIO (with a > clear message) for the operations it can't support without flock. > Otherwise it looks as if you can potentially get invalid data, or at > least waste time debugging other errors. > > I'd debug the common failure on the "error" test, but ptrace is > disabled > on the system. > > In case anyone else is in the same boat and can't get mounts changed, > I > suggested staging data to and from a PVFS2^WOrangeFS ephemeral > filesystem on jobs' TMPDIR local mounts if they will fit. Of course > other libraries will potentially corrupt data on nolock mounts.
ROMIO uses fcntl locks for Atomic mode, Shared file pointer updates, and to prefent false sharing in the data sieving optimization for noncontiguous writes. it's hard to implement fcntl-lock-free versions of Atomic mode and Shared file pointer so file systems like PVFS don't support those modes (and return an error indicating such at open time). You can run lock-free for noncontiguous writes, though at a significant performance cost. In ROMIO we can disable data sieving write by setting the hint "romio_ds_write" to "disable", which will fall back to piece-wise operations. Could be OK if you know your noncontiguous accesses are only a little bit noncontiguous. Perhaps OMPIO has a similar option, but I am not familiar with its tuning knobs. ==rob _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
