I opened an issue on this, hope to have the fix available next week.
https://github.com/open-mpi/ompi/issues/4334 Thanks Edgar On 10/12/2017 8:36 PM, Edgar Gabriel wrote:
try for now to switch to the romio314 component with OpenMPI. There is an issue with NFS and OMPIO that I am aware of and working on, that might trigger this behavior (although it should actually work for collective I/O even in that case). try to set something like mpirun --mca io romio314 ... Thanks Edgar On 10/12/2017 8:26 PM, Stephen Guzik wrote:Hi, I'm having trouble with parallel I/O to a file system mounted with NFS over an infiniband network. In my test code, I'm simply writing 1 byte per process to the same file. When using two nodes, some bytes are not written (zero bits in the unwritten bytes). Usually at least some data from each node is written---it appears to be all data from one node and partial from the other. This used to work fine but broke when the cluster was upgraded from Debian 8 to Debian 9. I suspect an issue with NFS and not with OpenMPI. However, if anyone can suggest a work-around or ways to get more information, I would appreciate it. In the sole case where the file system is exported with 'sync' and mounted with 'hard,intr', I get the error: [node1:14823] mca_sharedfp_individual_file_open: Error during datafile file open MPI_ERR_FILE: invalid file [node2:14593] (same) ---------- Some additional info: - tested versions 1.8.8, 2.1.1, and 3.0.0 self-compiled and packaged and vendor-supplied versions. All have same behavior. - all write methods (individual or collective) fail similarly. - exporting the file system to two workstations across ethernet and running the job across the two workstations seems to work fine. - on a single node, everything works as expected in all cases. In the case described above where I get an error, the error is only observed with processes on two nodes. - code follows. Thanks, Stephen Guzik ---------- #include <iostream> #include <mpi.h> int main(int argc, const char* argv[]) { MPI_File fh; MPI_Status status; int mpierr; char mpistr[MPI_MAX_ERROR_STRING]; int mpilen; int numProc; int procID; MPI_Init(&argc, const_cast<char***>(&argv)); MPI_Comm_size(MPI_COMM_WORLD, &numProc); MPI_Comm_rank(MPI_COMM_WORLD, &procID); const int filesize = numProc; const int bufsize = filesize/numProc; char *buf = new char[bufsize]; buf[0] = (char)(48 + procID); int numChars = bufsize/sizeof(char); mpierr = MPI_File_open(MPI_COMM_WORLD, "dataio", MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &fh); if (mpierr != MPI_SUCCESS) { MPI_Error_string(mpierr, mpistr, &mpilen); std::cout << "Error: " << mpistr << std::endl; } mpierr = MPI_File_write_at_all(fh, (MPI_Offset)(procID*bufsize), buf, numChars, MPI_CHAR, &status); if (mpierr != MPI_SUCCESS) { MPI_Error_string(mpierr, mpistr, &mpilen); std::cout << "Error: " << mpistr << std::endl; } mpierr = MPI_File_close(&fh); if (mpierr != MPI_SUCCESS) { MPI_Error_string(mpierr, mpistr, &mpilen); std::cout << "Error: " << mpistr << std::endl; } delete[] buf; MPI_Finalize(); return 0; } _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
