Hi Mark,"[...] MPI makes neither requirements nor recommendations for the output [...]" (MPI4.0, §2.9.1)
From my experience, an application can avoid such scrambling (still no guarantee), if the output of lines is written atomically. C++ streams are worst for concurrent output, as every stream operator writes a chunk. It can help to collect output into a stringstream and print out at once. Using printf in C is typically least problematic. Flushing the buffer (fflush) helpts to avoid that the output buffer fills up and is flushed in the middle of printing.
I'm not the Fortran expert. But, I think there are some options to change to a buffered output mode (at least I found such options for file I/O). Again, the goal should be that a write statement is printed at once and the buffer doesn't fill up while printing.
In any case, it could help to write warnings to stderr and separate the stdout and stderr streams.
Best Joachim Am 02.12.21 um 16:48 schrieb Fisher (US), Mark S via users:
We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having issues with lines scrambling together occasionally. This causes issues our convergence checking code since we put convergence data there. We are not using any mpirun options for stdout we just redirect stdout/stderr to a file before we run the mpirun command so all output goes there. We had similar issue with Intel MPI in the past and used the -ordered-output to fix it but I do not see any similar option for OpenMPI. See example below. Is there anyway to ensure a line from a process gets one line in the output file?*The data in red below is scrambled up and should look like the cleaned-up version. You can see it put a line from a different process inside a line from another processes and the rest of the line ended up a couple of lines down.*ZONE 0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 1.004E-01 sec*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 aerosurfs*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 aerosurfs*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 Aircraft-Total*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 699 1625 12Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 111 1626 6 aft-Total*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 -4.961E-06 Aircraft-OML*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 -2.791E-05 Aircraft-OML*Cleaned up version:*ZONE 0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 1.004E-01 sec*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 aerosurfs*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 aerosurfs*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 Aircraft-Total*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 Aircraft-TotalWarning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 699 1625 12 Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 111 1626 6*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 -4.961E-06 Aircraft-OML*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 -2.791E-05 Aircraft-OMLThanks!
-- Dr. rer. nat. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 pro...@itc.rwth-aachen.de www.itc.rwth-aachen.de
smime.p7s
Description: S/MIME Cryptographic Signature
