Re: [OMPI users] stdout scrambled in file

[Fisher (US), Mark S via users]

So we do that for residuals/loads since we only print one residual output or 
group loads total. The warning messages are send from the process that had the 
issue and that is what is interrupting the output of loads data.


From: users  On Behalf Of Ralph Castain via 
users
Sent: Sunday, December 5, 2021 2:48 PM
To: Open MPI Users 
Cc: Ralph Castain 
Subject: [EXTERNAL] Re: [OMPI users] stdout scrambled in file


EXT email: be mindful of links/attachments.




There are several output-controlling options - e.g., you could redirect the 
output from each process to its own file or directory.

However, it makes little sense to me for someone to write convergence data into 
a file and then parse it. Typically, convergence data results from all procs 
reaching the end of a computational epoch or cycle - i.e., you need all the 
procs to reach the same point. So why not just have the procs report their 
convergence data to rank=0 using an MPI_Gather collective, and then have that 
proc output whatever info you want to see?

You would then no longer be dependent on some implementation-specific "mpirun" 
cmd line option, so you could run the same code using srun, aprun, prun, or 
mpirun and get the exact same output.

Am I missing something?
Ralph



On Dec 5, 2021, at 12:19 PM, Gus Correa via users 
mailto:users@lists.open-mpi.org>> wrote:

Hi Mark

Back in the days I liked the mpirun/mpiexec --tag-output option.
Jeff: Does it still exist?
It may not prevent 100% the splitting of output lines,
but tagging the lines with the process rank helps.
You can grep the stdout log for the rank that you want,
which helps a lot when several processes are talking.

I hope this helps,
Gus Correa


On Sun, Dec 5, 2021 at 1:12 PM Jeff Squyres (jsquyres) via users 
mailto:users@lists.open-mpi.org>> wrote:
FWIW: Open MPI 4.1.2 has been released -- you can probably stop using an RC 
release.

I think you're probably running into an issue that is just a fact of life.  
Especially when there's a lot of output simultaneously from multiple MPI 
processes (potentially on different nodes), the stdout/stderr lines can just 
get munged together.

Can you check for convergence a different way?

--
Jeff Squyres
jsquy...@cisco.com


From: users 
mailto:users-boun...@lists.open-mpi.org>> on 
behalf of Fisher (US), Mark S via users 
mailto:users@lists.open-mpi.org>>
Sent: Thursday, December 2, 2021 10:48 AM
To: users@lists.open-mpi.org
Cc: Fisher (US), Mark S
Subject: [OMPI users] stdout scrambled in file

We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having issues 
with lines scrambling together occasionally. This causes issues our convergence 
checking code since we put convergence data there. We are not using any mpirun 
options for stdout we just redirect stdout/stderr to a file before we run the 
mpirun command so all output goes there. We had similar issue with Intel MPI in 
the past and used the -ordered-output to fix it but I do not see any similar 
option for OpenMPI. See example below. Is there anyway to ensure a line from a 
process gets one line in the output file?


The data in red below is scrambled up and should look like the cleaned-up 
version. You can see it put a line from a different process inside a line from 
another processes and the rest of the line ended up a couple of lines down.

ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 1.004E-01 sec

*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06  
aerosurfs
*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05  
aerosurfs
*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06  
Aircraft-Total
*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05  
Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  1625 12
Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6
aft-Total
*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 -4.961E-06  
Aircraft-OML
*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 -2.791E-05 
 Aircraft-OML


Cleaned up version:

ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 1.004E-01 sec

*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06  
aerosurfs
*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05  
aerosurfs
*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06  
Aircraft-Total
*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05  
Aircraft-Total
 Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  1625 12
Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6
*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 -4.961E-06  
Aircraft-OML
*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 -2.791E-05 

Re: [OMPI users] stdout scrambled in file

[Joachim Protze via users]

Hi Mark,

"[...] MPI makes neither requirements nor recommendations for the output 
[...]" (MPI4.0, §2.9.1)


From my experience, an application can avoid such scrambling (still no 
guarantee), if the output of lines is written atomically. C++ streams 
are worst for concurrent output, as every stream operator writes a 
chunk. It can help to collect output into a stringstream and print out 
at once. Using printf in C is typically least problematic. Flushing the 
buffer (fflush) helpts to avoid that the output buffer fills up and is 
flushed in the middle of printing.


I'm not the Fortran expert. But, I think there are some options to 
change to a buffered output mode (at least I found such options for file 
I/O). Again, the goal should be that a write statement is printed at 
once and the buffer doesn't fill up while printing.


In any case, it could help to write warnings to stderr and separate the 
stdout and stderr streams.


Best
Joachim

Am 02.12.21 um 16:48 schrieb Fisher (US), Mark S via users:
We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having 
issues with lines scrambling together occasionally. This causes issues 
our convergence checking code since we put convergence data there. We 
are not using any mpirun options for stdout we just redirect 
stdout/stderr to a file before we run the mpirun command so all output 
goes there. We had similar issue with Intel MPI in the past and used the 
-ordered-output to fix it but I do not see any similar option for 
OpenMPI. See example below. Is there anyway to ensure a line from a 
process gets one line in the output file?


*The data in red below is scrambled up and should look like the 
cleaned-up version. You can see it put a line from a different process 
inside a line from another processes and the rest of the line ended up a 
couple of lines down.*


ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 
1.004E-01 sec


*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
-4.945E-06  aerosurfs


*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
-2.785E-05  aerosurfs


*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
-4.945E-06  Aircraft-Total


*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
-2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  
1625 12


Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6

aft-Total

*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 
-4.961E-06  Aircraft-OML


*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 
-2.791E-05  Aircraft-OML


*Cleaned up version:*

ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 
1.004E-01 sec


*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
-4.945E-06  aerosurfs


*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
-2.785E-05  aerosurfs


*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
-4.945E-06  Aircraft-Total


*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
-2.785E-05 Aircraft-Total


  Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  1625 12

Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6

*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 
-4.961E-06  Aircraft-OML


*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 
-2.791E-05  Aircraft-OML


Thanks!




--
Dr. rer. nat. Joachim Protze

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
pro...@itc.rwth-aachen.de
www.itc.rwth-aachen.de


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [OMPI users] stdout scrambled in file

[Fisher (US), Mark S via users]

All strings are writing as one output so that is not the issue. Adding in some 
flushing is a good idea and we can try that. We do not open stdout just write 
to unit 6, but we could open it if there is some un-buffered option that could 
help. I will look into that also.  Thanks!

-Original Message-
From: Joachim Protze  
Sent: Monday, December 6, 2021 9:24 AM
To: Open MPI Users 
Cc: Fisher (US), Mark S 
Subject: Re: [OMPI users] stdout scrambled in file

Hi Mark,

"[...] MPI makes neither requirements nor recommendations for the output 
[...]" (MPI4.0, §2.9.1)

 From my experience, an application can avoid such scrambling (still no 
guarantee), if the output of lines is written atomically. C++ streams 
are worst for concurrent output, as every stream operator writes a 
chunk. It can help to collect output into a stringstream and print out 
at once. Using printf in C is typically least problematic. Flushing the 
buffer (fflush) helpts to avoid that the output buffer fills up and is 
flushed in the middle of printing.

I'm not the Fortran expert. But, I think there are some options to 
change to a buffered output mode (at least I found such options for file 
I/O). Again, the goal should be that a write statement is printed at 
once and the buffer doesn't fill up while printing.

In any case, it could help to write warnings to stderr and separate the 
stdout and stderr streams.

Best
Joachim

Am 02.12.21 um 16:48 schrieb Fisher (US), Mark S via users:
> We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having 
> issues with lines scrambling together occasionally. This causes issues 
> our convergence checking code since we put convergence data there. We 
> are not using any mpirun options for stdout we just redirect 
> stdout/stderr to a file before we run the mpirun command so all output 
> goes there. We had similar issue with Intel MPI in the past and used the 
> -ordered-output to fix it but I do not see any similar option for 
> OpenMPI. See example below. Is there anyway to ensure a line from a 
> process gets one line in the output file?
> 
> *The data in red below is scrambled up and should look like the 
> cleaned-up version. You can see it put a line from a different process 
> inside a line from another processes and the rest of the line ended up a 
> couple of lines down.*
> 
> ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 
> 1.004E-01 sec
> 
> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
> -4.945E-06  aerosurfs
> 
> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
> -2.785E-05  aerosurfs
> 
> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
> -4.945E-06  Aircraft-Total
> 
> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
> -2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  
> 1625 12
> 
> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6
> 
> aft-Total
> 
> *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 
> -4.961E-06  Aircraft-OML
> 
> *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 
> -2.791E-05  Aircraft-OML
> 
> *Cleaned up version:*
> 
> ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 
> 1.004E-01 sec
> 
> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
> -4.945E-06  aerosurfs
> 
> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
> -2.785E-05  aerosurfs
> 
> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 
> -4.945E-06  Aircraft-Total
> 
> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 
> -2.785E-05 Aircraft-Total
> 
>   Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  1625 12
> 
> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6
> 
> *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 
> -4.961E-06  Aircraft-OML
> 
> *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 
> -2.791E-05  Aircraft-OML
> 
> Thanks!
> 


-- 
Dr. rer. nat. Joachim Protze

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
pro...@itc.rwth-aachen.de
www.itc.rwth-aachen.de

Re: [OMPI users] stdout scrambled in file

[Joachim Protze via users]

I would assume, that the buffering mode is compiler/runtime specific. At 
least for Intel compiler, the default seems to be/have been unbuffered 
for stdout, but there is a flag for buffered output:


https://community.intel.com/t5/Intel-Fortran-Compiler/Enabling-buffered-I-O-to-stdout-with-Intel-ifort-compiler/td-p/993203

In the worst case, each character might be written individually. If the 
scrambling only happens from time to time, I guess you really just see 
the buffer flush when the buffer filled up.


- Joachim

Am 06.12.21 um 16:42 schrieb Fisher (US), Mark S:

All strings are writing as one output so that is not the issue. Adding in some 
flushing is a good idea and we can try that. We do not open stdout just write 
to unit 6, but we could open it if there is some un-buffered option that could 
help. I will look into that also.  Thanks!

-Original Message-
From: Joachim Protze 
Sent: Monday, December 6, 2021 9:24 AM
To: Open MPI Users 
Cc: Fisher (US), Mark S 
Subject: Re: [OMPI users] stdout scrambled in file

Hi Mark,

"[...] MPI makes neither requirements nor recommendations for the output
[...]" (MPI4.0, §2.9.1)

  From my experience, an application can avoid such scrambling (still no
guarantee), if the output of lines is written atomically. C++ streams
are worst for concurrent output, as every stream operator writes a
chunk. It can help to collect output into a stringstream and print out
at once. Using printf in C is typically least problematic. Flushing the
buffer (fflush) helpts to avoid that the output buffer fills up and is
flushed in the middle of printing.

I'm not the Fortran expert. But, I think there are some options to
change to a buffered output mode (at least I found such options for file
I/O). Again, the goal should be that a write statement is printed at
once and the buffer doesn't fill up while printing.

In any case, it could help to write warnings to stderr and separate the
stdout and stderr streams.

Best
Joachim

Am 02.12.21 um 16:48 schrieb Fisher (US), Mark S via users:

We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having
issues with lines scrambling together occasionally. This causes issues
our convergence checking code since we put convergence data there. We
are not using any mpirun options for stdout we just redirect
stdout/stderr to a file before we run the mpirun command so all output
goes there. We had similar issue with Intel MPI in the past and used the
-ordered-output to fix it but I do not see any similar option for
OpenMPI. See example below. Is there anyway to ensure a line from a
process gets one line in the output file?

*The data in red below is scrambled up and should look like the
cleaned-up version. You can see it put a line from a different process
inside a line from another processes and the rest of the line ended up a
couple of lines down.*

ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10
1.004E-01 sec

*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
-4.945E-06  aerosurfs

*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
-2.785E-05  aerosurfs

*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
-4.945E-06  Aircraft-Total

*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
-2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699
1625 12

Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6

aft-Total

*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04
-4.961E-06  Aircraft-OML

*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04
-2.791E-05  Aircraft-OML

*Cleaned up version:*

ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10
1.004E-01 sec

*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
-4.945E-06  aerosurfs

*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
-2.785E-05  aerosurfs

*IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
-4.945E-06  Aircraft-Total

*IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
-2.785E-05 Aircraft-Total

   Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  1625 12

Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6

*IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04
-4.961E-06  Aircraft-OML

*IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04
-2.791E-05  Aircraft-OML

Thanks!







--
Dr. rer. nat. Joachim Protze

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
pro...@itc.rwth-aachen.de
www.itc.rwth-aachen.de


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [OMPI users] stdout scrambled in file

[Fisher (US), Mark S via users]

This usually happens if we get a number of warning message from multiple 
processes. Seems like unbuffered is what we want but not sure how this 
interacts with MPI since stdout/stderr is pulled back from different hosts. Not 
sure how you are doing that. 

-Original Message-
From: Joachim Protze  
Sent: Monday, December 06, 2021 11:12 AM
To: Fisher (US), Mark S ; Open MPI Users 

Subject: Re: [OMPI users] stdout scrambled in file

I would assume, that the buffering mode is compiler/runtime specific. At 
least for Intel compiler, the default seems to be/have been unbuffered 
for stdout, but there is a flag for buffered output:

https://community.intel.com/t5/Intel-Fortran-Compiler/Enabling-buffered-I-O-to-stdout-with-Intel-ifort-compiler/td-p/993203

In the worst case, each character might be written individually. If the 
scrambling only happens from time to time, I guess you really just see 
the buffer flush when the buffer filled up.

- Joachim

Am 06.12.21 um 16:42 schrieb Fisher (US), Mark S:
> All strings are writing as one output so that is not the issue. Adding in 
> some flushing is a good idea and we can try that. We do not open stdout just 
> write to unit 6, but we could open it if there is some un-buffered option 
> that could help. I will look into that also.  Thanks!
> 
> -Original Message-
> From: Joachim Protze 
> Sent: Monday, December 6, 2021 9:24 AM
> To: Open MPI Users 
> Cc: Fisher (US), Mark S 
> Subject: Re: [OMPI users] stdout scrambled in file
> 
> Hi Mark,
> 
> "[...] MPI makes neither requirements nor recommendations for the output
> [...]" (MPI4.0, §2.9.1)
> 
>   From my experience, an application can avoid such scrambling (still no
> guarantee), if the output of lines is written atomically. C++ streams
> are worst for concurrent output, as every stream operator writes a
> chunk. It can help to collect output into a stringstream and print out
> at once. Using printf in C is typically least problematic. Flushing the
> buffer (fflush) helpts to avoid that the output buffer fills up and is
> flushed in the middle of printing.
> 
> I'm not the Fortran expert. But, I think there are some options to
> change to a buffered output mode (at least I found such options for file
> I/O). Again, the goal should be that a write statement is printed at
> once and the buffer doesn't fill up while printing.
> 
> In any case, it could help to write warnings to stderr and separate the
> stdout and stderr streams.
> 
> Best
> Joachim
> 
> Am 02.12.21 um 16:48 schrieb Fisher (US), Mark S via users:
>> We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having
>> issues with lines scrambling together occasionally. This causes issues
>> our convergence checking code since we put convergence data there. We
>> are not using any mpirun options for stdout we just redirect
>> stdout/stderr to a file before we run the mpirun command so all output
>> goes there. We had similar issue with Intel MPI in the past and used the
>> -ordered-output to fix it but I do not see any similar option for
>> OpenMPI. See example below. Is there anyway to ensure a line from a
>> process gets one line in the output file?
>>
>> *The data in red below is scrambled up and should look like the
>> cleaned-up version. You can see it put a line from a different process
>> inside a line from another processes and the rest of the line ended up a
>> couple of lines down.*
>>
>> ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10
>> 1.004E-01 sec
>>
>> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
>> -4.945E-06  aerosurfs
>>
>> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
>> -2.785E-05  aerosurfs
>>
>> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
>> -4.945E-06  Aircraft-Total
>>
>> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
>> -2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699
>> 1625 12
>>
>> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6
>>
>> aft-Total
>>
>> *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04
>> -4.961E-06  Aircraft-OML
>>
>> *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04
>> -2.791E-05  Aircraft-OML
>>
>> *Cleaned up version:*
>>
>> ZONE   0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10
>> 1.004E-01 sec
>>
>> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
>> -4.945E-06  aerosurfs
>>
>> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
>> -2.785E-05  aerosurfs
>>
>> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04
>> -4.945E-06  Aircraft-Total
>>
>> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04
>> -2.785E-05 Aircraft-Total
>>
>>    Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  699  1625 12
>>
>> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin:  111  1626  6
>>
>> *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03