Re: [OMPI users] stdout scrambled in file

2021-12-18 Thread Ralph Castain via users
___ > From: users on behalf of Fisher (US), Mark > S via users > Sent: Monday, December 6, 2021 3:45 PM > To: Joachim Protze; Open MPI Users > Cc: Fisher (US), Mark S > Subject: Re: [OMPI users] stdout scrambled in file > > This usual

Re: [OMPI users] stdout scrambled in file

2021-12-07 Thread Jeff Squyres (jsquyres) via users
m: users on behalf of Fisher (US), Mark S via users Sent: Monday, December 6, 2021 3:45 PM To: Joachim Protze; Open MPI Users Cc: Fisher (US), Mark S Subject: Re: [OMPI users] stdout scrambled in file This usually happens if we get a number of warning message from multiple processes. Seems

Re: [OMPI users] stdout scrambled in file

2021-12-06 Thread Fisher (US), Mark S via users
Protze Sent: Monday, December 06, 2021 11:12 AM To: Fisher (US), Mark S ; Open MPI Users Subject: Re: [OMPI users] stdout scrambled in file I would assume, that the buffering mode is compiler/runtime specific. At least for Intel compiler, the default seems to be/have been unbuffered for

Re: [OMPI users] stdout scrambled in file

2021-12-06 Thread Joachim Protze via users
: Joachim Protze Sent: Monday, December 6, 2021 9:24 AM To: Open MPI Users Cc: Fisher (US), Mark S Subject: Re: [OMPI users] stdout scrambled in file Hi Mark, "[...] MPI makes neither requirements nor recommendations for the output [...]" (MPI4.0, §2.9.1) From my experience, an appli

Re: [OMPI users] stdout scrambled in file

2021-12-06 Thread Fisher (US), Mark S via users
Message- From: Joachim Protze Sent: Monday, December 6, 2021 9:24 AM To: Open MPI Users Cc: Fisher (US), Mark S Subject: Re: [OMPI users] stdout scrambled in file Hi Mark, "[...] MPI makes neither requirements nor recommendations for the output [...]" (MPI4.0, §2.9.1) From my

Re: [OMPI users] stdout scrambled in file

2021-12-06 Thread Joachim Protze via users
Hi Mark, "[...] MPI makes neither requirements nor recommendations for the output [...]" (MPI4.0, §2.9.1) From my experience, an application can avoid such scrambling (still no guarantee), if the output of lines is written atomically. C++ streams are worst for concurrent output, as every str

Re: [OMPI users] stdout scrambled in file

2021-12-06 Thread Fisher (US), Mark S via users
5, 2021 2:48 PM To: Open MPI Users Cc: Ralph Castain Subject: [EXTERNAL] Re: [OMPI users] stdout scrambled in file EXT email: be mindful of links/attachments. There are several output-controlling options - e.g., you could redirect the output from each process to its own file or directory

Re: [OMPI users] stdout scrambled in file

2021-12-05 Thread Ralph Castain via users
There are several output-controlling options - e.g., you could redirect the output from each process to its own file or directory. However, it makes little sense to me for someone to write convergence data into a file and then parse it. Typically, convergence data results from all procs reachin

Re: [OMPI users] stdout scrambled in file

2021-12-05 Thread Gus Correa via users
Hi Mark Back in the days I liked the mpirun/mpiexec *--tag-output *option. Jeff: Does it still exist? It may not prevent 100% the splitting of output lines, but tagging the lines with the process rank helps. You can grep the stdout log for the rank that you want, which helps a lot when several pro

Re: [OMPI users] stdout scrambled in file

2021-12-05 Thread Jeff Squyres (jsquyres) via users
FWIW: Open MPI 4.1.2 has been released -- you can probably stop using an RC release. I think you're probably running into an issue that is just a fact of life. Especially when there's a lot of output simultaneously from multiple MPI processes (potentially on different nodes), the stdout/stderr