On Sep 5, 2018, at 4:11 AM, emre brookes <broo...@uthscsa.edu> wrote:
Thanks Gilles,
My goal is to separate openmpi errors from the stdout of the MPI
program itself so that errors can be identified externally (in
particular in an external framework running MPI jobs from various
developers).
My not so "well written MPI program" was doing this:
MPI_Finalize();
exit( errorcode );
Which I assume you are telling me was bad practice & will replace
with
MPI_Abort( MPI_COMM_WORLD, errorcode );
MPI_Finalize();
exit( errorcode );
I was previously a bit put off of MPI_Abort due to the vagueness
of the man page:
_Description_
This routine makes a "best attempt" to abort all tasks in the
group of comm. This function does not require that the invoking
environment take any action with the error code. However, a UNIX
or POSIX environment should handle this as a return errorcode
from the main program or an abort (errorcode).
& I didn't really have an MPI issue to "Abort", but had used this
for a user input or parameter issue.
Nevertheless, I accept your best practice recommendation.
It was not only the originally reported message, other messages
went to stdout.
Initially used the Ubuntu 16 LTS "$ apt install openmpi-bin
libopenmpi-dev" which got me version (1.10.2),
but this morning compiled and tested 2.1.5, with the same
behavior, e.g.:
$ /src/ompi-2.1.5/bin/mpicxx test_using_mpi_abort.cpp
$ /src/ompi-2.1.5/bin/mpirun -np 2 a.out > stdout
[domain-name-embargoed:26078] 1 more process has sent help message
help-mpi-api.txt / mpi-abort
[domain-name-embargoed:26078] Set MCA parameter
"orte_base_help_aggregate" to 0 to see all help / error messages
$ cat stdout
hello from 0
hello from 1
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
$
Tested 3.1.2, where this has been *somewhat* fixed:
$ /src/ompi-3.1.2/bin/mpicxx test_using_mpi_abort.cpp
$ /src/ompi-3.1.2/bin/mpirun -np 2 a.out > stdout
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[domain-name-embargoed:19784] 1 more process has sent help message
help-mpi-api.txt / mpi-abort
[domain-name-embargoed:19784] Set MCA parameter
"orte_base_help_aggregate" to 0 to see all help / error messages
$ cat stdout
hello from 1
hello from 0
$
But the originally reported error still goes to stdout:
$ /src/ompi-3.1.2/bin/mpicxx test_without_mpi_abort.cpp
$ /src/ompi-3.1.2/bin/mpirun -np 2 a.out > stdout
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero
status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[22380,1],0]
Exit code: 255
--------------------------------------------------------------------------
$ cat stdout
hello from 0
hello from 1
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
$
Summary:
1.10.2, 2.1.5 both send most openmpi generated messages to stdout.
3.1.2 sends at least one type of openmpi generated messages to
stdout.
I'll continue with my "wrapper" strategy for now, as it seems
safest and
most broadly deployable [e.g. on compute resources where I need to
use admin installed versions of MPI],
but it would be nice for openmpi to ensure all generated messages
end up in stderr.
-Emre
Gilles Gouaillardet wrote:
Open MPI should likely write this message on stderr, I will have
a look at that.
That being said, and though I have no intention to dodge the
question, this case should not happen.
A well written (MPI) program should either exit(0) or have main()
return 0, so this scenario
(e.g. all MPI tasks call MPI_Finalize() and then at least one MPI
task exit with a non zero error code)
should not happen.
If your program might fail, it should call MPI_Abort() with a non
zero error code *before* calling MPI_Finalize().
note this error can occur if your main() subroutine does not
return any value (e.g. it returns an undefined value, that might
be non zero)
Cheers,
Gilles
On 9/5/2018 6:08 AM, emre brookes wrote:
Background:
---
Running on ubuntu 16.04 with apt install openmpi-bin libopenmpi-dev
$ mpirun --version
mpirun (Open MPI) 1.10.2
I did search thru the docs a bit (ok, maybe I missed something
obvious, my apologies if so)
---
Question:
Is there some setting to turn off the extra messages generated
by openmpi ?
e.g.
$ mpirun -np 2 my_job > my_job.stdout
adds this message to my_job.stdout
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been
aborted.
-------------------------------------------------------
which strangely goes to stdout and not stderr.
I would intuitively expect error or notice messages to go to
stderr.
Is there a way to redirect these messages to stderr or some
specified file?
I need to separate this from the collected stdout of the job
processes themselves.
Somewhat kludgy options that come to mind:
1. I can use --output-filename outfile, which does separate the
"openmpi" messages,
but this creates a file for each process and I'd rather keep
them as produced in one file,
but without any messages from openmpi, which I'd like to keep
separately.
2. Or I could write a script to filter the output and separate.
A bit risky as someone could conceivably put something that
looks like a openmpi message pattern in the mpi executable output.
3. hack the source code of openmpi.
Any suggestions as to a more elegant or standard way of dealing
with this?
TIA,
Emre.
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
