On Oct 31, 2019, at 6:43 PM, Joseph Schuchart via users <users@lists.open-mpi.org> wrote: > > Just to throw in my $0.02: I recently found that the output to stdout/stderr > may not be desirable: in an application that writes a lot of log data to > stderr on all ranks, stdout was significantly slower than the files I > redirected stdio to (I ended up seeing the application complete in the file > output while the terminal wasn't even halfway through). Redirecting stderr to > /dev/null as Jeff suggests does not help much because the output first has to > be sent to the head node.
Fair point. The redirection of stdout/stderr to mpirun was never intended to be a high performance channel. Specifically: the data from individual MPI processes' stdout/stderr is sent via an "out of band" communications channel -- almost certainly via TCP, and possibly not even the same network as what high-speed MPI traffic flows across. We did not take great (or any?) pains to make this channel be particularly efficient, either. The stdout/stderr data flows -- it'll all get there -- but it'll be slow. If you don't have a lot of output data, this is definitely good enough -- you'll never even notice the "slowness". But if you have oodles of output data, it's definitely noticeable. As Joseph says: 1. If you redirect mpirun's stdout and/or stderr it still takes this slow / inefficient route to get to mpirun, and *then* it is discarded. 2. Therefore, if you have a large volume of output, writing to a file directly may be significantly more efficient / faster. > Point to make: it would be nice to have an option to suppress the output on > stdout and/or stderr when output redirection to file is requested. In my > case, having stdout still visible on the terminal is desirable but having a > way to suppress output of stderr to the terminal would be immensely helpful. I do believe that --output-file will write to a *local* file on the node where it is running (vs. being sent to mpirun, and mpirun writing to the output file). So snipping off the output from being sent to mpirun in the first place would actually be an efficiency-gaining feature. I'll make a feature request to track all the issues that have come out of this discussion. -- Jeff Squyres jsquy...@cisco.com
