Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

[Jeff Squyres (jsquyres) via users]

Please keep "users@lists.open-mpi.org" in the 
CC so that others can Google to find this info in the future.

More below.

On Oct 30, 2019, at 9:08 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

Thank you for the suggestions. I will try it.

About the WRF model, I used WRF-Chem V3.5. My model worked before my service 
upgraded. After that, I could not run my model anymore, and the version of Open 
MPI is the only condition changed to v1.10.7. Thus, I suspected my WRF didn't 
support later versions of Open MPI.

What specifically does "could not run my model any more" mean -- does it mean 
the application failed to compile?  Did the application fail to launch?  Did it 
generate a run-time error?  ...?

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

[Jeff Squyres (jsquyres) via users]

Please keep users@lists.open-mpi.org in the CC 
so that other users can benefit from this information.

More below.


On Oct 30, 2019, at 10:18 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

I added --enable-no-build=btl:openib on the configure command line. I still got 
an error message when I run "make".

Sorry, I was going off memory and misspoke.  The option you want is 
--enable-mca-no-build=btl-openib.

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

[Qianjin Zheng via users]

Hi Jeff,

I tried again, but I still got the error when I ran "make".
Here is the commend when I run the configure:
./configure --prefix=$PATH/LIBRARIES/openmpi --enable-static 
--enable-mpi-thread-multiple --without-usnic --enable-mpi-cxx 
--enable-mca-no-build=btl-openib CC=gcc CXX=g++ FC=gfortran FCFLAGS=-m64 
F77=gfortran FFLAGS=-m64

I also attached my make_log within this email.

Thank you,

Qianjin



From: Jeff Squyres (jsquyres) 
Sent: Thursday, October 31, 2019 5:21 AM
To: Qianjin Zheng 
Cc: Open MPI User's List 
Subject: Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

Please keep users@lists.open-mpi.org in the CC 
so that other users can benefit from this information.

More below.


On Oct 30, 2019, at 10:18 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

I added --enable-no-build=btl:openib on the configure command line. I still got 
an error message when I run "make".

Sorry, I was going off memory and misspoke.  The option you want is 
--enable-mca-no-build=btl-openib.

--
Jeff Squyres
jsquy...@cisco.com

Making all in config
make[1]: Entering directory `/students/WRFtest/lib_test/openmpi-1.10.1/config'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/students/WRFtest/lib_test/openmpi-1.10.1/config'
Making all in contrib
make[1]: Entering directory `/students/WRFtest/lib_test/openmpi-1.10.1/contrib'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/students/WRFtest/lib_test/openmpi-1.10.1/contrib'
Making all in opal
make[1]: Entering directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal'
Making all in include
make[2]: Entering directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/include'
make  all-am
make[3]: Entering directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/include'
make[3]: Leaving directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/include'
make[2]: Leaving directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/include'
Making all in asm
make[2]: Entering directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-amd64-linux.s" atomic-asm.S
  CPPASatomic-asm.lo
  CCLD libasm.la
make[2]: Leaving directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal/asm'
Making all in datatype
make[2]: Entering directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/datatype'
  CC   libdatatype_reliable_la-opal_datatype_pack.lo
  CC   libdatatype_reliable_la-opal_datatype_unpack.lo
  CCLD libdatatype_reliable.la
  CC   opal_convertor.lo
  CC   opal_convertor_raw.lo
  CC   opal_copy_functions.lo
  CC   opal_copy_functions_heterogeneous.lo
  CC   opal_datatype_add.lo
  CC   opal_datatype_clone.lo
  CC   opal_datatype_copy.lo
  CC   opal_datatype_create.lo
  CC   opal_datatype_create_contiguous.lo
  CC   opal_datatype_destroy.lo
  CC   opal_datatype_dump.lo
  CC   opal_datatype_fake_stack.lo
  CC   opal_datatype_get_count.lo
  CC   opal_datatype_module.lo
  CC   opal_datatype_optimize.lo
  CC   opal_datatype_pack.lo
  CC   opal_datatype_position.lo
  CC   opal_datatype_resize.lo
  CC   opal_datatype_unpack.lo
  CCLD libdatatype.la
make[2]: Leaving directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/datatype'
Making all in etc
make[2]: Entering directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal/etc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal/etc'
Making all in util
make[2]: Entering directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/util'
Making all in keyval
make[3]: Entering directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/util/keyval'
  CC   keyval_lex.lo
  CCLD libopalutilkeyval.la
make[3]: Leaving directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/util/keyval'
make[3]: Entering directory 
`/students/WRFtest/lib_test/openmpi-1.10.1/opal/util'
  CC   fd.lo
  CC   alfg.lo
  CC   arch.lo
  CC   argv.lo
  CC   basename.lo
  CC   cmd_line.lo
  CC   crc.lo
  CC   convert.lo
  CC   daemon_init.lo
  CC   error.lo
  CC   few.lo
  CC   if.lo
  CC   keyval_parse.lo
  CC   malloc.lo
  CC   net.lo
  CC   numtostr.lo
  CC   opal_environ.lo
  CC   opal_getcwd.lo
  CC   opal_pty.lo
  CC   os_dirpath.lo
  CC   os_path.lo
  CC   output.lo
  CC   path.lo
  CC   printf.lo
  CC   qsort.lo
  CC   show_help.lo
  CC   show_help_lex.lo
  CC   stacktrace.lo
  CC   strncpy.lo
  CC   sys_limits.lo
  CC   uri.lo
  CCLD libopalutil.la
make[3]: Leaving directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal/util'
make[2]: Leaving directory `/students/WRFtest/lib_test/openmpi-1.10.1/opal/util'
Making a

Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

[Jeff Squyres (jsquyres) via users]

Blarg!  Somehow it still tried to build the openib BTL.

Can you send the stdout from when you ran configure?



On Oct 31, 2019, at 1:57 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

I tried again, but I still got the error when I ran "make".
Here is the commend when I run the configure:
./configure --prefix=$PATH/LIBRARIES/openmpi --enable-static 
--enable-mpi-thread-multiple --without-usnic --enable-mpi-cxx 
--enable-mca-no-build=btl-openib CC=gcc CXX=g++ FC=gfortran FCFLAGS=-m64 
F77=gfortran FFLAGS=-m64

I also attached my make_log within this email.

Thank you,

Qianjin



From: Jeff Squyres (jsquyres) mailto:jsquy...@cisco.com>>
Sent: Thursday, October 31, 2019 5:21 AM
To: Qianjin Zheng mailto:qianjin.zh...@hotmail.com>>
Cc: Open MPI User's List 
mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

Please keep users@lists.open-mpi.org in the CC 
so that other users can benefit from this information.

More below.


On Oct 30, 2019, at 10:18 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

I added --enable-no-build=btl:openib on the configure command line. I still got 
an error message when I run "make".

Sorry, I was going off memory and misspoke.  The option you want is 
--enable-mca-no-build=btl-openib.

--
Jeff Squyres
jsquy...@cisco.com




--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

[Qianjin Zheng via users]

Hi Jeff,

I did not see any stdout from when I ran configure. Can you more specify file 
name?

Thank you,
Qianjin

From: Jeff Squyres (jsquyres) 
Sent: Thursday, October 31, 2019 1:55 PM
To: Qianjin Zheng 
Cc: Open MPI User's List 
Subject: Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

Blarg!  Somehow it still tried to build the openib BTL.

Can you send the stdout from when you ran configure?



On Oct 31, 2019, at 1:57 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

I tried again, but I still got the error when I ran "make".
Here is the commend when I run the configure:
./configure --prefix=$PATH/LIBRARIES/openmpi --enable-static 
--enable-mpi-thread-multiple --without-usnic --enable-mpi-cxx 
--enable-mca-no-build=btl-openib CC=gcc CXX=g++ FC=gfortran FCFLAGS=-m64 
F77=gfortran FFLAGS=-m64

I also attached my make_log within this email.

Thank you,

Qianjin



From: Jeff Squyres (jsquyres) mailto:jsquy...@cisco.com>>
Sent: Thursday, October 31, 2019 5:21 AM
To: Qianjin Zheng mailto:qianjin.zh...@hotmail.com>>
Cc: Open MPI User's List 
mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

Please keep users@lists.open-mpi.org in the CC 
so that other users can benefit from this information.

More below.


On Oct 30, 2019, at 10:18 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

Hi Jeff,

I added --enable-no-build=btl:openib on the configure command line. I still got 
an error message when I run "make".

Sorry, I was going off memory and misspoke.  The option you want is 
--enable-mca-no-build=btl-openib.

--
Jeff Squyres
jsquy...@cisco.com




--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Configure Error for installation of openmpi-1.10.1

[Jeff Squyres (jsquyres) via users]

On Oct 31, 2019, at 4:17 PM, Qianjin Zheng 
mailto:qianjin.zh...@hotmail.com>> wrote:

I did not see any stdout from when I ran configure. Can you more specify file 
name?

When you run Open MPI's "configure" script, there is a ton of output to stdout. 
 Check out https://www.open-mpi.org/community/help/ to see how to capture the 
stdout/stderr from running configure.

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] mpirun --output-filename behavior

[Jeff Squyres (jsquyres) via users]

On Oct 30, 2019, at 2:16 PM, Kulshrestha, Vipul 
mailto:vipul_kulshres...@mentor.com>> wrote:

Given that this is an intended behavior, I have a couple of follow up questions:

1.   What is the purpose of the directory “1” that gets created currently? 
(in /app.log/1/rank./stdout ) Is this hard-coded to “1” or 
could be something else?

The "1" is actually the internal Open MPI (PMIx) job ID.  In most cases it will 
be 1, but it can be larger numbers, too.

FWIW: It looks like when this feature was added, you can add an optional 
":nojobid" to the filename to suppress this level of directories.  For example:

mpirun --output-file foo:nojobid ...

Will result in:

foo/rank.0/...
foo/rank.1/...

As opposed to:

foo/1/rank.0/...
foo/1/rank.1/...

This ":nojobid" suffix needs to go in the documentation for this option, too...

2.   Mpirun also has an option “—merge-stderr-to-stdout”, that allows us to 
merge the stderr into stdout, so we are left with only 1 file (‘stdout’). Are 
there any future change that we need to worry about in using this option to 
merge the 2 streams?

There is nothing planned at the present.

FWIW, it doesn't look like this area of the code has changed substantially 
since 2016.  It's not an area of active development.

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] mpirun --output-filename behavior

[Kulshrestha, Vipul via users]

Thanks Jeff.

“:nojobid” worked well for me and helps me remove 1 extra level of hierarchy 
for log files.

Regards
Vipul


From: Jeff Squyres (jsquyres) [mailto:jsquy...@cisco.com]
Sent: Thursday, October 31, 2019 6:21 PM
To: Kulshrestha, Vipul 
Cc: Open MPI User's List 
Subject: Re: [OMPI users] mpirun --output-filename behavior

On Oct 30, 2019, at 2:16 PM, Kulshrestha, Vipul 
mailto:vipul_kulshres...@mentor.com>> wrote:

Given that this is an intended behavior, I have a couple of follow up questions:

1.   What is the purpose of the directory “1” that gets created currently? 
(in /app.log/1/rank./stdout ) Is this hard-coded to “1” or 
could be something else?

The "1" is actually the internal Open MPI (PMIx) job ID.  In most cases it will 
be 1, but it can be larger numbers, too.

FWIW: It looks like when this feature was added, you can add an optional 
":nojobid" to the filename to suppress this level of directories.  For example:

mpirun --output-file foo:nojobid ...

Will result in:

foo/rank.0/...
foo/rank.1/...

As opposed to:

foo/1/rank.0/...
foo/1/rank.1/...

This ":nojobid" suffix needs to go in the documentation for this option, too...

2.   Mpirun also has an option “—merge-stderr-to-stdout”, that allows us to 
merge the stderr into stdout, so we are left with only 1 file (‘stdout’). Are 
there any future change that we need to worry about in using this option to 
merge the 2 streams?

There is nothing planned at the present.

FWIW, it doesn't look like this area of the code has changed substantially 
since 2016.  It's not an area of active development.

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] mpirun --output-filename behavior

[Joseph Schuchart via users]

On 10/30/19 2:06 AM, Jeff Squyres (jsquyres) via users wrote:


Oh, did the prior behavior *only* output to the file and not to 
stdout/stderr?  Huh.


I guess a workaround for that would be:

     mpirun  ... > /dev/null


Just to throw in my $0.02: I recently found that the output to 
stdout/stderr may not be desirable: in an application that writes a lot 
of log data to stderr on all ranks, stdout was significantly slower than 
the files I redirected stdio to (I ended up seeing the application 
complete in the file output while the terminal wasn't even halfway 
through). Redirecting stderr to /dev/null as Jeff suggests does not help 
much because the output first has to be sent to the head node.


Things got even worse when I tried to use the stdout redirection with 
DDT: it barfed at me for doing pipe redirection in the command 
specification! The DDT terminal is just really slow and made the whole 
exercise worthless.


Point to make: it would be nice to have an option to suppress the output 
on stdout and/or stderr when output redirection to file is requested. In 
my case, having stdout still visible on the terminal is desirable but 
having a way to suppress output of stderr to the terminal would be 
immensely helpful.


Joseph



--
Jeff Squyres
jsquy...@cisco.com 

Re: [OMPI users] mpirun --output-filename behavior

[Gilles Gouaillardet via users]

Joseph,


you can achieve this via an agent (and it works with DDT too)


For example, the nostderr script below redirects each MPI task's stderr 
to /dev/null (so it is not forwarded to mpirun)



$ cat nostderr
#!/bin/sh

exec 2> /dev/null

exec "$@"


and then you can simply


$ mpirun --mca orte_fork_agent /.../nostderr ...



FWIW, and even simpler option (that might not work with DDT though) is to

$ mpirun bash -c './a.out 2> /dev/null'


Cheers,


Gilles

On 11/1/2019 7:43 AM, Joseph Schuchart via users wrote:

On 10/30/19 2:06 AM, Jeff Squyres (jsquyres) via users wrote:


Oh, did the prior behavior *only* output to the file and not to 
stdout/stderr?  Huh.


I guess a workaround for that would be:

     mpirun  ... > /dev/null


Just to throw in my $0.02: I recently found that the output to 
stdout/stderr may not be desirable: in an application that writes a 
lot of log data to stderr on all ranks, stdout was significantly 
slower than the files I redirected stdio to (I ended up seeing the 
application complete in the file output while the terminal wasn't even 
halfway through). Redirecting stderr to /dev/null as Jeff suggests 
does not help much because the output first has to be sent to the head 
node.


Things got even worse when I tried to use the stdout redirection with 
DDT: it barfed at me for doing pipe redirection in the command 
specification! The DDT terminal is just really slow and made the whole 
exercise worthless.


Point to make: it would be nice to have an option to suppress the 
output on stdout and/or stderr when output redirection to file is 
requested. In my case, having stdout still visible on the terminal is 
desirable but having a way to suppress output of stderr to the 
terminal would be immensely helpful.


Joseph



--
Jeff Squyres
jsquy...@cisco.com