Re: [OMPI users] MCA version error

2017-10-16 Thread Jeff Squyres (jsquyres)
Yes, it looks like you are mixing multiple versions of Open MPI.

It looks like you have installed Open MPI v1.10.0 via RPM in the system 
defaults location (perhaps in /usr/bin and /usr/lib and the like?), and then 
installed Open MPI v2.1.1 via building from source under your home directory 
(/home/user/EMAN2/bin and /home/user/EMAN2/lib and the like).

Your statement about your program working fine with MCA 2.0 is a little 
suspicious -- your should first clear up your PATH issues, recompile your 
application, and try running again.  If that fails, please provide all the 
information listed in https://www.open-mpi.org/community/help/.



> On Oct 13, 2017, at 11:41 AM, abhisek Mondal  wrote:
> 
> Hi,
> 
> I'm bit confused now.
> When I supply ompi_info, it gives:
> [user@localhost openmpi-x86_64]$ ompi_info 
> [localhost.localdomain:11068] Error: Unable to get the current working 
> directory
>  Package: Open MPI root@3fecdf07461e Distribution
> Open MPI: 2.1.1
>   Open MPI repo revision: v2.1.0-100-ga2fdb5b
>Open MPI release date: May 10, 2017
> Open RTE: 2.1.1
>   Open RTE repo revision: v2.1.0-100-ga2fdb5b
>Open RTE release date: May 10, 2017
> OPAL: 2.1.1
>   OPAL repo revision: v2.1.0-100-ga2fdb5b
>OPAL release date: May 10, 2017
>  MPI API: 3.1.0
> Ident string: 2.1.1
>   Prefix: /home/user/EMAN2
>  Configured architecture: x86_64-unknown-linux-gnu
> 
> Form the Bold line, I assume it is sourcing this version and loading 2.1.1 by 
> default.
> 
> But when I provide: rpm -qf /usr/lib64/openmpi/lib/openmpi/mca_shmem_mmap.so
> It gives me: openmpi-1.10.0-10.el7.x86_64
> 
> So, I think I'm having some sort of path issue. Is that right ? The program I 
> wish to run will go fine with MCA 2.0. 
> 
> On Fri, Oct 13, 2017 at 3:06 PM, Jeff Squyres (jsquyres)  
> wrote:
> From the output you supplied, it looks like you are running Open MPI v2.1.x.
> 
> Did you install Open MPI v2.1.x from source, and install it into the same 
> directory that you had previously installed Open MPI v2.0.x?
> 
> If so, the warnings you are seeing (not errors) are likely the fact that 
> there are some Open MPI v2.0.x plugins left over in the Open MPI v2.1.x 
> installation tree (installing version B of Open MPI over a previous 
> installation of Open MPI version A does *not* remove all of version A's 
> files).
> 
> You should probably get a clean install of Open MPI v2.1.x, and the warnings 
> will disappear.
> 
> See https://www.open-mpi.org/faq/?category=building#install-overwrite for 
> more details.
> 
> 
> 
> > On Oct 13, 2017, at 10:02 AM, Gilles Gouaillardet 
> >  wrote:
> >
> > one more thing,
> > can you run
> > rpm -qf /usr/lib64/openmpi/lib/openmpi/mca_shmem_mmap.so
> > and make sure this comes from the openmpi rpm ?
> >
> > if not, you can remove the openmpi rpm, rm -rf
> > /usr/lib64/openmpi/lib/openmpi and then re-install the rpm
> >
> > Cheers,
> >
> > Gilles
> >
> > On Fri, Oct 13, 2017 at 10:49 PM, Gilles Gouaillardet
> >  wrote:
> >> can you run ldd on your binary ?
> >> i suspect it might not use the right Open MPI library
> >>
> >> Cheers,
> >>
> >> Gilles
> >>
> >> On Fri, Oct 13, 2017 at 10:41 PM, abhisek Mondal  
> >> wrote:
> >>> Hello,
> >>>
> >>> I'm getting this following error:
> >>> [localhost.localdomain:00307] mca_base_component_repository_open: shmem
> >>> "/usr/lib64/openmpi/lib/openmpi/mca_shmem_mmap" uses an MCA interface that
> >>> is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- 
> >>> ignored
> >>> [localhost.localdomain:00307] mca_base_component_repository_open: unable 
> >>> to
> >>> open mca_shmem_posix: /usr/lib64/openmpi/lib/openmpi/mca_shmem_posix.so:
> >>> undefined symbol: opal_shmem_base_output (ignored)
> >>> [localhost.localdomain:00307] mca_base_component_repository_open: shmem
> >>> "/usr/lib64/openmpi/lib/openmpi/mca_shmem_sysv" uses an MCA interface that
> >>> is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- 
> >>> ignored
> >>>
> >>> I had installed it using yum command: yum install openmpi-1.10.0-10.el7
> >>>
> >>> My current MCA version is showing 2.1.0.
> >>>
> >>> On Fri, Oct 13, 2017 at 1:24 PM, Gilles Gouaillardet
> >>>  wrote:
> 
>  Hi,
> 
>  let's take one or two steps back.
> 
>  which version of Open MPI did you use in order to build your program ?
> 
>  what does "not working under MCA2.1" mean ?
>  link error ? unresolved symbols ? runtime crash ?
> 
>  please detail your environment and post all relevant error messages
> 
>  Cheers,
> 
>  Gilles
> 
>  On Fri, Oct 13, 2017 at 10:17 PM, abhisek Mondal 
>  wrote:
> >
> > Hi,
> >
> > I have installed an openmpi using following command:
> > yum install openmpi-1.10.0-10.el7
> >
> > When I put ompi_info command it shows me that it is using MCAv.2.1. I

[OMPI users] MPI_Bcast performance doesn't improve after enabling tree implementation

2017-10-16 Thread Konstantinos Konstantinidis
I have implemented some algorithms in C++ which are greatly affected by
shuffling time among nodes which is done by some broadcast calls. Up to
now, I have been testing them by running something like

mpirun -mca btl ^openib -mca plm_rsh_no_tree_spawn 1 ./my_test

which I think make MPI_Bcast to work serially. Now, I want to improve the
communication time so I have configured the appropriate SSH access from
every node to every other node and I have enabled the binary tree
implementation of Open MPI collective calls by running

mpirun -mca btl ^openib ./my_test

My problem is that throughout various experiments with files of different
sizes, I realized that there is no improvement in terms of transmission
time even though theoretically I would expect a gain of approximately
(log(k))/(k-1) where k is the size of the group that the communication
takes place within.

I compile the code with

mpic++ my_test.cc -o my_test

and all of the experiments are done on Amazon EC2 r3.large or m3.large
machines. I have also set different values of rate limits to avoid bursty
behavior of Amazon's EC2 transmission rate. The Open MPI I have installed
is described on the txt I have attached after running ompi_info.

What can be wrong here?
 Package: Open MPI buildd@lgw01-57 Distribution
Open MPI: 1.10.2
  Open MPI repo revision: v1.10.1-145-g799148f
   Open MPI release date: Jan 21, 2016
Open RTE: 1.10.2
  Open RTE repo revision: v1.10.1-145-g799148f
   Open RTE release date: Jan 21, 2016
OPAL: 1.10.2
  OPAL repo revision: v1.10.1-145-g799148f
   OPAL release date: Jan 21, 2016
 MPI API: 3.0.0
Ident string: 1.10.2
  Prefix: /usr
 Configured architecture: x86_64-pc-linux-gnu
  Configure host: lgw01-57
   Configured by: buildd
   Configured on: Thu Feb 25 16:33:01 UTC 2016
  Configure host: lgw01-57
Built by: buildd
Built on: Thu Feb 25 16:40:59 UTC 2016
  Built host: lgw01-57
  C bindings: yes
C++ bindings: yes
 Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
   Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
  limitations in the gfortran compiler, does not
  support the following: array subsections, direct
  passthru (where possible) to underlying Open MPI's
  C functionality
  Fort mpi_f08 subarrays: no
   Java bindings: no
  Wrapper compiler rpath: runpath
  C compiler: gcc
 C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
  C compiler version: 5.3.1
C++ compiler: g++
   C++ compiler absolute: /usr/bin/g++
   Fort compiler: gfortran
   Fort compiler abs: /usr/bin/gfortran
 Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
  Fort optional args: yes
  Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
   Fort STORAGE_SIZE: yes
  Fort BIND(C) (all): yes
  Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
   Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
  Fort PROTECTED: yes
   Fort ABSTRACT: yes
   Fort ASYNCHRONOUS: yes
  Fort PROCEDURE: yes
 Fort USE...ONLY: yes
   Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
 Fort MPI_SIZEOF: yes
 C profiling: yes
   C++ profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
  C++ exceptions: no
  Thread support: posix (MPI_THREAD_MULTIPLE: no, OPAL support: yes,
  OMPI progress: no, ORTE progress: yes, Event lib:
  yes)
   Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
 MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
  dl support: yes
   Heterogeneous support: yes
 mpirun default --prefix: no
 MPI I/O support: yes
   MPI_WTIME support: gettimeofday
 Symbol vis. support: yes
   Host topology support: yes
  MPI extensions: 
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
 VampirTrace support: no
  MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
 MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
   MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
   MCA backtrace: execinfo (MCA v2.0.0, API v2.0.0, Component
  v1.10.2)
MCA compress: gzip (MCA v2.0.0, API v2.0.0, Component v1.10.2)
MCA compress: bzip (MCA v2.0.0, API v2.0.0,