I would very much appreciate some advice in how to debug this problem. I
am trying to get OpenMPI to work on my reconfigured cluster - upgrading
from Centos 5 to Ubuntu 18. The problem is that a simple job using
Intel's IMB message passing test code will not run on any of the new
clients (4 so far).
mpirun -np 2 IMB-MPI1
just hangs - no printout, no messages in syslog. I left it for 1 hr and
it remained in the same state.
On the other hand the same code runs fine on the server (see outfoam).
Comparing the two it seems the client version hangs while trying to load
the openib module (it works with tcp,self or vader,self).
Digging a bit more I found the --mca btl_base_verbose option. Now I can
see a difference in the two cases:
On the server: ibv_obj->logical_index=1, my_obj->logical_index=0
On the client: ibv_obj->type set to NULL. I don't believe this is a good
sign, but I don't understand what it means. My guess is that openib is
not being initialized in this case.
The server (foam) is SuperMicro server with X10DAi m'board and 2XE52630
(10 core).
The client (f34) is a Dell R410 server with 2XE5620 (4 core). The
outputs from ompi_info are attached.
They are both running Ubuntu 18.04 with the latest updates. I installed
openmpi-bin 2.1.1-8. Both boxes have Mellanox Connect X2 cards with the
latest firmware (2.9.1000). I have checked that the cards send and
receive packets using the IB protocols and pass the Mellanox diagnostics.
I did notice that the Mellanox card has the PCI address 81:00.0 on the
server but 03:00.0 on the client. Not sure of the significance of this.
Any help anyone can offer would be much appreciated. I am stuck.
Thanks
Tony
--
Tony Ladd
Chemical Engineering Department
University of Florida
Gainesville, Florida 32611-6005
USA
Email: tladd-"(AT)"-che.ufl.edu
Webhttp://ladd.che.ufl.edu
Tel: (352)-392-6509
FAX: (352)-392-9514
Package: Open MPI buildd@lcy01-amd64-009 Distribution
Open MPI: 2.1.1
Open MPI repo revision: v2.1.0-100-ga2fdb5b
Open MPI release date: May 10, 2017
Open RTE: 2.1.1
Open RTE repo revision: v2.1.0-100-ga2fdb5b
Open RTE release date: May 10, 2017
OPAL: 2.1.1
OPAL repo revision: v2.1.0-100-ga2fdb5b
OPAL release date: May 10, 2017
MPI API: 3.1.0
Ident string: 2.1.1
Prefix: /usr
Configured architecture: x86_64-pc-linux-gnu
Configure host: lcy01-amd64-009
Configured by: buildd
Configured on: Mon Feb 5 19:59:59 UTC 2018
Configure host: lcy01-amd64-009
Built by: buildd
Built on: Mon Feb 5 20:05:56 UTC 2018
Built host: lcy01-amd64-009
C bindings: yes
C++ bindings: yes
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
limitations in the gfortran compiler, does not support the following: array
subsections, direct passthru (where possible) to underlying Open MPI's C
functionality
Fort mpi_f08 subarrays: no
Java bindings: yes
Wrapper compiler rpath: disabled
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 7.3.0
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fort compiler: gfortran
Fort compiler abs: /usr/bin/gfortran
Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: yes
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib: yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: yes
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
MPI extensions: affinity, cuda
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: sm (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: self (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v2.1.1)
MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA pmix: pmix112 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v2.1.1)
MCA sec: basic (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v2.1.1)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v2.1.1)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA iof: mr_orted (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: mr_hnp (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA oob: ud (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA oob: usock (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: staged (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rml: oob (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: debruijn (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: radix (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: binomial (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: direct (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rtc: freq (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: staged_orted (MCA v2.1.0, API v1.0.0, Component
v2.1.1)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: staged_hnp (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: static (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v2.1.1)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v2.1.1)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component v2.1.1)
Package: Open MPI buildd@lcy01-amd64-009 Distribution
Open MPI: 2.1.1
Open MPI repo revision: v2.1.0-100-ga2fdb5b
Open MPI release date: May 10, 2017
Open RTE: 2.1.1
Open RTE repo revision: v2.1.0-100-ga2fdb5b
Open RTE release date: May 10, 2017
OPAL: 2.1.1
OPAL repo revision: v2.1.0-100-ga2fdb5b
OPAL release date: May 10, 2017
MPI API: 3.1.0
Ident string: 2.1.1
Prefix: /usr
Configured architecture: x86_64-pc-linux-gnu
Configure host: lcy01-amd64-009
Configured by: buildd
Configured on: Mon Feb 5 19:59:59 UTC 2018
Configure host: lcy01-amd64-009
Built by: buildd
Built on: Mon Feb 5 20:05:56 UTC 2018
Built host: lcy01-amd64-009
C bindings: yes
C++ bindings: yes
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
limitations in the gfortran compiler, does not support the following: array
subsections, direct passthru (where possible) to underlying Open MPI's C
functionality
Fort mpi_f08 subarrays: no
Java bindings: yes
Wrapper compiler rpath: disabled
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 7.3.0
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fort compiler: gfortran
Fort compiler abs: /usr/bin/gfortran
Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: yes
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib: yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: yes
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
MPI extensions: affinity, cuda
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA btl: self (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA btl: sm (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v2.1.1)
MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA pmix: pmix112 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v2.1.1)
MCA sec: basic (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v2.1.1)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v2.1.1)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: mr_orted (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA iof: mr_hnp (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA oob: ud (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA oob: usock (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: staged (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rml: oob (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: binomial (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: debruijn (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: radix (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA routed: direct (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA rtc: freq (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: staged_hnp (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA state: staged_orted (MCA v2.1.0, API v1.0.0, Component
v2.1.1)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v2.1.1)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: static (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v2.1.1)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v2.1.1)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v2.1.1)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v2.1.1)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component v2.1.1)
f34:tladd(~)> mpirun -d --report-bindings -mca btl openib,self --mca
btl_base_verbose 30 -np 2 mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1 SendRecv
[f34:19197] procdir: /tmp/openmpi-sessions-501@f34_0/57378/0/0
[f34:19197] jobdir: /tmp/openmpi-sessions-501@f34_0/57378/0
[f34:19197] top: openmpi-sessions-501@f34_0
[f34:19197] tmp: /tmp
[f34:19197] sess_dir_cleanup: job session dir does not exist
[f34:19197] procdir: /tmp/openmpi-sessions-501@f34_0/57378/0/0
[f34:19197] jobdir: /tmp/openmpi-sessions-501@f34_0/57378/0
[f34:19197] top: openmpi-sessions-501@f34_0
[f34:19197] tmp: /tmp
[f34:19197] [[57378,0],0] Releasing job data for [INVALID]
[f34:19197] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 1[hwt
0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
[f34:19197] MCW rank 1 bound to socket 0[core 0[hwt 0]], socket 0[core 1[hwt
0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
MPIR_being_debugged = 0
MPIR_debug_state = 1
MPIR_partial_attach_ok = 1
MPIR_i_am_starter = 0
MPIR_forward_output = 0
MPIR_proctable_size = 2
MPIR_proctable:
(i, host, exe, pid) = (0, f34,
/home/tladd/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 19202)
(i, host, exe, pid) = (1, f34,
/home/tladd/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 19203)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[f34:19202] procdir: /tmp/openmpi-sessions-501@f34_0/57378/1/0
[f34:19202] jobdir: /tmp/openmpi-sessions-501@f34_0/57378/1
[f34:19202] top: openmpi-sessions-501@f34_0
[f34:19202] tmp: /tmp
[f34:19203] procdir: /tmp/openmpi-sessions-501@f34_0/57378/1/1
[f34:19203] jobdir: /tmp/openmpi-sessions-501@f34_0/57378/1
[f34:19203] top: openmpi-sessions-501@f34_0
[f34:19203] tmp: /tmp
[f34:19202] mca: base: components_register: registering framework btl components
[f34:19202] mca: base: components_register: found loaded component openib
[f34:19202] mca: base: components_register: component openib register function
successful
[f34:19202] mca: base: components_register: found loaded component self
[f34:19202] mca: base: components_register: component self register function
successful
[f34:19202] mca: base: components_open: opening btl components
[f34:19202] mca: base: components_open: found loaded component openib
[f34:19202] mca: base: components_open: component openib open function
successful
[f34:19202] mca: base: components_open: found loaded component self
[f34:19202] mca: base: components_open: component self open function successful
[f34:19202] select: initializing btl component openib
[f34:19203] mca: base: components_register: registering framework btl components
[f34:19203] mca: base: components_register: found loaded component openib
[f34:19202] Checking distance from this process to device=mlx4_0
[f34:19202] hwloc_distances->nbobjs=2
[f34:19202] hwloc_distances->latency[0]=1.000000
[f34:19202] hwloc_distances->latency[1]=2.000000
[f34:19202] hwloc_distances->latency[2]=2.000000
[f34:19202] hwloc_distances->latency[3]=1.000000
[f34:19202] ibv_obj->type set to NULL
[f34:19202] Process is bound: distance to device is 0.000000
[f34:19203] mca: base: components_register: component openib register function
successful
[f34:19203] mca: base: components_register: found loaded component self
[f34:19203] mca: base: components_register: component self register function
successful
[f34:19203] mca: base: components_open: opening btl components
[f34:19203] mca: base: components_open: found loaded component openib
[f34:19203] mca: base: components_open: component openib open function
successful
[f34:19203] mca: base: components_open: found loaded component self
[f34:19203] mca: base: components_open: component self open function successful
[f34:19203] select: initializing btl component openib
[f34:19203] Checking distance from this process to device=mlx4_0
[f34:19203] hwloc_distances->nbobjs=2
[f34:19203] hwloc_distances->latency[0]=1.000000
[f34:19203] hwloc_distances->latency[1]=2.000000
[f34:19203] hwloc_distances->latency[2]=2.000000
[f34:19203] hwloc_distances->latency[3]=1.000000
[f34:19203] ibv_obj->type set to NULL
[f34:19203] Process is bound: distance to device is 0.000000
[f34:19202] [rank=0] openib: using port mlx4_0:1
[f34:19202] select: init of component openib returned success
[f34:19202] select: initializing btl component self
[f34:19202] select: init of component self returned success
[f34:19203] [rank=1] openib: using port mlx4_0:1
[f34:19203] select: init of component openib returned success
[f34:19203] select: initializing btl component self
[f34:19203] select: init of component self returned success
[f34:19202] mca: bml: Using self btl for send to [[57378,1],0] on node f34
[f34:19203] mca: bml: Using self btl for send to [[57378,1],1] on node f34
^C[
f34:19197] sess_dir_finalize: proc session dir does not exist
[f34:19197] sess_dir_finalize: proc session dir does not exist
[f34:19197] sess_dir_finalize: proc session dir does not exist
[f34:19197] sess_dir_finalize: proc session dir does not exist
[f34:19197] sess_dir_finalize: proc session dir does not exist
[f34:19197] sess_dir_cleanup: job session dir does not exist
exiting with status 1
foam:tladd(tmp)> mpirun -d --report-bindings -mca btl openib,self --mca
btl_base_verbose 30 -np 2 mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1 SendRecv
[foam:37080] procdir: /tmp/openmpi-sessions-501@foam_0/34762/0/0
[foam:37080] jobdir: /tmp/openmpi-sessions-501@foam_0/34762/0
[foam:37080] top: openmpi-sessions-501@foam_0
[foam:37080] tmp: /tmp
[foam:37080] sess_dir_cleanup: job session dir does not exist
[foam:37080] procdir: /tmp/openmpi-sessions-501@foam_0/34762/0/0
[foam:37080] jobdir: /tmp/openmpi-sessions-501@foam_0/34762/0
[foam:37080] top: openmpi-sessions-501@foam_0
[foam:37080] tmp: /tmp
[foam:37080] [[34762,0],0] Releasing job data for [INVALID]
[foam:37080] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt
0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 0[core
4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket
0[core 7[hwt 0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 0-1]]:
[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
[foam:37080] MCW rank 1 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt
0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 0[core
4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket
0[core 7[hwt 0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt 0-1]]:
[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
MPIR_being_debugged = 0
MPIR_debug_state = 1
MPIR_partial_attach_ok = 1
MPIR_i_am_starter = 0
MPIR_forward_output = 0
MPIR_proctable_size = 2
MPIR_proctable:
(i, host, exe, pid) = (0, foam,
/tmp/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 37085)
(i, host, exe, pid) = (1, foam,
/tmp/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 37086)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[foam:37085] procdir: /tmp/openmpi-sessions-501@foam_0/34762/1/0
[foam:37085] jobdir: /tmp/openmpi-sessions-501@foam_0/34762/1
[foam:37085] top: openmpi-sessions-501@foam_0
[foam:37085] tmp: /tmp
[foam:37086] procdir: /tmp/openmpi-sessions-501@foam_0/34762/1/1
[foam:37086] jobdir: /tmp/openmpi-sessions-501@foam_0/34762/1
[foam:37086] top: openmpi-sessions-501@foam_0
[foam:37086] tmp: /tmp
[foam:37085] mca: base: components_register: registering framework btl
components
[foam:37085] mca: base: components_register: found loaded component self
[foam:37085] mca: base: components_register: component self register function
successful
[foam:37085] mca: base: components_register: found loaded component openib
[foam:37085] mca: base: components_register: component openib register function
successful
[foam:37085] mca: base: components_open: opening btl components
[foam:37085] mca: base: components_open: found loaded component self
[foam:37085] mca: base: components_open: component self open function successful
[foam:37085] mca: base: components_open: found loaded component openib
[foam:37085] mca: base: components_open: component openib open function
successful
[foam:37085] select: initializing btl component self
[foam:37085] select: init of component self returned success
[foam:37085] select: initializing btl component openib
[foam:37085] Checking distance from this process to device=mlx4_0
[foam:37085] hwloc_distances->nbobjs=2
[foam:37085] hwloc_distances->latency[0]=1.000000
[foam:37085] hwloc_distances->latency[1]=2.100000
[foam:37085] hwloc_distances->latency[2]=2.100000
[foam:37085] hwloc_distances->latency[3]=1.000000
[foam:37085] ibv_obj->logical_index=1
[foam:37085] my_obj->logical_index=0
[foam:37085] Process is bound: distance to device is 2.100000
[foam:37085] [rank=0] openib: using port mlx4_0:1
[foam:37085] select: init of component openib returned success
[foam:37086] mca: base: components_register: registering framework btl
components
[foam:37086] mca: base: components_register: found loaded component self
[foam:37086] mca: base: components_register: component self register function
successful
[foam:37086] mca: base: components_register: found loaded component openib
[foam:37086] mca: base: components_register: component openib register function
successful
[foam:37086] mca: base: components_open: opening btl components
[foam:37086] mca: base: components_open: found loaded component self
[foam:37086] mca: base: components_open: component self open function successful
[foam:37086] mca: base: components_open: found loaded component openib
[foam:37086] mca: base: components_open: component openib open function
successful
[foam:37086] select: initializing btl component self
[foam:37086] select: init of component self returned success
[foam:37086] select: initializing btl component openib
[foam:37086] Checking distance from this process to device=mlx4_0
[foam:37086] hwloc_distances->nbobjs=2
[foam:37086] hwloc_distances->latency[0]=1.000000
[foam:37086] hwloc_distances->latency[1]=2.100000
[foam:37086] hwloc_distances->latency[2]=2.100000
[foam:37086] hwloc_distances->latency[3]=1.000000
[foam:37086] ibv_obj->logical_index=1
[foam:37086] my_obj->logical_index=0
[foam:37086] Process is bound: distance to device is 2.100000
[foam:37086] [rank=1] openib: using port mlx4_0:1
[foam:37086] select: init of component openib returned success
[foam:37086] mca: bml: Using self btl for send to [[34762,1],1] on node foam
[foam:37085] mca: bml: Using self btl for send to [[34762,1],0] on node foam
[foam:37086] mca: bml: Using openib btl for send to [[34762,1],0] on node foam
[foam:37085] mca: bml: Using openib btl for send to [[34762,1],1] on node foam
#------------------------------------------------------------
# Intel(R) MPI Benchmarks 2018, MPI-1 part
#------------------------------------------------------------
# Date : Mon Aug 17 13:50:33 2020
# Machine : x86_64
# System : Linux
# Release : 4.15.0-112-generic
# Version : #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
# MPI Version : 3.1
# MPI Thread Environment:
# Calling sequence was:
# mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1 SendRecv
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Sendrecv
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 1.64 1.64 1.64 0.00
1 1000 1.83 1.84 1.84 1.09
2 1000 1.82 1.82 1.82 2.20
4 1000 1.84 1.84 1.84 4.34
8 1000 1.89 1.89 1.89 8.45
16 1000 1.91 1.91 1.91 16.79
32 1000 1.89 1.89 1.89 33.80
64 1000 2.12 2.12 2.12 60.27
128 1000 3.13 3.13 3.13 81.71
256 1000 3.38 3.39 3.38 151.26
512 1000 3.87 3.88 3.87 264.21
1024 1000 4.67 4.67 4.67 438.79
2048 1000 5.89 5.90 5.89 694.78
4096 1000 7.23 7.24 7.23 1131.99
8192 1000 9.99 9.99 9.99 1639.68
16384 1000 16.46 16.46 16.46 1990.64
32768 1000 26.25 26.25 26.25 2496.95
65536 640 45.52 45.53 45.52 2879.11
131072 320 87.88 87.88 87.88 2983.01
262144 160 169.35 169.36 169.35 3095.74
524288 80 331.92 331.94 331.93 3158.95
1048576 40 660.48 660.51 660.50 3175.05
2097152 20 1318.52 1318.55 1318.53 3181.00
4194304 10 2933.09 2933.24 2933.16 2859.84
# All processes entering MPI_Finalize
[foam:37086] mca: base: close: component self closed
[foam:37086] mca: base: close: unloading component self
[foam:37086] mca: base: close: component openib closed
[foam:37086] mca: base: close: unloading component openib
[foam:37085] mca: base: close: component self closed
[foam:37085] mca: base: close: unloading component self
[foam:37085] mca: base: close: component openib closed
[foam:37085] mca: base: close: unloading component openib
[foam:37086] sess_dir_finalize: proc session dir does not exist
[foam:37085] sess_dir_finalize: proc session dir does not exist
[foam:37080] sess_dir_finalize: proc session dir does not exist
[foam:37080] sess_dir_finalize: proc session dir does not exist
[foam:37080] sess_dir_finalize: proc session dir does not exist
[foam:37080] sess_dir_cleanup: job session dir does not exist
exiting with status 0
