Attached are the results of "grep thread" on my configure output. There appears
to be some amount of threading, but is there anything I should look for in
particular?
I see Mike Dubman's questions on the mailing list website, but his message
didn't appear to make it to my inbox. The answers to his questions are:
[binford:fischega] $ rpm -qa | grep ofed
ofed-doc-1.5.4.1-0.11.5
ofed-kmp-default-1.5.4.1_3.0.76_0.11-0.11.5
ofed-1.5.4.1-0.11.5
Distro: SLES11 SP3
HCA:
[binf102:fischega] $ /usr/sbin/ibstat
CA 'mlx4_0'
CA type: MT26428
Command line (path and LD_LIBRARY_PATH are set correctly):
mpirun -x LD_LIBRARY_PATH -mca btl openib,sm,self -mca btl_openib_verbose 1 -np
31 $CTF_EXEC
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Maxime
Boissonneault
Sent: Tuesday, June 24, 2014 6:41 PM
To: Open MPI Users
Subject: Re: [OMPI users] poor performance using the openib btl
What are your threading options for OpenMPI (when it was built) ?
I have seen OpenIB BTL completely lock when some level of threading is enabled
before.
Maxime Boissonneault
Le 2014-06-24 18:18, Fischer, Greg A. a écrit :
Hello openmpi-users,
A few weeks ago, I posted to the list about difficulties I was having getting
openib to work with Torque (see "openib segfaults with Torque", June 6, 2014).
The issues were related to Torque imposing restrictive limits on locked memory,
and have since been resolved.
However, now that I've had some time to test the applications, I'm seeing
abysmal performance over the openib layer. Applications run with the tcp btl
execute about 10x faster than with the openib btl. Clearly something still
isn't quite right.
I tried running with "-mca btl_openib_verbose 1", but didn't see anything
resembling a smoking gun. How should I go about determining the source of the
problem? (This uses the same OpenMPI Version 1.8.1 / SLES11 SP3 / GCC 4.8.3
setup discussed previously.)
Thanks,
Greg
_______________________________________________
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/06/24697.php
--
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Ph. D. en physique
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking if C compiler and POSIX threads work as is... no
checking if C++ compiler and POSIX threads work as is... no
checking if Fortran compiler and POSIX threads work as is... no
checking if C compiler and POSIX threads work with -Kthread... no
checking if C compiler and POSIX threads work with -kthread... no
checking if C compiler and POSIX threads work with -pthread... yes
checking if C++ compiler and POSIX threads work with -Kthread... no
checking if C++ compiler and POSIX threads work with -kthread... no
checking if C++ compiler and POSIX threads work with -pthread... yes
checking if Fortran compiler and POSIX threads work with -Kthread... no
checking if Fortran compiler and POSIX threads work with -kthread... no
checking if Fortran compiler and POSIX threads work with -pthread... yes
checking for pthread_mutexattr_setpshared... yes
checking for pthread_condattr_setpshared... yes
checking for working POSIX threads package... yes
checking for type of thread support... posix
checking if threads have different pids (pthreads on linux)... no
checking for pthread_t... yes
checking pthread_np.h usability... no
checking pthread_np.h presence... no
checking for pthread_np.h... no
checking whether pthread_setaffinity_np is declared... yes
checking whether pthread_getaffinity_np is declared... yes
checking for library containing pthread_getthrds_np... no
checking for pthread_mutex_lock... yes
checking libevent configuration args... --disable-dns --disable-http
--disable-rpc --disable-openssl --enable-thread-support --disable-evport
configure: running /bin/sh
'../../../../../../openmpi-1.8.1/opal/mca/event/libevent2021/libevent/configure'
--disable-dns --disable-http --disable-rpc --disable-openssl
--enable-thread-support --disable-evport
'--prefix=/casl/vera_ib/gcc-4.8.3/toolset/openmpi-1.8.1' --cache-file=/dev/null
--srcdir=../../../../../../openmpi-1.8.1/opal/mca/event/libevent2021/libevent
--disable-option-checking
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking size of pthread_t... 8
config.status: creating libevent_pthreads.pc
checking for thread support (needed for rdmacm/udcm)... posix
configure: running /bin/sh
'../../../../../../openmpi-1.8.1/ompi/mca/io/romio/romio/configure'
FROM_OMPI=yes CC="gcc -std=gnu99" CFLAGS="-O3 -DNDEBUG -finline-functions
-fno-strict-aliasing -pthread" CPPFLAGS="
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/opal/mca/hwloc/hwloc172/hwloc/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/build/opal/mca/hwloc/hwloc172/hwloc/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/opal/mca/event/libevent2021/libevent
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/opal/mca/event/libevent2021/libevent/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/build/opal/mca/event/libevent2021/libevent/include"
FFLAGS="" LDFLAGS=" " --enable-shared --disable-static
--prefix=/casl/vera_ib/gcc-4.8.3/toolset/openmpi-1.8.1 --disable-aio
--cache-file=/dev/null
--srcdir=../../../../../../openmpi-1.8.1/ompi/mca/io/romio/romio
--disable-option-checking
checking for a thread-safe mkdir -p... /bin/mkdir -p
setting CFLAGS to -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -pthread
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DHAVE_ROMIOCONF_H
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_condattr_getpshared... yes
checking for pthread_condattr_setpshared... yes
checking for pthread_mutexattr_getpshared... yes
checking for pthread_mutexattr_setpshared... yes
configure: running /bin/sh
../../../../../../../openmpi-1.8.1/ompi/contrib/vt/vt/extlib/otf/configure
'--with-openmpi-inside=1.7'
'--prefix=/casl/vera_ib/gcc-4.8.3/toolset/openmpi-1.8.1'
'--disable-option-checking' 'CFLAGS=-O3 -DNDEBUG -finline-functions
-fno-strict-aliasing -pthread' 'LDFLAGS=
-L/home/fischega/vera_ib_build/scratch/openmpi.BASE/build/ompi/contrib/vt/vt/../../../.libs'
'LIBS=-lrt -lnsl -lutil -lm ' 'CPPFLAGS=-DINSIDE_OPENMPI
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/opal/mca/hwloc/hwloc172/hwloc/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/build/opal/mca/hwloc/hwloc172/hwloc/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/opal/mca/event/libevent2021/libevent
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/opal/mca/event/libevent2021/libevent/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/build/opal/mca/event/libevent2021/libevent/include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/openmpi-1.8.1/ompi/contrib/vt/vt/../../../include
-I/home/fischega/vera_ib_build/scratch/openmpi.BASE/build/ompi/contrib/vt/vt/../../../include'
'CXXFLAGS=-O3 -DNDEBUG -finline-functions -pthread' 'FCFLAGS=' MPICC="gcc"
MPICXX="g++" --with-mpi --with-mpi-lib="-lmpi" --with-zlib-lib="-lz"
--without-zoidfs --includedir="${prefix}/include/vampirtrace"
--docdir="${datarootdir}/doc/otf" --cache-file="/dev/null"
--srcdir="../../../../../../../openmpi-1.8.1/ompi/contrib/vt/vt/extlib/otf"
--disable-option-checking
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking which of CFLAGS are ok for debugger modules... -DNDEBUG
-fno-strict-aliasing -pthread
checking for OPAL CFLAGS... -pthread
checking for OPAL CXXFLAGS... -pthread
checking for ORTE CFLAGS... -pthread
checking for OMPI CFLAGS... -pthread
checking for OMPI CXXFLAGS... -pthread
checking for OMPI FCFLAGS... -pthread
config.status: creating test/threads/Makefile
