I honestly haven't thought through the ramifications of doing a multi-
lib build with OPAL_PREFIX et al. :-\
If you setenv OPAL_LIBDIR, it'll use whatever you set it to, so it
doesn't matter what you configured --libdir with. Additionally mca/
installdirs/config/install_dirs.h has this by default:
#define OPAL_LIBDIR "${exec_prefix}/lib"
Hence, if you use a default --libdir and setenv OPAL_PREFIX, then the
libdir should pick up the right thing (because it's based on the
prefix). But if you use --libdir that is *not* based on $
{exec_prefix}, then you might run into problems.
Perhaps you can '--libdir="${exec_prefix}/lib64"' so that you can have
your custom libdir, but still have it dependent upon the prefix that
gets expanded at run time...?
(again, I'm not thinking all of this through -- just offering a few
suggestions off the top of my head that you'll need to test / trace
the code to be sure...)
On Jan 5, 2009, at 1:35 PM, Ethan Mallove wrote:
On Thu, Dec/25/2008 08:12:49AM, Jeff Squyres wrote:
It's quite possible that we don't handle this situation properly.
Won't
you need to libdir's (one for the 32 bit OMPI executables, and one
for the
64 bit MPI apps)?
I don't need an OPAL environment variable for the executables, just a
single OPAL_LIBDIR var for the libraries. (One set of 32-bit
executables runs with both 32-bit and 64-bit libraries.) I'm guessing
OPAL_LIBDIR will not work for you if you configure with a non-standard
--libdir option.
-Ethan
On Dec 23, 2008, at 3:58 PM, Ethan Mallove wrote:
I think the problem is that I am doing a multi-lib build. I have
32-bit libraries in lib/, and 64-bit libraries in lib/64. I assume I
do not see the issue for 32-bit tests, because all the dependencies
are where Open MPI expects them to be. For the 64-bit case, I tried
setting OPAL_LIBDIR to /opt/openmpi-relocated/lib/lib64, but no
luck.
Given the below configure arguments, what do my OPAL_* env vars need
to be? (Also, could using --enable-orterun-prefix-by-default
interfere
with OPAL_PREFIX?)
$ ./configure CC=cc CXX=CC F77=f77 FC=f90 --with-openib
--without-udapl --disable-openib-ibcm --enable-heterogeneous
--enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by-
default
--with-sge --enable-mpi-f90 --with-mpi-f90-size=small
--disable-mpi-threads --disable-progress-threads --disable-debug
CFLAGS="-m32 -xO5" CXXFLAGS="-m32 -xO5" FFLAGS="-m32 -xO5"
FCFLAGS="-m32
-xO5"
--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include
--without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-
install32
--with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-
scratch/burl-ct-v!
20z-12/ompi-tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include
LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib"
$ ./confgiure CC=cc CXX=CC F77=f77 FC=f90 --with-openib
--without-udapl --disable-openib-ibcm --enable-heterogeneous
--enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by-
default
--with-sge --enable-mpi-f90 --with-mpi-f90-size=small
--disable-mpi-threads --disable-progress-threads --disable-debug
CFLAGS="-m64 -xO5" CXXFLAGS="-m64 -xO5" FFLAGS="-m64 -xO5"
FCFLAGS="-m64
-xO5"
--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib/lib64
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include/64
--without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-
install64
--with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-
scratch/!
burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib/lib64
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include/64
LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib"
--disable-binaries
-Ethan
On Dec 22, 2008, at 12:42 PM, Ethan Mallove wrote:
Can anyone get OPAL_PREFIX to work on Linux? A simple test is to
see
if the following works for any mpicc/mpirun:
$ mv <openmpi-installation> /tmp/foo
$ set OPAL_PREFIX /tmp/foo
$ mpicc ...
$ mpirun ...
If you are able to get the above to run successfully, I'm
interested
in your config.log file.
Thanks,
Ethan
On Thu, Dec/18/2008 11:03:25AM, Ethan Mallove wrote:
Hello,
The below FAQ lists instructions on how to use a relocated Open
MPI
installation:
http://www.open-mpi.org/faq/?category=building#installdirs
On Solaris, OPAL_PREFIX and friends (documented in the FAQ)
work for
me with both MPI (hello_c) and non-MPI (hostname) programs. On
Linux,
I can only get the non-MPI case to work. Here are the environment
variables I am setting:
$ cat setenv_opal_prefix.csh
set opal_prefix = "/opt/openmpi-relocated"
setenv OPAL_PREFIX $opal_prefix
setenv OPAL_BINDIR $opal_prefix/bin
setenv OPAL_SBINDIR $opal_prefix/sbin
setenv OPAL_DATAROOTDIR $opal_prefix/share
setenv OPAL_SYSCONFDIR $opal_prefix/etc
setenv OPAL_SHAREDSTATEDIR $opal_prefix/com
setenv OPAL_LOCALSTATEDIR $opal_prefix/var
setenv OPAL_LIBDIR $opal_prefix/lib
setenv OPAL_INCLUDEDIR $opal_prefix/include
setenv OPAL_INFODIR $opal_prefix/info
setenv OPAL_MANDIR $opal_prefix/man
setenv PATH $opal_prefix/bin:$PATH
setenv LD_LIBRARY_PATH $opal_prefix/lib:$opal_prefix/lib/64
Here is the error I get:
$ mpirun -np 2 hello_c
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel
process
is
likely to abort. There are many reasons that a parallel
process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal
failure;
here's some additional information (which may only be relevant
to an
Open MPI developer):
opal_carto_base_select failed
--> Returned value -13 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
[burl-ct-v20z-0:27737] [[INVALID],INVALID] ORTE_ERROR_LOG: Not
found in
file runtime/orte_init.c at line 77
Any ideas on what's going on?
Thanks,
Ethan
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
