Josh and I spent some quality time on the phone today talking through
this issue. Here's some points (not in strict order):
- "--enable-static --enable-shared" (either explicit or implicit)
works fine -- it'll build both static and shared library versions of
OMPI's libraries. Unless otherwise specified, the OMPI executables
will link against the shared libraries, but Libtool typically -
rpath's in where the libraries build, so there's no need to set
LD_LIBRARY_PATH to run them.
- It turns out that there is a critical app at Sandia that *requires*
-static to build (those crazy app writers!). Sandia does have all
the right .so libraries installed on all their nodes, so the warnings
about dlopen() and gethostname() are all ok.
- Sandia also does have all the Right .a libraries installed (e.g.,
libsysfs.a, libdl.a, libc.a, libtorque.a, libibverbs.a, ...etc.). So
everything *should* compile properly to get a fully static executable.
- Because horrid details that no one cares about, you can't compile
MPI apps with -static *and* have Open MPI have an internal memory
manager. So it is necessary to configure OMPI with --without-memory-
manager. This is actually ok in OS-bypass network environments
because OMPI defaults to never returning NIC-registered memory to the
user (and therefore you don't need a memory manager). If you want a
memory manager, then you can't compile your MPI apps with -static;
sorry, this is a limitation of Linux.
--> Configuring OMPI with --without-memory-manager resolved all the
duplicate symbol problems.
- OMPI is not correctly determining that libsysfs is a dependency (I
think because it's an implicit dependency from libibverbs). Hence,
compiling MPI applications with the wrapper results in missing
symbols because the wrapper is not adding -lsysfs to the command
line. The workaround is to edit $prefix/share/openmpi/
mpi*wrapper.txt to add "-lsysfs" to the "libs" field (these files are
what the wrapper compilers read to know how to build command lines
for the underlying compiler).
--> Hand-editing mpi*wrapper.txt to insert "-lsysfs" fixed the
missing symbols problem.
End of the day: Sandia was able to get a 100% static MPI application
executable, although ld still emits a bunch of warnings (which are ok).
=================
This is probably FAQ fodder; I have some time on airplanes coming
up. I'll morph this material into stuff that is suitable for the FAQ.
On Oct 16, 2006, at 2:03 AM, Josh England wrote:
On Sun, 2006-10-15 at 23:31 -0600, Josh England wrote:
On Sat, 2006-10-14 at 09:35 -0400, Jeff Squyres wrote:
Given the messages from the linker and my prior mail, I'm not sure
that a) is possible -- it will require more poking around to know
for
sure. b) is definitely possible simply by using --enable-static and
and then using "mpicc" (and friends) as normal.
OMPI will be built statically (to include torque and libibverbs,
assuming you have .a's versions of both of those), but as Scott
mentioned, system libraries such as dl and sysfs will be linked
dynamically.
OK. I've got a build with --enable-static and --disable-shared,
and I'm
still seeing the *exact* same behavior. I cannot build a static
library
either with mpicc or specifying a manual link line. Note that this
appears to be a problem only with 1.1.2_rc4. Open MPI version 1.1.1
seems to work.
Sorry. I wrote too fast. Version 1.1.1 actually *does not* work for
static builds either. It exhibits the same problem described in the
original post.
-JE
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems
