On Mar 15, 2007, at 5:02 PM, Michael wrote:
I would like to hear just how portable an executable compiled against
OpenMPI shared libraries should be.
This is a hard question to answer:
1. We have not done this explicit kind of testing.
2. Open MPI's libmpi.so itself is plain vanilla C. If you have an
application that is already portable, linking it against Open MPI
should not cause it to be less portable.
3. Open MPI, however, can use many support libraries (e.g., libmosal
in your previous mail). This myriad of extra libraries may create
difficulties in creating a truly portable application.
The best practices that I have seen have been:
- start with an application that itself is already portable (without
MPI)
- compile everything 100% static
But this has drawbacks as well -- consider if you link in libmosal.a
to your MPI application and then take it to another system that has a
slightly different version of VAPI (e.g., a kernel interface
changed). Although your application will load and start running
(i.e., no runtime linker resolution failures), it may fail in
unpredictable ways later because the libmosal.a in your application
calls down to the kernel in ways that are unsupported by the VAPI/
libmosal on the current system.
Make sense?
This is unfortunately not an MPI (or Open MPI) specific issue; it's a
larger problem of creating truly portable software. To have a better
chance of success, you probably want to ensure that all relevant
points of interaction between your application and the outside system
are either the same version or "compatible enough".
- high-speed networking support libraries
- resource manager support libraries
- libc
- ...etc.
Specifically, even though you won't be looking for .so's at runtime,
you need to ensure that the way the .a's compiled into your
application interact with the system is either the same way or "close
enough" to how the corresponding support libraries work on the target
machine.
All this being said, Open MPI did try to take steps in its design to
be able to effect more portability (e.g., for ISV's). Theoretically
-- we have not explicitly tested this -- the following setup may
provide a better degree of portability:
- have the same version of Open MPI available on each machine,
compiled against whatever support libraries are relevant on that
machine (using plugins, not --enable-static).
- compile your application *dynamically* against Open MPI. Note that
some of the upper-level configuration of Open MPI must be either the
same or "close enough" between machines such that runtime linking
will work properly (e.g., don't use a 32 bit libmpi on one machine
and a 64 bit libmpi on another, etc. There's more details here, but
you get the general idea)
- ensure that other (non-MPI-related) interaction points in your
application are the same or "close enough" to be portable
By linking dynamically against Open MPI (which is plain vanilla C),
the application will only be looking for Open MPI's plain C support
libraries -- not the other support libraries (such as libmosal),
because those are linked against OMPI's plugins -- not libmpi.so
(etc.). This design effectively takes MPI out of the portability
equation.
That's the theory, anyway. :-)
I skipped many nit-picky details, so I'm sure there will be issues to
figure out. But *in theory*, it's possible...
I'm compiling on a Debian Linux system with dual 1.3 GHz AMD Opterons
per node and an internal network of dual gigabit ethernet.
I'm planning on a SUSE Linux Enterprise Server 9 system with dual 3.6
GHz Intel Xeon EM64T per node and an internal network using Myrinet.
I can't speak for how portable Myrinet support libraries are...
Myricom?
--
Jeff Squyres
Cisco Systems
