On Mar 24, 2006, at 6:43 PM, Troy Telford wrote:
On Fri, 24 Mar 2006 16:30:37 -0700, Brian Barrett <brbarret@open-
mpi.org>
wrote:
The freely available derivatives of PBS all
seemed more than happy to help with building the tm libraries with -
fPIC.
It's a pity that I'm not in a position where I don't have to deal
with PBS
Professional.
Still, thanks for the info; I guess when PBS Pro is there, there's
only
one option...
Unfortunately, I think so. We have the same problem on some of our
University-wide systems that use PBS Pro. Thankfully, just one
system is x86_64, so not a big deal. But still less than ideal (if
only because I usually only remember to add the --enable-static --
disable-shared after the build aborts).
One thought that occurs to me that *might* work, but would be really
icky, would be to build a complete build of Open MPI (binaries and
all) with --enable-static --disable-shared and with TM. This means
all the *Open MPI* binaries (mpirun, the wrapper compilers) will be
built statically and not be able load components. Then do another
build with --enable-shared, without tm, but all the other flags the
same. Just install the headers and libraries for this build. This
build will be able to load components, since it's shared. Since
Linux prefers shared libraries over static libraries, the user codes
will be linked against the shared libraries and able to be load
dynamic modules.
I think this might break MPI_COMM_SPAWN, which is unfortunately (more
to the point, MPI_COMM_SPAWN would not use TM to start new procs).
But it would allow loadable modules for things like the point-to-
point network support, which I'm guessing is what you're really after.
I'm not sure we could do something like this internally in Open MPI's
build system and we definitely don't have the developer time to look
into it in the foreseeable future.
Thinking about it some more, there is one even more icky solution to
the problem. Again, this is not something the Open MPI development
team can put cycles into. Last I heard / looked, the TM interface
and the wire protocol to the pbs_mom hadn't changed much in years and
years. There's an open source version of PBS out there that includes
code to implement this wire protocol. There's no reason Open MPI pls
and ras components couldn't be developed that talked directly to the
pbs_mom rather than using the published API. Nasty, yes. But it
might work. And since the PBS Pro libraries wouldn't need to be
linked in, the -fPIC problem is avoided.
If you're interested in having a stab at either work-around approach,
let me know (might be easiest to post to the devel mailing list so
that we don't bore all the users with the awful details). I'd be
interested in seeing if something couldn't be done between now and
when Altair releases PBS Pro libraries for the Opteron compiled with -
fPIC.
Brian
--
Brian Barrett
Open MPI developer
http://www.open-mpi.org/
