Thank you for your response!
Just to clarify some things for my understanding:
Do users load a single module that specifies both compiler and mpi
version (as opposed to loading two different modules, one for
complier, and one for mpi)?
So, if the MPI implementations need to be built separately for each
compiler, what about other software packages that users' link against?
For example, as far as I know, there are not seperate versions of
libc for each compiler (but I may be ignorant here....please correct
me if I'm wrong). How about other packages that code #includes? How
do I know when something needs to be compiled separately vs one
version for everyone?
Another thing that I'm still not quite clear on: Can I have all
versions of the various MPI libraries in the ld path at once (so the
user doesn't have to "select" the correct one somehow prior to code
execution)?
Are there any tricks to building/installing all this stuff in a fairly
automated manner?
Thanks again!
--Jim
On Nov 15, 2007 9:13 AM, Katherine Holcomb <kholc...@virginia.edu> wrote:
> We have almost exactly the situation you describe with our clusters. I'm
> not the system administrator but I am the one who actually writes the module
> scripts.
>
> It is best to compile OpenMPI (and any other MPI) with each compiler
> separately; this is especially necessary for the Fortran and C++ bindings.
> We simply have a directory layout like
>
> /opt/openmpi/intel
> /opt/openmpi/pgi
> /opt/openmpi/gnu
>
> etc.
>
> To compile OpenMPI with a different compiler, you can use environment
> variables such as (for the Intel compiler)
> FC=ifort
> F90=ifort
> CC=icc
> CXX=icpc
> You set these within the %build part of the rpm spec file.
>
> Similarly for MPICH2. (I would recommend MPICH2 over MPICH1 at this point.)
>
> Users load a module to select a compiler+MPI combination. Loading the
> appropriate module sets the path for the version of mpicc/cxx/f90 chosen by
> the user. That in turn knows where its associated libraries live.
>
> Most users link MPI libraries statically. If dynamic libraries are needed
> then they can be added to LD_LIBRARY_PATH by the module script.
>
> I am away from my office until after the Thanksgiving holiday but if you
> email me personally
> then, I can send you some sample module scripts. I can also send you some
> sample spec files for the rpms we use.
>
>
> ----- Original Message -----
> From: "Jim Kusznir" <jkusz...@gmail.com>
> To: <us...@open-mpi.org>
> Sent: Thursday, November 15, 2007 11:54 AM
> Subject: [OMPI users] Suggestions on multi-compiler/multi-mpi build?
>
>
> > Hi all:
> >
> > I'm trying to set up a cluster for a group of users with very
> > different needs. So far, it looks like I need gcc, pgi, and intel to
> > work with openmpi and mpich, with each user able to control what
> > combination they get. This is turning out to be much more difficult
> > than I expected.
> >
> > Someone has pointed me to enviornment-modules ("Modules"), which looks
> > like it will be a critical part of the solution. I even noticed that
> > the provided openmpi.spec file has some direct support for modules,
> > which makes me quite happy!
> >
> > However, I still have many questions about how to set things up.
> >
> > First, I get the impression that openmpi will need to be compiled with
> > each compiler that will use it. If this is true, I'm not quite sure
> > how to go about it. I could install in different directories for the
> > user commands, but what about the libraries? I don't think I have a
> > feesable way of selecting which library to use on the fly on the
> > entire cluster for each user, so it seems like it would be better to
> > have all the libraries available. In addition, I will need RPMs to
> > deploy efficiently on the cluster. I suspect I can change the
> > versioning info and build with each compiler, but at this point, I
> > don't even know how to reliably select what compiler rpmbuild will use
> > (I've only succeeded in using gcc).
> >
> > Finally, using modules, how do I set it up so that if a user changes
> > compilers, but stays with openmpi, it will load the correct openmpi
> > paths? I know I can set up the openmpi module file to load after the
> > compiler module and based on that select different paths based on the
> > currently-loaded compiler module. If the user changes the compiler
> > module, will that cause the mpi module to also be reloaded so the new
> > settings will be loaded? Or do I need this at all?
> >
> > Thanks for all your help!
> > --Jim
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
