Hi David,
Hmm, your demo is well-chosen and crystal-clear, yet the output is unexpected.
I do not see environment vars passed by default here:
login3$ qsub -l nodes=2:ppn=1 -I
qsub: waiting for job 34683.mds01 to start
qsub: job 34683.mds01 ready
n102$ mpirun -n 2 -machinefile $PBS_NODEFILE hostname
n102
n085
n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO
n102$ export FOO=BAR
n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO
FOO=BAR
n102$ type mpirun
mpirun is hashed (/opt/soft/openmpi-1.3.2-intel10-1/bin/mpirun)
Curious, what do you get upon:
where mpirun
I built OpenMPI-1.3.2 here from source with:
CC=icc CXX=icpc FC=ifort F77=ifort \
LDFLAGS='-Wl,-z,noexecstack' \
CFLAGS='-O2 -g -fPIC' \
CXXFLAGS='-O2 -g -fPIC' \
FFLAGS='-O2 -g -fPIC' \
./configure --prefix=$prefix \
--with-libnuma=/usr \
--with-openib=/usr \
--with-udapl \
--enable-mpirun-prefix-by-default \
--without-tm
I did't find the behavior I saw strange, given that orterun(1) talks only about
$OPMI_* and inheritance from the remote shell. It also mentions a "boot MCA
module", about which I couldn't find much on open-mpi.org - hmm.
In the meantime, I did find a possible solution, namely, to tell ssh to pass a
variable using SendEnv/AcceptEnv. That variable is then seen by and can be
interpreted (cautiously) in /etc/profile.d/ scripts. A user could set it in
the job file (or even qalter it post submission):
#PBS -v VARNAME=foo:bar:baz
For VARNAME, I think simply "MODULES" or "EXTRAMODULES" could do.
With best regards,
Michael
On Nov 17, 2009, at 4:29 , David Singleton wrote:
>
> I'm not sure why you dont see Open MPI behaving like other MPI's w.r.t.
> modules/environment on remote MPI tasks - we do.
>
> xe:~ > qsub -q express -lnodes=2:ppn=8,walltime=10:00,vmem=2gb -I
> qsub: waiting for job 376366.xepbs to start
> qsub: job 376366.xepbs ready
>
> [dbs900@x27 ~]$ module load openmpi
> [dbs900@x27 ~]$ mpirun -n 2 --bynode hostname
> x27
> x28
> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO
> [dbs900@x27 ~]$ setenv FOO BAR
> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO
> FOO=BAR
> FOO=BAR
> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber
> [dbs900@x27 ~]$ module load amber
> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber
> LOADEDMODULES=openmpi/1.3.3:amber/9
> PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin::/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe
> _LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9
> AMBERHOME=/apps/amber/9
> LOADEDMODULES=openmpi/1.3.3:amber/9
> PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin:/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe
> _LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9
> AMBERHOME=/apps/amber/9
>
> David
>
>
> Michael Sternberg wrote:
>> Dear readers,
>> With OpenMPI, how would one go about requesting to load environment modules
>> (of the http://modules.sourceforge.net/ kind) on remote nodes, augmenting
>> those normally loaded there by shell dotfiles?
>> Background:
>> I run a RHEL-5/CentOS-5 cluster. I load a bunch of default modules through
>> /etc/profile.d/ and recommend to users to customize modules in ~/.bashrc. A
>> problem arises for PBS jobs which might need job-specific modules, e.g., to
>> pick a specific flavor of an application. With other MPI implementations
>> (ahem) which export all (or judiciously nearly all) environment variables by
>> default, you can say:
>> #PBS ...
>> module load foo # not for OpenMPI
>> mpirun -np 42 ... \
>> bar-app
>> Not so with OpenMPI - any such customization is only effective for processes
>> on the master (=local) node of the job, and any variables changed by a given
>> module would have to be specifically passed via mpirun -x VARNAME. On the
>> remote nodes, those variables are not available in the dotfiles because they
>> are passed only once orted is live (after dotfile processing by the shell),
>> which then immediately spawns the application binaries (right?)
>> I thought along the following lines:
>> (1) I happen to run Lustre, which would allow writing a file coherently
>> across nodes prior to mpirun, and thus hook into the shell dotfile
>> processing, but that seems rather crude.
>> (2) "mpirun -x PATH -x LD_LIBRARY_PATH …" would take care of a lot, but is
>> not really general.
>> Is there a recommended way?
>> regards,
>> Michael
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
