Hi, On Nov 17, 2009, at 9:10 , Ralph Castain wrote: > Not exactly. It completely depends on how Torque was setup - OMPI isn't > forwarding the environment. Torque is.
I actually tried compiling OMPI with the tm interface a couple of versions back for both packages but ran into memory trouble, which is why I didn't pursue this. With torque-2.4.x out and OpenMPI getting close to 1.3.4 I'll try again. > We made a design decision at the very beginning of the OMPI project not to > forward non-OMPI envars unless directed to do so by the user. I'm afraid I > disagree with Michael's claim that other MPIs do forward them - yes, MPICH > does, but not all others do. > > The world is bigger than MPICH and OMPI :-) Yup, I saw your message from just last month http://www.open-mpi.org/community/lists/users/2009/10/10994.php ; I didn't mean to make a global claim :-) I'm aware that exporting environment variables (including $PWD) under MPI is implementation dependent. I just happened to have MPICH, Intel MPI (same roots), and OpenMPI on my cluster. > First, if you are using a managed environment like Torque, we recommend that > you work with your sys admin to decide how to configure it. This is the best > way to resolve a problem. Yeah, I wish that guy would know better and not have to ask around mailing lists :-) > Second, if you are not using a managed environment and/or decide not to have > that environment do the forwarding, you can tell OMPI to forward the envars > you need by specifying them via the -x cmd line option. We already have a > request to expand this capability, and I will be doing so as time permits. > One option I'll be adding is the reverse of -x - i.e., "forward all envars > -except- the specified one(s)". The issue with -x is that modules may set any random variable. The reverse option to -x would be great of course. MPICH2 and Intel MPI pass all but a few (known to be host-specific) variables by default, and counter that with "none" and "all" options. Thanks! Michael > HTH > ralph > > On Nov 17, 2009, at 5:55 AM, David Singleton wrote: > >> >> I can see the difference - we built Open MPI with tm support. For some >> reason, I thought mpirun fed its environment to orted (after orted is >> launched) so orted can pass it on to MPI tasks. That should be portable >> between different launch mechanisms. But it looks like tm launches >> orted with the full mpirun environment (at the request of mpirun). >> >> Cheers, >> David >> >> >> Michael Sternberg wrote: >>> Hi David, >>> Hmm, your demo is well-chosen and crystal-clear, yet the output is >>> unexpected. I do not see environment vars passed by default here: >>> login3$ qsub -l nodes=2:ppn=1 -I >>> qsub: waiting for job 34683.mds01 to start >>> qsub: job 34683.mds01 ready >>> n102$ mpirun -n 2 -machinefile $PBS_NODEFILE hostname >>> n102 >>> n085 >>> n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO >>> n102$ export FOO=BAR >>> n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO >>> FOO=BAR >>> n102$ type mpirun >>> mpirun is hashed (/opt/soft/openmpi-1.3.2-intel10-1/bin/mpirun) >>> Curious, what do you get upon: >>> where mpirun >>> I built OpenMPI-1.3.2 here from source with: >>> CC=icc CXX=icpc FC=ifort F77=ifort \ >>> LDFLAGS='-Wl,-z,noexecstack' \ >>> CFLAGS='-O2 -g -fPIC' \ >>> CXXFLAGS='-O2 -g -fPIC' \ >>> FFLAGS='-O2 -g -fPIC' \ >>> ./configure --prefix=$prefix \ >>> --with-libnuma=/usr \ >>> --with-openib=/usr \ >>> --with-udapl \ >>> --enable-mpirun-prefix-by-default \ >>> --without-tm >>> I did't find the behavior I saw strange, given that orterun(1) talks only >>> about $OPMI_* and inheritance from the remote shell. It also mentions a >>> "boot MCA module", about which I couldn't find much on open-mpi.org - hmm. >>> In the meantime, I did find a possible solution, namely, to tell ssh to >>> pass a variable using SendEnv/AcceptEnv. That variable is then seen by and >>> can be interpreted (cautiously) in /etc/profile.d/ scripts. A user could >>> set it in the job file (or even qalter it post submission): >>> #PBS -v VARNAME=foo:bar:baz >>> For VARNAME, I think simply "MODULES" or "EXTRAMODULES" could do. >>> With best regards, >>> Michael >>> On Nov 17, 2009, at 4:29 , David Singleton wrote: >>>> I'm not sure why you dont see Open MPI behaving like other MPI's w.r.t. >>>> modules/environment on remote MPI tasks - we do. >>>> >>>> xe:~ > qsub -q express -lnodes=2:ppn=8,walltime=10:00,vmem=2gb -I >>>> qsub: waiting for job 376366.xepbs to start >>>> qsub: job 376366.xepbs ready >>>> >>>> [dbs900@x27 ~]$ module load openmpi >>>> [dbs900@x27 ~]$ mpirun -n 2 --bynode hostname >>>> x27 >>>> x28 >>>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO >>>> [dbs900@x27 ~]$ setenv FOO BAR >>>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO >>>> FOO=BAR >>>> FOO=BAR >>>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber >>>> [dbs900@x27 ~]$ module load amber >>>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber >>>> LOADEDMODULES=openmpi/1.3.3:amber/9 >>>> PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin::/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe >>>> _LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9 >>>> AMBERHOME=/apps/amber/9 >>>> LOADEDMODULES=openmpi/1.3.3:amber/9 >>>> PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin:/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe >>>> _LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9 >>>> AMBERHOME=/apps/amber/9 >>>> >>>> David >>>> >>>> >>>> Michael Sternberg wrote: >>>>> Dear readers, >>>>> With OpenMPI, how would one go about requesting to load environment >>>>> modules (of the http://modules.sourceforge.net/ kind) on remote nodes, >>>>> augmenting those normally loaded there by shell dotfiles? >>>>> Background: >>>>> I run a RHEL-5/CentOS-5 cluster. I load a bunch of default modules >>>>> through /etc/profile.d/ and recommend to users to customize modules in >>>>> ~/.bashrc. A problem arises for PBS jobs which might need job-specific >>>>> modules, e.g., to pick a specific flavor of an application. With other >>>>> MPI implementations (ahem) which export all (or judiciously nearly all) >>>>> environment variables by default, you can say: >>>>> #PBS ... >>>>> module load foo # not for OpenMPI >>>>> mpirun -np 42 ... \ >>>>> bar-app >>>>> Not so with OpenMPI - any such customization is only effective for >>>>> processes on the master (=local) node of the job, and any variables >>>>> changed by a given module would have to be specifically passed via mpirun >>>>> -x VARNAME. On the remote nodes, those variables are not available in >>>>> the dotfiles because they are passed only once orted is live (after >>>>> dotfile processing by the shell), which then immediately spawns the >>>>> application binaries (right?) >>>>> I thought along the following lines: >>>>> (1) I happen to run Lustre, which would allow writing a file coherently >>>>> across nodes prior to mpirun, and thus hook into the shell dotfile >>>>> processing, but that seems rather crude. >>>>> (2) "mpirun -x PATH -x LD_LIBRARY_PATH …" would take care of a lot, but >>>>> is not really general. >>>>> Is there a recommended way? >>>>> regards, >>>>> Michael >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> -- >> -------------------------------------------------------------------------- >> Dr David Singleton ANU Supercomputer Facility >> HPC Systems Manager and NCI National Facility >> david.single...@anu.edu.au Leonard Huxley Bldg (No. 56) >> Phone: +61 2 6125 4389 Australian National University >> Fax: +61 2 6125 8199 Canberra, ACT, 0200, Australia >> -------------------------------------------------------------------------- >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
