Hmm, what CLE release is your development cluster running? It is the value after PrgEnv. Ex. on Cielito we have 4.1.40.
32) PrgEnv-gnu/4.1.40 We have not yet fully tested Open MPI on CLE 5.x.x. -Nathan Hjelm HPC-3, LANL On Tue, Sep 03, 2013 at 10:33:57PM +0000, Teranishi, Keita wrote: > Hi, > > Here is what I put in my PBS script to allocate only single node (I want > to use 16 MPI processes in a single node). > > #PBS -l mppwidth=16 > #PBS -l mppnppn=16 > #PBS -l mppdepth=1 > > Here is the output from aprun (aprun -n 16 -N 16). > Process 2 of 16 is on nid00017 > Process 5 of 16 is on nid00017 > Process 8 of 16 is on nid00017 > Process 12 of 16 is on nid00017 > Process 4 of 16 is on nid00017 > Process 14 of 16 is on nid00017 > Process 0 of 16 is on nid00017 > Process 1 of 16 is on nid00017 > Process 3 of 16 is on nid00017 > Process 13 of 16 is on nid00017 > Process 9 of 16 is on nid00017 > Process 6 of 16 is on nid00017 > Process 11 of 16 is on nid00017 > Process 10 of 16 is on nid00017 > Process 7 of 16 is on nid00017 > Process 15 of 16 is on nid00017 > > > > I am guessing that the CrayXE6 here is different from the others in > production (it is 1 cabinet configuration for code development) and I am > afraid mpirun/mpiexec does wrong instantiation of aprun command. Do I have > to edit the script in contrib? > > > Thanks, > Keita > > On 9/3/13 2:51 PM, "Ralph Castain" <r...@open-mpi.org> wrote: > > >Interesting - and do you have an allocation? If so, what was it - i.e., > >can you check the allocation envar to see if you have 16 slots? > > > > > >On Sep 3, 2013, at 1:38 PM, "Teranishi, Keita" <knte...@sandia.gov> wrote: > > > >> It is what I got. > >> > >> > >>------------------------------------------------------------------------- > >>- > >> There are not enough slots available in the system to satisfy the 16 > >>slots > >> that were requested by the application: > >> /home/knteran/test-openmpi/cpi > >> > >> Either request fewer slots for your application, or make more slots > >> available > >> for use. > >> > >>------------------------------------------------------------------------- > >>- > >> > >> Thanks, > >> Keita > >> > >> > >> > >> On 9/3/13 1:26 PM, "Ralph Castain" <r...@open-mpi.org> wrote: > >> > >>> How does it fail? > >>> > >>> On Sep 3, 2013, at 1:19 PM, "Teranishi, Keita" <knte...@sandia.gov> > >>>wrote: > >>> > >>>> Nathan, > >>>> > >>>> Thanks for the help. I can run a job using openmpi, assigning a > >>>>signle > >>>> process per node. However, I have been failing to run a job using > >>>> multiple MPI ranks in a single node. In other words, "mpiexec > >>>> --bind-to-core --npernode 16 --n 16 ./test" never works (apron -n 16 > >>>> works > >>>> fine). DO you have any thought about it? > >>>> > >>>> Thanks, > >>>> --------------------------------------------- > >>>> Keita Teranishi > >>>> R&D Principal Staff Member > >>>> Scalable Modeling and Analysis Systems > >>>> Sandia National Laboratories > >>>> Livermore, CA 94551 > >>>> > >>>> > >>>> > >>>> > >>>> On 8/30/13 8:49 AM, "Hjelm, Nathan T" <hje...@lanl.gov> wrote: > >>>> > >>>>> Replace install_path to where you want Open MPI installed. > >>>>> > >>>>> ./configure --prefix=install_path > >>>>> --with-platform=contrib/platform/lanl/cray_xe6/optimized-luster > >>>>> make > >>>>> make install > >>>>> > >>>>> To use Open MPI just set the PATH and LD_LIBRARY_PATH: > >>>>> > >>>>> PATH=install_path/bin:$PATH > >>>>> LD_LIBRARY_PATH=install_path/lib:$LD_LIBRARY_PATH > >>>>> > >>>>> You can then use mpicc, mpicxx, mpif90, etc to compile and either > >>>>> mpirun > >>>>> or aprun to run. If you are running at scale I would recommend > >>>>>against > >>>>> using aprun for now. I also recommend you change your programming > >>>>> environment to either PrgEnv-gnu or PrgEnv-intel. The PGI compiler > >>>>>can > >>>>> be > >>>>> a PIA. It is possible to build with the Cray compiler but it takes > >>>>> patching the config.guess and changing some autoconf stuff. > >>>>> > >>>>> -Nathan > >>>>> > >>>>> Please excuse the horrible Outlook-style quoting. > >>>>> ________________________________________ > >>>>> From: users [users-boun...@open-mpi.org] on behalf of Teranishi, > >>>>>Keita > >>>>> [knte...@sandia.gov] > >>>>> Sent: Thursday, August 29, 2013 8:01 PM > >>>>> To: Open MPI Users > >>>>> Subject: Re: [OMPI users] [EXTERNAL] Re: What version of PMI (Cray > >>>>>XE6) > >>>>> is working for OpenMPI-1.6.5? > >>>>> > >>>>> Thanks for the info. Is it still possible to build by myself? What > >>>>>is > >>>>> the procedure other than configure script? > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On 8/23/13 2:37 PM, "Nathan Hjelm" <hje...@lanl.gov> wrote: > >>>>> > >>>>>> On Fri, Aug 23, 2013 at 09:14:25PM +0000, Teranishi, Keita wrote: > >>>>>>> Hi, > >>>>>>> I am trying to install OpenMPI 1.6.5 on Cray XE6 and very curious > >>>>>>> with the > >>>>>>> current support of PMI. In the previous discussions, there was a > >>>>>>> comment > >>>>>>> on the version of PMI (it works with 2.1.4, but fails with 3.0). > >>>>>>> Our > >>>>>> > >>>>>> Open MPI 1.6.5 does not have support for the XE-6. Use 1.7.2 > >>>>>>instead. > >>>>>> > >>>>>>> machine has PMI2.1.4 and PMI4.0 (default). Which version do you > >>>>>> > >>>>>> There was a regression in PMI 3.x.x that still exists in 4.0.x that > >>>>>> causes a warning to be printed on every rank when using mpirun. We > >>>>>>are > >>>>>> working with Cray to resolve the issue. For now use 2.1.4. See the > >>>>>> platform files in contrib/platform/lanl/cray_xe6. The platform files > >>>>>> you > >>>>>> would want to use are debug-lustre or optimized-lusre. > >>>>>> > >>>>>> BTW, 1.7.2 is installed on Cielo and Cielito. Just run: > >>>>>> > >>>>>> module swap PrgEnv-pgi PrgEnv-gnu (PrgEnv-intel also works) > >>>>>> module unload cray-mpich2 xt-libsci > >>>>>> module load openmpi/1.7.2 > >>>>>> > >>>>>> > >>>>>> -Nathan Hjelm > >>>>>> Open MPI Team, HPC-3, LANL > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> us...@open-mpi.org > >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>> > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > >_______________________________________________ > >users mailing list > >us...@open-mpi.org > >http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
