Hi, Here is what I put in my PBS script to allocate only single node (I want to use 16 MPI processes in a single node).
#PBS -l mppwidth=16 #PBS -l mppnppn=16 #PBS -l mppdepth=1 Here is the output from aprun (aprun -n 16 -N 16). Process 2 of 16 is on nid00017 Process 5 of 16 is on nid00017 Process 8 of 16 is on nid00017 Process 12 of 16 is on nid00017 Process 4 of 16 is on nid00017 Process 14 of 16 is on nid00017 Process 0 of 16 is on nid00017 Process 1 of 16 is on nid00017 Process 3 of 16 is on nid00017 Process 13 of 16 is on nid00017 Process 9 of 16 is on nid00017 Process 6 of 16 is on nid00017 Process 11 of 16 is on nid00017 Process 10 of 16 is on nid00017 Process 7 of 16 is on nid00017 Process 15 of 16 is on nid00017 I am guessing that the CrayXE6 here is different from the others in production (it is 1 cabinet configuration for code development) and I am afraid mpirun/mpiexec does wrong instantiation of aprun command. Do I have to edit the script in contrib? Thanks, Keita On 9/3/13 2:51 PM, "Ralph Castain" <r...@open-mpi.org> wrote: >Interesting - and do you have an allocation? If so, what was it - i.e., >can you check the allocation envar to see if you have 16 slots? > > >On Sep 3, 2013, at 1:38 PM, "Teranishi, Keita" <knte...@sandia.gov> wrote: > >> It is what I got. >> >> >>------------------------------------------------------------------------- >>- >> There are not enough slots available in the system to satisfy the 16 >>slots >> that were requested by the application: >> /home/knteran/test-openmpi/cpi >> >> Either request fewer slots for your application, or make more slots >> available >> for use. >> >>------------------------------------------------------------------------- >>- >> >> Thanks, >> Keita >> >> >> >> On 9/3/13 1:26 PM, "Ralph Castain" <r...@open-mpi.org> wrote: >> >>> How does it fail? >>> >>> On Sep 3, 2013, at 1:19 PM, "Teranishi, Keita" <knte...@sandia.gov> >>>wrote: >>> >>>> Nathan, >>>> >>>> Thanks for the help. I can run a job using openmpi, assigning a >>>>signle >>>> process per node. However, I have been failing to run a job using >>>> multiple MPI ranks in a single node. In other words, "mpiexec >>>> --bind-to-core --npernode 16 --n 16 ./test" never works (apron -n 16 >>>> works >>>> fine). DO you have any thought about it? >>>> >>>> Thanks, >>>> --------------------------------------------- >>>> Keita Teranishi >>>> R&D Principal Staff Member >>>> Scalable Modeling and Analysis Systems >>>> Sandia National Laboratories >>>> Livermore, CA 94551 >>>> >>>> >>>> >>>> >>>> On 8/30/13 8:49 AM, "Hjelm, Nathan T" <hje...@lanl.gov> wrote: >>>> >>>>> Replace install_path to where you want Open MPI installed. >>>>> >>>>> ./configure --prefix=install_path >>>>> --with-platform=contrib/platform/lanl/cray_xe6/optimized-luster >>>>> make >>>>> make install >>>>> >>>>> To use Open MPI just set the PATH and LD_LIBRARY_PATH: >>>>> >>>>> PATH=install_path/bin:$PATH >>>>> LD_LIBRARY_PATH=install_path/lib:$LD_LIBRARY_PATH >>>>> >>>>> You can then use mpicc, mpicxx, mpif90, etc to compile and either >>>>> mpirun >>>>> or aprun to run. If you are running at scale I would recommend >>>>>against >>>>> using aprun for now. I also recommend you change your programming >>>>> environment to either PrgEnv-gnu or PrgEnv-intel. The PGI compiler >>>>>can >>>>> be >>>>> a PIA. It is possible to build with the Cray compiler but it takes >>>>> patching the config.guess and changing some autoconf stuff. >>>>> >>>>> -Nathan >>>>> >>>>> Please excuse the horrible Outlook-style quoting. >>>>> ________________________________________ >>>>> From: users [users-boun...@open-mpi.org] on behalf of Teranishi, >>>>>Keita >>>>> [knte...@sandia.gov] >>>>> Sent: Thursday, August 29, 2013 8:01 PM >>>>> To: Open MPI Users >>>>> Subject: Re: [OMPI users] [EXTERNAL] Re: What version of PMI (Cray >>>>>XE6) >>>>> is working for OpenMPI-1.6.5? >>>>> >>>>> Thanks for the info. Is it still possible to build by myself? What >>>>>is >>>>> the procedure other than configure script? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 8/23/13 2:37 PM, "Nathan Hjelm" <hje...@lanl.gov> wrote: >>>>> >>>>>> On Fri, Aug 23, 2013 at 09:14:25PM +0000, Teranishi, Keita wrote: >>>>>>> Hi, >>>>>>> I am trying to install OpenMPI 1.6.5 on Cray XE6 and very curious >>>>>>> with the >>>>>>> current support of PMI. In the previous discussions, there was a >>>>>>> comment >>>>>>> on the version of PMI (it works with 2.1.4, but fails with 3.0). >>>>>>> Our >>>>>> >>>>>> Open MPI 1.6.5 does not have support for the XE-6. Use 1.7.2 >>>>>>instead. >>>>>> >>>>>>> machine has PMI2.1.4 and PMI4.0 (default). Which version do you >>>>>> >>>>>> There was a regression in PMI 3.x.x that still exists in 4.0.x that >>>>>> causes a warning to be printed on every rank when using mpirun. We >>>>>>are >>>>>> working with Cray to resolve the issue. For now use 2.1.4. See the >>>>>> platform files in contrib/platform/lanl/cray_xe6. The platform files >>>>>> you >>>>>> would want to use are debug-lustre or optimized-lusre. >>>>>> >>>>>> BTW, 1.7.2 is installed on Cielo and Cielito. Just run: >>>>>> >>>>>> module swap PrgEnv-pgi PrgEnv-gnu (PrgEnv-intel also works) >>>>>> module unload cray-mpich2 xt-libsci >>>>>> module load openmpi/1.7.2 >>>>>> >>>>>> >>>>>> -Nathan Hjelm >>>>>> Open MPI Team, HPC-3, LANL >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >_______________________________________________ >users mailing list >us...@open-mpi.org >http://www.open-mpi.org/mailman/listinfo.cgi/users
