Here is the module environment, and I allocate interactive node by "qsub -I -lmppwidth=32 -lmppnppn=16." module list Currently Loaded Modulefiles: 1) modules/3.2.6.7 2) craype-network-gemini 3) cray-mpich2/5.6.4 4) atp/1.6.3 5) xe-sysroot/4.1.40 6) switch/1.0-1.0401.36779.2.72.gem 7) shared-root/1.0-1.0401.37253.3.50.gem 8) pdsh/2.26-1.0401.37449.1.1.gem 9) nodehealth/5.0-1.0401.38460.12.18.gem 10) lbcd/2.1-1.0401.35360.1.2.gem 11) hosts/1.0-1.0401.35364.1.115.gem 12) configuration/1.0-1.0401.35391.1.2.gem 13) ccm/2.2.0-1.0401.37254.2.142 14) audit/1.0.0-1.0401.37969.2.32.gem 15) rca/1.0.0-2.0401.38656.2.2.gem 16) dvs/1.8.6_0.9.0-1.0401.1401.1.120 17) csa/3.0.0-1_2.0401.37452.4.50.gem 18) job/1.5.5-0.1_2.0401.35380.1.10.gem 19) xpmem/0.1-2.0401.36790.4.3.gem 20) gni-headers/2.1-1.0401.5675.4.4.gem 21) dmapp/3.2.1-1.0401.5983.4.5.gem 22) pmi/4.0.1-1.0000.9421.73.3.gem 23) ugni/4.0-1.0401.5928.9.5.gem 24) udreg/2.3.2-1.0401.5929.3.3.gem 25) xt-libsci/12.0.00 26) xt-totalview/8.12.0 27) totalview-support/1.1.5 28) gcc/4.7.2 29) xt-asyncpe/5.22 30) eswrap/1.0.8 31) craype-mc8 32) PrgEnv-gnu/4.1.40 33) moab/5.4.4
In interactive mode (as well as batch mode), "aprun –np 32" can run my OpenMPI code. aprun -n 32 ./cpi Process 5 of 32 is on nid00015 Process 7 of 32 is on nid00015 Process 2 of 32 is on nid00015 Process 0 of 32 is on nid00015 Process 13 of 32 is on nid00015 Process 10 of 32 is on nid00015 Process 3 of 32 is on nid00015 Process 1 of 32 is on nid00015 Process 6 of 32 is on nid00015 Process 4 of 32 is on nid00015 Process 15 of 32 is on nid00015 Process 9 of 32 is on nid00015 Process 12 of 32 is on nid00015 Process 8 of 32 is on nid00015 Process 11 of 32 is on nid00015 Process 14 of 32 is on nid00015 Process 29 of 32 is on nid00014 Process 22 of 32 is on nid00014 Process 17 of 32 is on nid00014 Process 28 of 32 is on nid00014 Process 31 of 32 is on nid00014 Process 26 of 32 is on nid00014 Process 30 of 32 is on nid00014 Process 16 of 32 is on nid00014 Process 25 of 32 is on nid00014 Process 24 of 32 is on nid00014 Process 21 of 32 is on nid00014 Process 20 of 32 is on nid00014 Process 27 of 32 is on nid00014 Process 19 of 32 is on nid00014 Process 18 of 32 is on nid00014 Process 23 of 32 is on nid00014 pi is approximately 3.1415926544231265, Error is 0.0000000008333334 wall clock time = 0.004645 Here is what I have with openmpi. mpiexec --bind-to-core --mca plm_base_strip_prefix_from_node_names 0 -np 32 ./cpi -------------------------------------------------------------------------- There are not enough slots available in the system to satisfy the 32 slots that were requested by the application: ./cpi Either request fewer slots for your application, or make more slots available for use. -------------------------------------------------------------------------- From: Ralph Castain <r...@open-mpi.org<mailto:r...@open-mpi.org>> Reply-To: Open MPI Users <us...@open-mpi.org<mailto:us...@open-mpi.org>> List-Post: users@lists.open-mpi.org Date: Saturday, November 23, 2013 2:27 PM To: Open MPI Users <us...@open-mpi.org<mailto:us...@open-mpi.org>> Subject: [EXTERNAL] Re: [OMPI users] (OpenMPI for Cray XE6 ) How to set mca parameters through aprun? My guess is that you aren't doing the allocation correctly - since you are using qsub, can I assume you have Moab as your scheduler? aprun should be forwarding the envars - do you see them if you just run "aprun -n 1 printenv"? On Nov 23, 2013, at 2:13 PM, Teranishi, Keita <knte...@sandia.gov<mailto:knte...@sandia.gov>> wrote: Hi, I installed OpenMPI on our small XE6 using the configure options under /contrib directory. It appears it is working fine, but it ignores MCA parameters (set in env var). So I switched to mpirun (in OpenMPI) and it can handle MCA parameters somehow. However, mpirun fails to allocate process by cores. For example, I allocated 32 cores (on 2 nodes) by "qsub –lmppwidth=32 –lmppnppn=16", mpirun recognizes it as 2 slots. Is it possible to mpirun to handle mluticore nodes of XE6 properly or is there any options to handle MCA parameters for aprun? Regards, ----------------------------------------------------------------------------- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> http://www.open-mpi.org/mailman/listinfo.cgi/users
