Your are correct. "socket" means that the resource is socket, "ppr:2" means 2 processes per resource. PE=<n> is Processing Elements per process.
Perhaps the dev's can shed some light on why PE uses "=" and ppr has ":" as delimiter for resource request? This "old" slide show from Jeff shows the usage (although the input have changed since 1.7): http://www.slideshare.net/jsquyres/open-mpi-explorations-in-process-affinity-eurompi13-presentation 2016-01-06 21:33 GMT+01:00 Matt Thompson <fort...@gmail.com>: > A ha! The Gurus know all. The map-by was the magic sauce: > > (1176) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 -map-by > ppr:2:socket:pe=7 ./hello-hybrid.x | sort -g -k 18 > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU 0 > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU 1 > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU 2 > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU 3 > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU 4 > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU 5 > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU 6 > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU 7 > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU 8 > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU 9 > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU 10 > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU 11 > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU 12 > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU 13 > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU 14 > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU 15 > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU 16 > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU 17 > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU 18 > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU 19 > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU 20 > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU 21 > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU 22 > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU 23 > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU 24 > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU 25 > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU 26 > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU 27 > > So, a question: what does "ppr" mean? The man page seems to accept it as > an axiom of Open MPI: > > --map-by <foo> > Map to the specified object, defaults to socket. Supported > options include slot, hwthread, core, L1cache, L2cache, L3cache, socket, > numa, > board, node, sequential, distance, and ppr. Any object can > include modifiers by adding a : and any combination of PE=n (bind n > processing > elements to each proc), SPAN (load balance the processes > across the allocation), OVERSUBSCRIBE (allow more processes on a node than > pro‐ > cessing elements), and NOOVERSUBSCRIBE. This includes PPR, > where the pattern would be terminated by another colon to separate it from > the > modifiers. > > Is it an acronym/initialism? From some experimenting it seems to be > ppr:2:socket means 2 processes per socket? And pe=7 means leave 7 processes > between them? Is that about right? > > Matt > > On Wed, Jan 6, 2016 at 3:19 PM, Ralph Castain <r...@open-mpi.org> wrote: > >> I believe he wants two procs/socket, so you’d need ppr:2:socket:pe=7 >> >> >> On Jan 6, 2016, at 12:14 PM, Nick Papior <nickpap...@gmail.com> wrote: >> >> I do not think KMP_AFFINITY should affect anything in OpenMPI, it is an >> MKL env setting? Or am I wrong? >> >> Note that these are used in an environment where openmpi automatically >> gets the host-file. Hence they are not present. >> With intel mkl and openmpi I got the best performance using these, rather >> long flags: >> >> export KMP_AFFINITY=verbose,compact,granularity=core >> export KMP_STACKSIZE=62M >> export KMP_SETTINGS=1 >> >> def_flags="--bind-to core -x OMP_PROC_BIND=true --report-bindings" >> def_flags="$def_flags -x KMP_AFFINITY=$KMP_AFFINITY" >> >> # in your case 7: >> ONP=7 >> flags="$def_flags -x MKL_NUM_THREADS=$ONP -x MKL_DYNAMIC=FALSE" >> flags="$flags -x OMP_NUM_THREADS=$ONP -x OMP_DYNAMIC=FALSE" >> flags="$flags -x KMP_STACKSIZE=$KMP_STACKSIZE" >> flags="$flags --map-by ppr:1:socket:pe=7" >> >> then run your program: >> >> mpirun $flags <app> >> >> A lot of the option flags are duplicated (and strictly not needed), but I >> provide them for easy testing changes. >> Surely this is application dependent, but for my case it was performing >> really well. >> >> >> 2016-01-06 20:48 GMT+01:00 Erik Schnetter <schnet...@gmail.com>: >> >>> Setting KMP_AFFINITY will probably override anything that OpenMPI >>> sets. Can you try without? >>> >>> -erik >>> >>> On Wed, Jan 6, 2016 at 2:46 PM, Matt Thompson <fort...@gmail.com> wrote: >>> > Hello Open MPI Gurus, >>> > >>> > As I explore MPI-OpenMP hybrid codes, I'm trying to figure out how to >>> do >>> > things to get the same behavior in various stacks. For example, I have >>> a >>> > 28-core node (2 14-core Haswells), and I'd like to run 4 MPI processes >>> and 7 >>> > OpenMP threads. Thus, I'd like the processes to be 2 processes per >>> socket >>> > with the OpenMP threads laid out on them. Using a "hybrid Hello World" >>> > program, I can achieve this with Intel MPI (after a lot of testing): >>> > >>> > (1097) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 >>> > ./hello-hybrid.x | sort -g -k 18 >>> > srun.slurm: cluster configuration lacks support for cpu binding >>> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 0 >>> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 1 >>> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 2 >>> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 3 >>> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 4 >>> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 5 >>> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 6 >>> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 7 >>> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 8 >>> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 9 >>> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 10 >>> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 11 >>> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 12 >>> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 13 >>> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 14 >>> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 15 >>> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 16 >>> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 17 >>> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 18 >>> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 19 >>> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 20 >>> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 21 >>> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 22 >>> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 23 >>> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 24 >>> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 25 >>> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 26 >>> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 27 >>> > >>> > Other than the odd fact that Process #0 seemed to start on Socket #1 >>> (this >>> > might be an artifact of how I'm trying to detect the CPU I'm on), this >>> looks >>> > reasonable. 14 threads on each socket and each process is laying out >>> its >>> > threads in a nice orderly fashion. >>> > >>> > I'm trying to figure out how to do this with Open MPI (version 1.10.0) >>> and >>> > apparently I am just not quite good enough to figure it out. The >>> closest >>> > I've gotten is: >>> > >>> > (1155) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 >>> -map-by >>> > ppr:2:socket ./hello-hybrid.x | sort -g -k 18 >>> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 0 >>> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 0 >>> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 1 >>> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 1 >>> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 2 >>> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 2 >>> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 3 >>> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 3 >>> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 4 >>> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 4 >>> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 5 >>> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 5 >>> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on >>> CPU 6 >>> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on >>> CPU 6 >>> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 14 >>> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 14 >>> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 15 >>> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 15 >>> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 16 >>> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 16 >>> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 17 >>> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 17 >>> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 18 >>> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 18 >>> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 19 >>> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 19 >>> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on >>> CPU 20 >>> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on >>> CPU 20 >>> > >>> > Obviously not right. Any ideas on how to help me learn? The man mpirun >>> page >>> > is a bit formidable in the pinning part, so maybe I've missed an >>> obvious >>> > answer. >>> > >>> > Matt >>> > -- >>> > Matt Thompson >>> > >>> > Man Among Men >>> > Fulcrum of History >>> > >>> > >>> > _______________________________________________ >>> > users mailing list >>> > us...@open-mpi.org >>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> > Link to this post: >>> > http://www.open-mpi.org/community/lists/users/2016/01/28217.php >>> >>> >>> >>> -- >>> Erik Schnetter <schnet...@gmail.com> >>> http://www.perimeterinstitute.ca/personal/eschnetter/ >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/01/28218.php >>> >> >> >> >> -- >> Kind regards Nick >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/01/28219.php >> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/01/28221.php >> > > > > -- > Matt Thompson > > Man Among Men > Fulcrum of History > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/01/28223.php > -- Kind regards Nick
