It was just that when I started using map-by I didn't get why: ppr:2 but PE=2 I would at least have expected: ppr=2:PE=2 or ppr:2:PE:2 ? Does this have a reason?
2016-01-06 21:54 GMT+01:00 Ralph Castain <r...@open-mpi.org>: > <LOL> ah yes, “r” = “resource”!! Thanks for the reminder :-) > > The difference in delimiter is just to simplify parsing - we can “split” > the string on colons to separate out the options, and then use “=“ to set > the value. Nothing particularly significant about the choice. > > > On Jan 6, 2016, at 12:48 PM, Nick Papior <nickpap...@gmail.com> wrote: > > Your are correct. "socket" means that the resource is socket, "ppr:2" > means 2 processes per resource. > PE=<n> is Processing Elements per process. > > Perhaps the dev's can shed some light on why PE uses "=" and ppr has ":" > as delimiter for resource request? > > This "old" slide show from Jeff shows the usage (although the input have > changed since 1.7): > > http://www.slideshare.net/jsquyres/open-mpi-explorations-in-process-affinity-eurompi13-presentation > > > 2016-01-06 21:33 GMT+01:00 Matt Thompson <fort...@gmail.com>: > >> A ha! The Gurus know all. The map-by was the magic sauce: >> >> (1176) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 -map-by >> ppr:2:socket:pe=7 ./hello-hybrid.x | sort -g -k 18 >> Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU 0 >> Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU 1 >> Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU 2 >> Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU 3 >> Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU 4 >> Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU 5 >> Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU 6 >> Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU 7 >> Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU 8 >> Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU 9 >> Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU 10 >> Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU 11 >> Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU 12 >> Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU 13 >> Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU 14 >> Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU 15 >> Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU 16 >> Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU 17 >> Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU 18 >> Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU 19 >> Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU 20 >> Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU 21 >> Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU 22 >> Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU 23 >> Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU 24 >> Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU 25 >> Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU 26 >> Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU 27 >> >> So, a question: what does "ppr" mean? The man page seems to accept it as >> an axiom of Open MPI: >> >> --map-by <foo> >> Map to the specified object, defaults to socket. >> Supported options include slot, hwthread, core, L1cache, L2cache, L3cache, >> socket, numa, >> board, node, sequential, distance, and ppr. Any object can >> include modifiers by adding a : and any combination of PE=n (bind n >> processing >> elements to each proc), SPAN (load balance the processes >> across the allocation), OVERSUBSCRIBE (allow more processes on a node than >> pro‐ >> cessing elements), and NOOVERSUBSCRIBE. This includes PPR, >> where the pattern would be terminated by another colon to separate it from >> the >> modifiers. >> >> Is it an acronym/initialism? From some experimenting it seems to be >> ppr:2:socket means 2 processes per socket? And pe=7 means leave 7 processes >> between them? Is that about right? >> >> Matt >> >> On Wed, Jan 6, 2016 at 3:19 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >>> I believe he wants two procs/socket, so you’d need ppr:2:socket:pe=7 >>> >>> >>> On Jan 6, 2016, at 12:14 PM, Nick Papior <nickpap...@gmail.com> wrote: >>> >>> I do not think KMP_AFFINITY should affect anything in OpenMPI, it is an >>> MKL env setting? Or am I wrong? >>> >>> Note that these are used in an environment where openmpi automatically >>> gets the host-file. Hence they are not present. >>> With intel mkl and openmpi I got the best performance using these, >>> rather long flags: >>> >>> export KMP_AFFINITY=verbose,compact,granularity=core >>> export KMP_STACKSIZE=62M >>> export KMP_SETTINGS=1 >>> >>> def_flags="--bind-to core -x OMP_PROC_BIND=true --report-bindings" >>> def_flags="$def_flags -x KMP_AFFINITY=$KMP_AFFINITY" >>> >>> # in your case 7: >>> ONP=7 >>> flags="$def_flags -x MKL_NUM_THREADS=$ONP -x MKL_DYNAMIC=FALSE" >>> flags="$flags -x OMP_NUM_THREADS=$ONP -x OMP_DYNAMIC=FALSE" >>> flags="$flags -x KMP_STACKSIZE=$KMP_STACKSIZE" >>> flags="$flags --map-by ppr:1:socket:pe=7" >>> >>> then run your program: >>> >>> mpirun $flags <app> >>> >>> A lot of the option flags are duplicated (and strictly not needed), but >>> I provide them for easy testing changes. >>> Surely this is application dependent, but for my case it was performing >>> really well. >>> >>> >>> 2016-01-06 20:48 GMT+01:00 Erik Schnetter <schnet...@gmail.com>: >>> >>>> Setting KMP_AFFINITY will probably override anything that OpenMPI >>>> sets. Can you try without? >>>> >>>> -erik >>>> >>>> On Wed, Jan 6, 2016 at 2:46 PM, Matt Thompson <fort...@gmail.com> >>>> wrote: >>>> > Hello Open MPI Gurus, >>>> > >>>> > As I explore MPI-OpenMP hybrid codes, I'm trying to figure out how to >>>> do >>>> > things to get the same behavior in various stacks. For example, I >>>> have a >>>> > 28-core node (2 14-core Haswells), and I'd like to run 4 MPI >>>> processes and 7 >>>> > OpenMP threads. Thus, I'd like the processes to be 2 processes per >>>> socket >>>> > with the OpenMP threads laid out on them. Using a "hybrid Hello World" >>>> > program, I can achieve this with Intel MPI (after a lot of testing): >>>> > >>>> > (1097) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 >>>> > ./hello-hybrid.x | sort -g -k 18 >>>> > srun.slurm: cluster configuration lacks support for cpu binding >>>> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 0 >>>> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 1 >>>> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 2 >>>> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 3 >>>> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 4 >>>> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 5 >>>> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 6 >>>> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 7 >>>> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 8 >>>> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 9 >>>> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 10 >>>> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 11 >>>> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 12 >>>> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 13 >>>> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 14 >>>> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 15 >>>> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 16 >>>> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 17 >>>> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 18 >>>> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 19 >>>> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 20 >>>> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 21 >>>> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 22 >>>> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 23 >>>> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 24 >>>> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 25 >>>> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 26 >>>> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 27 >>>> > >>>> > Other than the odd fact that Process #0 seemed to start on Socket #1 >>>> (this >>>> > might be an artifact of how I'm trying to detect the CPU I'm on), >>>> this looks >>>> > reasonable. 14 threads on each socket and each process is laying out >>>> its >>>> > threads in a nice orderly fashion. >>>> > >>>> > I'm trying to figure out how to do this with Open MPI (version >>>> 1.10.0) and >>>> > apparently I am just not quite good enough to figure it out. The >>>> closest >>>> > I've gotten is: >>>> > >>>> > (1155) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 >>>> -map-by >>>> > ppr:2:socket ./hello-hybrid.x | sort -g -k 18 >>>> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 0 >>>> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 0 >>>> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 1 >>>> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 1 >>>> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 2 >>>> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 2 >>>> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 3 >>>> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 3 >>>> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 4 >>>> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 4 >>>> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 5 >>>> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 5 >>>> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on >>>> CPU 6 >>>> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on >>>> CPU 6 >>>> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 14 >>>> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 14 >>>> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 15 >>>> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 15 >>>> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 16 >>>> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 16 >>>> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 17 >>>> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 17 >>>> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 18 >>>> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 18 >>>> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 19 >>>> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 19 >>>> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on >>>> CPU 20 >>>> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on >>>> CPU 20 >>>> > >>>> > Obviously not right. Any ideas on how to help me learn? The man >>>> mpirun page >>>> > is a bit formidable in the pinning part, so maybe I've missed an >>>> obvious >>>> > answer. >>>> > >>>> > Matt >>>> > -- >>>> > Matt Thompson >>>> > >>>> > Man Among Men >>>> > Fulcrum of History >>>> > >>>> > >>>> > _______________________________________________ >>>> > users mailing list >>>> > us...@open-mpi.org >>>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> > Link to this post: >>>> > http://www.open-mpi.org/community/lists/users/2016/01/28217.php >>>> >>>> >>>> >>>> -- >>>> Erik Schnetter <schnet...@gmail.com> >>>> http://www.perimeterinstitute.ca/personal/eschnetter/ >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2016/01/28218.php >>>> >>> >>> >>> >>> -- >>> Kind regards Nick >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/01/28219.php >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/01/28221.php >>> >> >> >> >> -- >> Matt Thompson >> >> Man Among Men >> Fulcrum of History >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/01/28223.php >> > > > > -- > Kind regards Nick > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/01/28224.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/01/28226.php > -- Kind regards Nick
