Ah, yes, my example was for 10 cores per socket, good catch :) 2016-01-06 21:19 GMT+01:00 Ralph Castain <r...@open-mpi.org>:
> I believe he wants two procs/socket, so you’d need ppr:2:socket:pe=7 > > > On Jan 6, 2016, at 12:14 PM, Nick Papior <nickpap...@gmail.com> wrote: > > I do not think KMP_AFFINITY should affect anything in OpenMPI, it is an > MKL env setting? Or am I wrong? > > Note that these are used in an environment where openmpi automatically > gets the host-file. Hence they are not present. > With intel mkl and openmpi I got the best performance using these, rather > long flags: > > export KMP_AFFINITY=verbose,compact,granularity=core > export KMP_STACKSIZE=62M > export KMP_SETTINGS=1 > > def_flags="--bind-to core -x OMP_PROC_BIND=true --report-bindings" > def_flags="$def_flags -x KMP_AFFINITY=$KMP_AFFINITY" > > # in your case 7: > ONP=7 > flags="$def_flags -x MKL_NUM_THREADS=$ONP -x MKL_DYNAMIC=FALSE" > flags="$flags -x OMP_NUM_THREADS=$ONP -x OMP_DYNAMIC=FALSE" > flags="$flags -x KMP_STACKSIZE=$KMP_STACKSIZE" > flags="$flags --map-by ppr:1:socket:pe=7" > > then run your program: > > mpirun $flags <app> > > A lot of the option flags are duplicated (and strictly not needed), but I > provide them for easy testing changes. > Surely this is application dependent, but for my case it was performing > really well. > > > 2016-01-06 20:48 GMT+01:00 Erik Schnetter <schnet...@gmail.com>: > >> Setting KMP_AFFINITY will probably override anything that OpenMPI >> sets. Can you try without? >> >> -erik >> >> On Wed, Jan 6, 2016 at 2:46 PM, Matt Thompson <fort...@gmail.com> wrote: >> > Hello Open MPI Gurus, >> > >> > As I explore MPI-OpenMP hybrid codes, I'm trying to figure out how to do >> > things to get the same behavior in various stacks. For example, I have a >> > 28-core node (2 14-core Haswells), and I'd like to run 4 MPI processes >> and 7 >> > OpenMP threads. Thus, I'd like the processes to be 2 processes per >> socket >> > with the OpenMP threads laid out on them. Using a "hybrid Hello World" >> > program, I can achieve this with Intel MPI (after a lot of testing): >> > >> > (1097) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 >> > ./hello-hybrid.x | sort -g -k 18 >> > srun.slurm: cluster configuration lacks support for cpu binding >> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU >> 0 >> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU >> 1 >> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU >> 2 >> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU >> 3 >> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU >> 4 >> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU >> 5 >> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU >> 6 >> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU >> 7 >> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU >> 8 >> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU >> 9 >> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU >> 10 >> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU >> 11 >> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU >> 12 >> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU >> 13 >> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU >> 14 >> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU >> 15 >> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU >> 16 >> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU >> 17 >> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU >> 18 >> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU >> 19 >> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU >> 20 >> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU >> 21 >> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU >> 22 >> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU >> 23 >> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU >> 24 >> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU >> 25 >> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU >> 26 >> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU >> 27 >> > >> > Other than the odd fact that Process #0 seemed to start on Socket #1 >> (this >> > might be an artifact of how I'm trying to detect the CPU I'm on), this >> looks >> > reasonable. 14 threads on each socket and each process is laying out its >> > threads in a nice orderly fashion. >> > >> > I'm trying to figure out how to do this with Open MPI (version 1.10.0) >> and >> > apparently I am just not quite good enough to figure it out. The closest >> > I've gotten is: >> > >> > (1155) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 -map-by >> > ppr:2:socket ./hello-hybrid.x | sort -g -k 18 >> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU >> 0 >> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU >> 0 >> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU >> 1 >> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU >> 1 >> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU >> 2 >> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU >> 2 >> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU >> 3 >> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU >> 3 >> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU >> 4 >> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU >> 4 >> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU >> 5 >> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU >> 5 >> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU >> 6 >> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU >> 6 >> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU >> 14 >> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU >> 14 >> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU >> 15 >> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU >> 15 >> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU >> 16 >> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU >> 16 >> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU >> 17 >> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU >> 17 >> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU >> 18 >> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU >> 18 >> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU >> 19 >> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU >> 19 >> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU >> 20 >> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU >> 20 >> > >> > Obviously not right. Any ideas on how to help me learn? The man mpirun >> page >> > is a bit formidable in the pinning part, so maybe I've missed an obvious >> > answer. >> > >> > Matt >> > -- >> > Matt Thompson >> > >> > Man Among Men >> > Fulcrum of History >> > >> > >> > _______________________________________________ >> > users mailing list >> > us...@open-mpi.org >> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> > Link to this post: >> > http://www.open-mpi.org/community/lists/users/2016/01/28217.php >> >> >> >> -- >> Erik Schnetter <schnet...@gmail.com> >> http://www.perimeterinstitute.ca/personal/eschnetter/ >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/01/28218.php >> > > > > -- > Kind regards Nick > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/01/28219.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/01/28221.php > -- Kind regards Nick
