We switched on hyper threading on our cluster with two eight core sockets per
node (32 threads per node).
We configured gridengine with 16 slots per node to allow the 16 extra threads
for kernel process use but this apparently does not work. Printout of the
gridengine hostfile shows that for a 32 slots job, 16 slots are placed on each
of two nodes as expected. Including the openmpi --display-map option shows that
all 32 processes are incorrectly placed on the head node. Here is part of the
output
master=cn6083
PE=orte
JOB_ID=2481793
Got 32 slots.
slots:
cn6083 16 par6.q@cn6083 <NULL>
cn6085 16 par6.q@cn6085 <NULL>
Sun Nov 9 16:50:59 GMT 2014
Data for JOB [44767,1] offset 0
======================== JOB MAP ========================
Data for node: cn6083 Num slots: 16 Max slots: 0 Num procs: 32
Process OMPI jobid: [44767,1] App: 0 Process rank: 0
Process OMPI jobid: [44767,1] App: 0 Process rank: 1
...
Process OMPI jobid: [44767,1] App: 0 Process rank: 31
=============================================================
I found some related mailings about a new warning in 1.8.2 about
oversubscription and I tried a few options to avoid the use of the extra
threads for MPI tasks by openmpi without success, e.g. variants of
--cpus-per-proc 1
--bind-to-core
and some others. Gridengine treats hw threads as cores==slots (?) but the
content of $PE_HOSTFILE suggests it distributes the slots sensibly so it seems
there is an option for openmpi required to get 16 cores per node?
I tried both 1.8.2, 1.8.3 and also 1.6.5.
Thanks for some clarification that anyone can give.
Henk
