Hi,
We a small slurm install with two 4 socket servers with Opteron 6380 CPUs. The
CPUs are 16 core chips, but on this architecture the cores come in pairs
("modules"), which share some FPU resources. So Linux denotes what AMD would
call a "core" as a "thread» instead, i.e. they are actually 8 core chips. As
our workload is not very FPU intensive, I wonder if there is there a way to
override this in Slurm, and treat the threads as real cores? I have tried to
change CoresPerSocket=8 ThreadsPerCore=2 in slurm.conf to CoresPerSocket=16
ThreadsPerCore=1, and tried SelectTypeParameters=CR_CPU_Memory and
CR_Core_Memory, but it didn’t seem to have any effect. We are using
TaskPlugin=task/affinity.
What I’m trying to improve is that tasks get spread across NUMA nodes rather
than sharing the same modules, which seems to affect performance negatively.
Let’s say I run 8 tasks with 8 CPUs each:
$ srun -n8 --cpus-per-task=8 -w vali sleep 100
Then I get the following assignment of CPUs:
[paalmbj@vali ~]$ ps auxw | awk '/sleep/ {print $2;}' |xargs -n1 taskset -cp
pid 63051's current affinity list: 0,4,8,12,32,36,40,44
pid 63052's current affinity list: 1,5,9,13,33,37,41,45
pid 63053's current affinity list: 2,6,10,14,34,38,42,46
pid 63054's current affinity list: 3,7,11,15,35,39,43,47
pid 63055's current affinity list: 16,20,24,28,48,52,56,60
pid 63056's current affinity list: 17,21,25,29,49,53,57,61
pid 63057's current affinity list: 18,22,26,30,50,54,58,62
pid 63058's current affinity list: 19,23,27,31,51,55,59,63
But the NUMA nodes are shown below:
[paalmbj@vali ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 8
Vendor ID: AuthenticAMD
CPU family: 21
Model: 2
Model name: AMD Opteron(tm) Processor 6380
Stepping: 0
CPU MHz: 2493.615
BogoMIPS: 4986.94
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 6144K
NUMA node0 CPU(s): 0,4,8,12,16,20,24,28
NUMA node1 CPU(s): 32,36,40,44,48,52,56,60
NUMA node2 CPU(s): 1,5,9,13,17,21,25,29
NUMA node3 CPU(s): 33,37,41,45,49,53,57,61
NUMA node4 CPU(s): 2,6,10,14,18,22,26,30
NUMA node5 CPU(s): 34,38,42,46,50,54,58,62
NUMA node6 CPU(s): 35,39,43,47,51,55,59,63
NUMA node7 CPU(s): 3,7,11,15,19,23,27,31
Any hints on how to improve this?
Thanks,
Marius Bjørnstad
Norwegian Sequencing Centre
