Well it seems like the usual solution worked. I upgraded from 14.11 to 16.05 and I’m getting a sane CPU mask. Sorry I didn’t try this first, but it takes a bit of coordination.
Cheers, Marius > 17. aug. 2016 kl. 11.38 skrev Marius Bjørnstad <[email protected]>: > > > Hi, > > We a small slurm install with two 4 socket servers with Opteron 6380 CPUs. > The CPUs are 16 core chips, but on this architecture the cores come in pairs > ("modules"), which share some FPU resources. So Linux denotes what AMD would > call a "core" as a "thread» instead, i.e. they are actually 8 core chips. As > our workload is not very FPU intensive, I wonder if there is there a way to > override this in Slurm, and treat the threads as real cores? I have tried to > change CoresPerSocket=8 ThreadsPerCore=2 in slurm.conf to CoresPerSocket=16 > ThreadsPerCore=1, and tried SelectTypeParameters=CR_CPU_Memory and > CR_Core_Memory, but it didn’t seem to have any effect. We are using > TaskPlugin=task/affinity. > > What I’m trying to improve is that tasks get spread across NUMA nodes rather > than sharing the same modules, which seems to affect performance negatively. > > Let’s say I run 8 tasks with 8 CPUs each: > > $ srun -n8 --cpus-per-task=8 -w vali sleep 100 > > Then I get the following assignment of CPUs: > > [paalmbj@vali ~]$ ps auxw | awk '/sleep/ {print $2;}' |xargs -n1 taskset -cp > pid 63051's current affinity list: 0,4,8,12,32,36,40,44 > pid 63052's current affinity list: 1,5,9,13,33,37,41,45 > pid 63053's current affinity list: 2,6,10,14,34,38,42,46 > pid 63054's current affinity list: 3,7,11,15,35,39,43,47 > pid 63055's current affinity list: 16,20,24,28,48,52,56,60 > pid 63056's current affinity list: 17,21,25,29,49,53,57,61 > pid 63057's current affinity list: 18,22,26,30,50,54,58,62 > pid 63058's current affinity list: 19,23,27,31,51,55,59,63 > > But the NUMA nodes are shown below: > > [paalmbj@vali ~]$ lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 64 > On-line CPU(s) list: 0-63 > Thread(s) per core: 2 > Core(s) per socket: 8 > Socket(s): 4 > NUMA node(s): 8 > Vendor ID: AuthenticAMD > CPU family: 21 > Model: 2 > Model name: AMD Opteron(tm) Processor 6380 > Stepping: 0 > CPU MHz: 2493.615 > BogoMIPS: 4986.94 > Virtualization: AMD-V > L1d cache: 16K > L1i cache: 64K > L2 cache: 2048K > L3 cache: 6144K > NUMA node0 CPU(s): 0,4,8,12,16,20,24,28 > NUMA node1 CPU(s): 32,36,40,44,48,52,56,60 > NUMA node2 CPU(s): 1,5,9,13,17,21,25,29 > NUMA node3 CPU(s): 33,37,41,45,49,53,57,61 > NUMA node4 CPU(s): 2,6,10,14,18,22,26,30 > NUMA node5 CPU(s): 34,38,42,46,50,54,58,62 > NUMA node6 CPU(s): 35,39,43,47,51,55,59,63 > NUMA node7 CPU(s): 3,7,11,15,19,23,27,31 > > > Any hints on how to improve this? > > Thanks, > Marius Bjørnstad > > Norwegian Sequencing Centre >
