Hi,
I want to allocate at the level of logical cores (each serial job gets one
thread on a hyperthreading system), which seems to be achievable only by
not setting threads_per_core on each node, and instead just setting
CPUs=<total_logical_cores>.
However with core binding, this will pack two tasks onto the same physical
core while all other cores are left idle. On a system with 20 cores and 40
threads I see this behavior:
$ srun bash -c "lstopo | head -n 6 ; sleep 10" &
NUMANode L#0 (P#0 47GB) + Package L#0 + L3 L#0 (25MB) + L2 L#0 (256KB) +
L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
NUMANode L#1 (P#1 47GB)
HostBridge L#0
PCIBridge
PCI 15b3:1003
$ srun bash -c "lstopo | head -n 6 ; sleep 10" &
NUMANode L#0 (P#0 47GB) + Package L#0 + L3 L#0 (25MB) + L2 L#0 (256KB) +
L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#20)
NUMANode L#1 (P#1 47GB)
HostBridge L#0
PCIBridge
PCI 15b3:1003
I expected to see the second job get logical core #1 (on the second
physical core) but instead it gets logical core #20 (the second thread on
the first physical core). I can't imagine that this is ever the desired
behavior, but I guess I could be missing some use case.
I have spent quite a bit of time reading the documentation/mailing list and
experimenting with different options, all to no avail. Is it possible to
achieve my desired setup with Slurm?
I also experimented with setting threads_per_core=2 and then setting
OverSubscribe=FORCE:2, but I am pretty unhappy with the results. I think
it is confusing that you request one cpu and end up with two (with your
--mem-per-cpu doubled), and best I can tell there is no way to only
oversubscribe if the user requested 1 core instead of two.
Thanks for your time,
Brendan
