For future users reference, if you set; Sockets=2 CoresPerSocket=8 ThreadsPerCore=2
or similarly; CPUs=32 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 Youre queue will show 32 cpus availlable (e.g. sinfo -o %C), but for some reason the lowest available allocation level will be by core, and hence any job you run will automatically allocate a whole core (2 CPUs) per job, and you will be limited to 16 jobs. It doesnt seem like changing it to a different resource allocation method makes a difference, and almost seems buggy to me, but I guess is just a quirk of multithread systems. On Mon, Sep 12, 2016 at 9:46 AM, alex straza <[email protected]> wrote: > Thanks Ashton, your solution worked. > > Uwe - don't know if your solution would have worked since I tried this > first, but thanks in any case! > > On Mon, Sep 12, 2016 at 12:12 PM, andrealphus <[email protected]> wrote: >> >> >> Hi Alex! Please try this solution first, as I just went through this >> exact same issue over the past two weeks, and believe setting anything >> related to cores/sockets/threads will not work. >> >> Leave; >> SelectType=select/cons_res >> SelectTypeParameters=CR_CPU >> >> But in the node configuration you need to only have the # of CPUs >> (virual+real threads), do not put anything about the number of >> sockets, threads or cores or else you will be limited to the number of >> cores (18); >> >> NodeName=node1 CPUs=32 RealMemory=128000 State=UNKNOWN >> >> This was the solution I got from the mailing list that eventually >> worked, and can be references in the slurm faq, #30 >> http://slurm.schedmd.com/faq.html >> [...] >> 30. Slurm documentation refers to CPUs, cores and threads. What exactly >> is considered a CPU? >> If your nodes are configured with hyperthreading, then a CPU is >> equivalent to a hyperthread. Otherwise a CPU is equivalent to a core. >> You can determine if your nodes have more than one thread per core using >> the command "scontrol show node" and looking at the values of >> "ThreadsPerCore". >> >> >> Also, I dont know your IO setup, but just remember if you are reading >> and/or writing small files then tossing more CPUs at it might not be >> beneficial as you can hit an IO wall. Best to run test cases for a >> varying number of CPUs runs and see how efficient it is. (unless your >> writing to independent drives seperatly) >> >> -ashton >> >> >> >> >> >> On Mon, Sep 12, 2016 at 7:55 AM, Uwe Sauter <[email protected]> >> wrote: >> > >> > Also. CPUs=32 is wrong. You need >> > >> > Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 >> > >> > Am 12.09.2016 um 16:02 schrieb alex straza: >> >> hello, >> >> >> >> We have some slurm nodes that have 32 CPUS - two 8-core processors with >> >> hyperthreading - and are trying to run some >> >> "embarrassingly parallel" jobs. However, no matter how we set values >> >> in slurm.conf, slurm will only run at most 16 jobs >> >> at a time on each machine. What are the right settings to use here? >> >> this is what we have at the moment in the config: >> >> >> >> SchedulerType=sched/builtin >> >> SelectType=select/cons_res >> >> SelectTypeParameters=CR_CPU >> >> >> >> NodeName=node1 CoresPerSocket=8 CPUs=32 RealMemory=128000 >> >> ThreadsPerCore=2 Sockets=2 State=UNKNOWN >> >> PartitionName=debug Nodes=node1 Default=YES MaxTime=INFINITE State=UP >> >> DefMemPerCPU=4000 MaxMemPerCPU=32000 > >
