Re: [slurm-users] Keep CPU Jobs Off GPU Nodes

2023-03-29 Thread Markus Kötter
Hello, On 29.03.23 10:08, René Sitt wrote: While the cited procedure works great in general, it gets more complicated for heterogeneous setups , i.e. if you have several GPU types defined in gres.conf, since the 'tres_per_' fields can then take the form of either 'gres:gpu:N' or 'gres:gpu::N'

Re: [slurm-users] Keep CPU Jobs Off GPU Nodes

2023-03-29 Thread René Sitt
Hello, maybe some additional notes: While the cited procedure works great in general, it gets more complicated for heterogeneous setups, i.e. if you have several GPU types defined in gres.conf, since the 'tres_per_' fields can then take the form of either 'gres:gpu:N' or 'gres:gpu::N' - depen

Re: [slurm-users] Keep CPU Jobs Off GPU Nodes

2023-03-29 Thread Wagner, Marcus
Hi Frank, use Features on the nodes, every cpu node gets e.g. "cpu", every gpu node e.g. "gpu". If a job asks for no gpus, set an additional constraint "cpu" for the job. Best Marcus Am 29.03.2023 um 01:24 schrieb Frank Pari: Well, I wanted to avoid using lua.  But, it looks like that's goi

Re: [slurm-users] Keep CPU Jobs Off GPU Nodes

2023-03-29 Thread Ward Poelmans
Hi, We have a dedicated partitions for GPUs (their name ends with _gpu) and simply forbid a job that is not requesting GPU resources to use this partition: local function job_total_gpus(job_desc) -- return total number of GPUs allocated to the job -- there are many ways to request a GPU

Re: [slurm-users] Keep CPU Jobs Off GPU Nodes

2023-03-28 Thread Frank Pari
Well, I wanted to avoid using lua. But, it looks like that's going to be the easiest way to do this without having to create a separate partition for the GPUs. Basically, check for at least one gpu in the job submission and if none exclude all GPU nodes for the job. [image: image.png] Now I'm w

[slurm-users] Keep CPU Jobs Off GPU Nodes

2023-03-28 Thread Frank Pari
Hi all, First, thank you all for participating in this list. I've learned so much by just following in other's threads. =) I'm looking at creating a scavenger partition with idle resources from CPU and GPU nodes and I'd like to keep this to one partition. But, I don't want CPU only jobs using