Re: [slurm-users] GRES and GPUs

2023-07-17 Thread Groner, Rob
That would certainly do it. If you look at the slurmctld log when it comes up, it will say that it's marking that node as invalid because it has less (0) gres resources then you say it should have. That's because slurmd on that node will come up and say "What gres resources??" For testing pur

Re: [slurm-users] GRES and GPUs

2023-07-17 Thread Xaver Stiensmeier
Hi Hermann, Good idea, but we are already using `SelectType=select/cons_tres`. After setting everything up again (in case I made an unnoticed mistake), I saw that the node got marked STATE=inval. To be honest, I thought I can just claim that a node has a gpu even if it doesn't have one - just fo

Re: [slurm-users] GRES and GPUs

2023-07-17 Thread Hermann Schwärzler
Hi Xaver, what kind of SelectType are you using in your slurm.conf? Per https://slurm.schedmd.com/gres.html you have to consider: "As for the --gpu* option, these options are only supported by Slurm's select/cons_tres plugin." So you can use "--gpus ..." only when you state SelectType

[slurm-users] GRES and GPUs

2023-07-17 Thread Xaver Stiensmeier
Hey, I am currently trying to understand how I can schedule a job that needs a GPU. I read about GRES https://slurm.schedmd.com/gres.html and tried to use: GresTypes=gpu NodeName=test Gres=gpu:1 But calling - after a 'sudo scontrol reconfigure': srun --gpus 1 hostname didn't work: srun: err