[slurm-users] Re: Jobs being denied for GrpCpuLimit despite having enough resource

2024-03-14 Thread Williams, Jenny Avis via slurm-users
Also -- scontrol show nodes -Original Message- From: Williams, Jenny Avis Sent: Thursday, March 14, 2024 6:46 PM To: Ole Holm Nielsen ; slurm-users@lists.schedmd.com Subject: RE: [slurm-users] Re: Jobs being denied for GrpCpuLimit despite having enough resource I use an alias slist = ` s

[slurm-users] Re: Jobs being denied for GrpCpuLimit despite having enough resource

2024-03-14 Thread Williams, Jenny Avis via slurm-users
I use an alias slist = ` sed 's/ /\n/g' |sort|uniq` -- do not cp/paste lines with "--" -- it is not the two hyphens intended. The examples below are for slurm 23.02.7 . These commands assume administrator access. This is a generalized set of areas I use to find why things just are not moving

[slurm-users] Re: Jobs being denied for GrpCpuLimit despite having enough resource

2024-03-14 Thread Ole Holm Nielsen via slurm-users
Hi Simon, Maybe you could print the user's limits using this tool: https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits Which version of Slurm do you run? /Ole On 3/14/24 17:47, Simon Andrews via slurm-users wrote: Our cluster has developed a strange intermittent behaviour

[slurm-users] Jobs being denied for GrpCpuLimit despite having enough resource

2024-03-14 Thread Simon Andrews via slurm-users
Our cluster has developed a strange intermittent behaviour where jobs are being put into a pending state because they aren't passing the AssocGrpCpuLimit, even though the user submitting has enough cpus for the job to run. For example: $ squeue -o "%.6i %.9P %.8j %.8u %.2t %.10M %.7m %.7c %.20R