Re: [slurm-users] Need to restart slurmctld for gres jobs to start

2022-06-24 Thread Ryan Novosielski
On 6/2/22 14:02, tluchko wrote: Hello, I have recently started to have problems where jobs sit in the queue waiting for resources to become available, even when the resources are available. If I stop and restart slurmctld, the pending jobs start running. This seems to be related to GRES jobs

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread gerard . gil
Hi Miguel, Good !! I'll try this options on all existing QOS and see if everything works as expected. I'll inform you on the results. Thanks a lot Best, Gérard - Mail original - > De: "Miguel Oliveira" > À: "Slurm-users" > Cc: "slurm-users" > Envoyé: Vendredi 24 Juin 2022 14:07:

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread Miguel Oliveira
Hi Bjørn-Helge, > On 24 Jun 2022, at 12:58, Bjørn-Helge Mevik wrote: > > Miguel Oliveira writes: > >> Hi Bjørn-Helge, >> >> Long time! > > Hi Miguel! Yes, definitely a long time! :D Indeed! > >> Why not? You can have multiple QoSs and you have other techniques to change >> priorities accor

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread Miguel Oliveira
Hi Gérard, I believe so. All our accounts correspond to one project and all have an associated QoS with NoDecay and DenyOnLimit. This is enough to restrict usage on each individual project. You only need these flags on the QoS. The association will carry on as usual and fairshare will not be i

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread Bjørn-Helge Mevik
Miguel Oliveira writes: > Hi Bjørn-Helge, > > Long time! Hi Miguel! Yes, definitely a long time! :D > Why not? You can have multiple QoSs and you have other techniques to change > priorities according to your policies. A job can only run in a single QoS, so if you submit a job with "sbatch

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread gerard . gil
Hi Miguel, > Why not? You can have multiple QoSs and you have other techniques to change > priorities according to your policies. Is this answer my question ? "If all configured QOS use NoDecay, we can take advantage of the FairShare priority with Decay and all jobs GrpTRESRaw with NoDecay ?"

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread Miguel Oliveira
Hi Bjørn-Helge, Long time! Why not? You can have multiple QoSs and you have other techniques to change priorities according to your policies. Best, MAO > On 24 Jun 2022, at 11:53, Bjørn-Helge Mevik wrote: > > Miguel Oliveira writes: > >> It is not exactly true that you have no solution to

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread Bjørn-Helge Mevik
Miguel Oliveira writes: > It is not exactly true that you have no solution to limit projects. If > you implement each project as an account then you can create an > account qos with the NoDecay flags. > This will not affect associations so priority and fair share are not impacted. Yes, that will

Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage

2022-06-24 Thread gerard . gil
Hi Miguel, It sounds good ! But does it mean you have to request this "NoDecay" QOS to benefit the fairshare priority ? Does this also mean that if all the QOS we use are created with NoDecay, we can take advantage of the FairShare priority and NoDecay for all jobs to use the GrpTRESMins l