Hi Gérard, If I understood you correctly your goal was to limit the number of minutes each project can run. By associating each project to a slurm account with a nodecay QoS then you will have achieved your goal. Try a project with a very small limit and you will see that it won’t run.
You don’t have to add anything. Each QoS will accumulate its respective usage, i.e, the usage of all users on that account. Users can even be on different accounts (projects) and charge the respective project with the parameter --account on sbatch. The GrpTRESMins is always changed on the QoS with a command like: sacctmgr update qos where qos=... set GrpTRESMin=cpu=…. Hope that makes sense! Best, MAO > On 28 Jun 2022, at 18:30, gerard....@cines.fr wrote: > > Hi Miguel, > > OK, I did'nt know this command. > > I'm not sure to understand how it works regarding to my goal. > I use the following command inspired by the command you gave me and I obtain > a UsageRaw for each QOS. > > scontrol -o show assoc_mgr -accounts=myaccount Users=" " > > > Do I have to sumup all QOS RawUsage to obtain the RawUsage of myaccount with > NoDecay ? > If I set GrpTRESMins for an Account and not for a QOS, does SLURM handle to > sumpup these QOS RawUsage to control if the GrpTRESMins account limit is > reach ? > > Thanks again for your precious help. > > Gérard > <http://www.cines.fr/> > > De: "Miguel Oliveira" <miguel.olive...@uc.pt> > À: "Slurm-users" <slurm-users@lists.schedmd.com> > Envoyé: Mardi 28 Juin 2022 17:23:18 > Objet: Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage > Hi Gérard, > > The way you are checking is against the association and as such it ought to > be decreasing in order to be used by fair share appropriately. > The counter used that does not decrease is on the QoS, not the association. > You can check that with: > > scontrol -o show assoc_mgr | grep "^QOS='+account+’” > > That ought to give you two numbers. The first is the limit, or N for not > limit, and the second in parenthesis the usage. > > Hope that helps. > > Best, > > Miguel Afonso Oliveira > > On 28 Jun 2022, at 08:58, gerard....@cines.fr <mailto:gerard....@cines.fr> > wrote: > > Hi Miguel, > > > I modified my test configuration to evaluate the effect of NoDecay. > > > > > I modified all QOS adding NoDecay Flag. > > > toto@login1:~/TEST$ sacctmgr show QOS > Name Priority GraceTime Preempt PreemptExemptTime PreemptMode > Flags UsageThres UsageFactor GrpTRES > GrpTRESMins GrpTRESRunMin GrpJobs GrpSubmit GrpWall MaxTRES > MaxTRESPerNode MaxTRESMins MaxWall MaxTRESPU MaxJobsPU MaxSubmitPU > MaxTRESPA MaxJobsPA MaxSubmitPA MinTRES > ---------- ---------- ---------- ---------- ------------------- ----------- > ---------------------------------------- ---------- ----------- ------------- > ------------- ------------- ------- --------- ----------- ------------- > -------------- ------------- ----------- ------------- --------- ----------- > ------------- --------- ----------- ------------- > normal 0 00:00:00 cluster > NoDecay 1.000000 > > > > interactif 10 00:00:00 cluster > NoDecay 1.000000 node=50 > node=22 > 1-00:00:00 node=50 > > petit 4 00:00:00 cluster > NoDecay 1.000000 node=1500 > node=22 > 1-00:00:00 node=300 > > gros 6 00:00:00 cluster > NoDecay 1.000000 node=2106 > node=700 > 1-00:00:00 node=700 > > court 8 00:00:00 cluster > NoDecay 1.000000 node=1100 > node=100 > 02:00:00 node=300 > > long 4 00:00:00 cluster > NoDecay 1.000000 node=500 > node=200 > 5-00:00:00 node=200 > > special 10 00:00:00 cluster > NoDecay 1.000000 node=2106 > node=2106 > 5-00:00:00 node=2106 > > support 10 00:00:00 cluster > NoDecay 1.000000 node=2106 > node=700 > 1-00:00:00 node=2106 > > visu 10 00:00:00 cluster > NoDecay 1.000000 node=4 > node=700 > 06:00:00 node=4 > > > > I submitted a bunch of jobs to control the NoDecay efficiency and I noticed > RawUsage as well as GrpTRESRaw cpu is still decreasing. > > > toto@login1:~/TEST$ sshare -A dci -u " " -o > account,user,GrpTRESRaw%80,GrpTRESMins,RawUsage > Account User > GrpTRESRaw GrpTRESMins RawUsage > -------------------- ---------- > ----------------------------------------------------- > ------------------------------ ----------- > dci > cpu=6932,mem=12998963,energy=0,node=216,billing=6932,fs/disk=0,vmem=0,pages=0 > cpu=17150 415966 > toto@login1:~/TEST$ sshare -A dci -u " " -o > account,user,GrpTRESRaw%80,GrpTRESMins,RawUsage > Account User > GrpTRESRaw GrpTRESMins RawUsage > -------------------- ---------- > ----------------------------------------------------- > ------------------------------ ----------- > dci > cpu=6931,mem=12995835,energy=0,node=216,billing=6931,fs/disk=0,vmem=0,pages=0 > cpu=17150 415866 > toto@login1:~/TEST$ sshare -A dci -u " " -o > account,user,GrpTRESRaw%80,GrpTRESMins,RawUsage > Account User > GrpTRESRaw GrpTRESMins RawUsage > -------------------- ---------- > ----------------------------------------------------- > ------------------------------ ----------- > dci > cpu=6929,mem=12992708,energy=0,node=216,billing=6929,fs/disk=0,vmem=0,pages=0 > cpu=17150 415766 > > > Something I forgot to do ? > > > Best, > Gérard > > Cordialement, > Gérard Gil > > Département Calcul Intensif > Centre Informatique National de l'Enseignement Superieur > 950, rue de Saint Priest > 34097 Montpellier CEDEX 5 > FRANCE > > tel : (334) 67 14 14 14 > fax : (334) 67 52 37 63 > web : http://www.cines.fr <http://www.cines.fr/> > > De: "Gérard Gil" <gerard....@cines.fr <mailto:gerard....@cines.fr>> > À: "Slurm-users" <slurm-users@lists.schedmd.com > <mailto:slurm-users@lists.schedmd.com>> > Cc: "slurm-users" <slurm-us...@schedmd.com <mailto:slurm-us...@schedmd.com>> > Envoyé: Vendredi 24 Juin 2022 14:52:12 > Objet: Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage > Hi Miguel, > > Good !! > > I'll try this options on all existing QOS and see if everything works as > expected. > I'll inform you on the results. > > > Thanks a lot > > Best, > Gérard > > > ----- Mail original ----- > De: "Miguel Oliveira" <miguel.olive...@uc.pt <mailto:miguel.olive...@uc.pt>> > À: "Slurm-users" <slurm-users@lists.schedmd.com > <mailto:slurm-users@lists.schedmd.com>> > Cc: "slurm-users" <slurm-us...@schedmd.com <mailto:slurm-us...@schedmd.com>> > Envoyé: Vendredi 24 Juin 2022 14:07:16 > Objet: Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage > > > Hi Gérard, > > I believe so. All our accounts correspond to one project and all have an > associated QoS with NoDecay and DenyOnLimit. This is enough to restrict usage > on each individual project. > You only need these flags on the QoS. The association will carry on as usual > and > fairshare will not be impacted. > > Hope that helps, > > Miguel Oliveira > > On 24 Jun 2022, at 12:56, gerard....@cines.fr <mailto:gerard....@cines.fr> > wrote: > > Hi Miguel, > > Why not? You can have multiple QoSs and you have other techniques to change > priorities according to your policies. > > > Is this answer my question ? > > "If all configured QOS use NoDecay, we can take advantage of the FairShare > priority with Decay and all jobs GrpTRESRaw with NoDecay ?" > > Thanks > > Best, > > > > Gérard > > >
smime.p7s
Description: S/MIME cryptographic signature
