Hi Couple questions:
First, I cannot tell from the given `sacctmgr` output where that limit is. Is it maxtres per-account or per-user? It could also be useful to see the output for the normal qos as well. It'd be more readable if you could use "format=name,..." and limit the number of fields displayed to those of interest (e.g. maxtresperaccount). Second- you have verified that users are exceeding that TRES? I ask because the partition QOS doesn't appear in the output of sacct/squeue. It's a "behind-the-scenes" kind of thing that is (for me at least) applied but not visible. For example, we use multiple partition and "regular" QOS's: # squeue -O jobid,partition,qos |head JOBID PARTITION QOS 46517910 campus normal 46517911 campus normal "normal" is the default QOS and the only limit we run there is on job submission. There's a "public" QOS that applies a maxtrespa cpu limit on the partition, but as you can see its not displayed. I'm not sure of a way to confirm the application of the limit outside of actually trying to exceed the limit. FWIW, we're well behind at 15.08.7. On Thu, Jan 19, 2017 at 12:37 AM, Frederik Himpe <[email protected]> wrote: > > I want to limit the amount of CPUs a single user can use in one > partition, so I created a QOS named "batch" and set this as the default > QOS in slurm.conf for the (default) partition "batch". > > However when I submit a job to the default partition, the QOS batch is > not used so the user can use much more resources than allowed in the > QOS, for example: > > $ squeue -j 3995 -O partition,QOS > PARTITION QOS > batch normal > > What could be wrong? > > > # scontrol show partition batch > PartitionName=batch > AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL > AllocNodes=ALL Default=YES QoS=batch > DefaultTime=NONE DisableRootJobs=YES ExclusiveUser=NO GraceTime=0 > Hidden=NO > MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=YES > MaxCPUsPerNode=UNLIMITED > Nodes=node[001-012] > PriorityJobFactor=3 PriorityTier=3 RootOnly=NO ReqResv=NO > OverSubscribe=FORCE:1 PreemptMode=GANG,SUSPEND > State=UP TotalCPUs=168 TotalNodes=12 SelectTypeParameters=NONE > DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED > > # sacctmgr list qos batch > Name Priority GraceTime Preempt PreemptMode > Flags > UsageThres UsageFactor GrpTRES GrpTRESMins GrpTRESRunMin GrpJobs > GrpSubmit GrpWall MaxTRES MaxTRESPerNode MaxTRESMins MaxWall > MaxTRESPU > MaxJobsPU MaxSubmitPU MaxTRESPA MaxJobsPA MaxSubmitPA MinTRES > ---------- ---------- ---------- ---------- ----------- > ---------------------------------------- ---------- ----------- > ------------- ------------- ------------- ------- --------- ----------- > ------------- -------------- ------------- ----------- ------------- > --------- ----------- ------------- --------- ----------- ------------- > batch 0 00:00:00 cluster > 1.000000 > > > cpu=80 > > > -- > Frederik Himpe >
