So, the overall answer is yes. But there's a lot to it. I can't detail
everything we did to get where you're talking, but let me try to hit some high
points. Basically, google anything I'm about to mention to get a fuller
explanation:
*
You can set the "cost" of things per partition via the TRESBillingWeights.
That includes CPUs, mem, gpus.
*
You'll need to define anything other than cpu/mem/nodes that you'll want to
bill for via AccountingStorageTRES. So, all your GPUs would be specified here.
*
Slurm provides a somewhat generic "billing" concept. So, for each minute of
job time, you can indicate how much "billing" they've accumulated.
*
You can see how much billing they've accumulated over the life of the account
via: sshare -a -h -p -o account,user,GrpTRESRaw,GrpTRESMins. The GrpTRESRaw
contains a "billing=". That's the number. There will be values for
cpu=, mem=...and those are also indicators of how much they've used. But the
billing is where you can set the rates via the TRESBillingWeights.
*
You can set how much billing they have available to use via: sacctmgr modify
account set grptresmins=billing= where account=.
*
You can keep people from running jobs if they wouldn't have enough "billing"
available by setting AccountingStorageEnforce=safe
*
NOTE: The GrpTRESRaw billing value just keeps going up. If the GrpTRESRaw
value is higher than what is in GrpTresMins=billing...then they won't be able
to run anything that could "cost" them some billing. In other wordsyou
don't set their GrpTresMins=billing=1000 to give them 1000 "credits". Instead,
you'd take the value from GrpTRESRaw billing, and ADD 1000 to that. Then set
GrpTresMins=billing to that new value. THEN they have 1000 to spend.
I HIGHLY recommend setting up a test system and messing around with values to
see what happens. That helps a lot better to understand how it works. I found
the documentation not very helpful in UNDERSTANDING how it works. I had to
trial/error it to get it.
Hope that helps. Bottom line...yes, it sounds like slurm can do most
everything I saw you wanted to do.
Rob
From: John Snowdon via slurm-users
Sent: Friday, October 18, 2024 4:56 AM
To: slurm-users@lists.schedmd.com
Subject: [slurm-users] Tracking costs - one single pool of credits, variable
costs per partition
We are trying to design the charging and accounting system for our new
institutional HPC facility and I'm having difficulty understanding exactly how
we can use sacctmgr to achieve what we need.
Until now, our previous HPC facilities have all operated as free delivery and
we have not needed to track costs by user/group/project. Account codes have
been purely optional.
However, our new facility will be split into various resource types, with free
partitions and paid/priority/reserved partitions across those resource types.
All jobs will need to be submitted with an account code.
For users submitting to 'free' partitions we don't need to track resource units
against a balance, but the submitted account code would still be used for
reporting purposes (i.e. "free resources accounted for % of all use by this
project in August-September").
When submitting to a 'paid' partition, the account code needs to be checked to
ensure it has a positive balance (or a balance that will not go past some
negative threshold).
Each of the 'paid' partitions may (will) have a different resource unit cost. A
simple example:
- Submit to a generic CPU paid partition
-- 1 resource unit/token/credit/£/$ per allocated cpu, per hour of compute
- Submit to a high-speed, non-blocking CPU paid partition
-- 2 resource unit/token/credit/£/$ per allocated cpu, per hour of compute
- Submit to a GPU paid partition
-- 4 resource unit/token/credit/£/$ per allocated GPU card, per hour of compute
We need to have *one* pool of resource units/tokens/credits per account - let's
say 1000 credits, and a group of users may well decide to spend all of their
credits on the generic CPU partition, all on the GPU partition, or some mixture
of the two.
So in the above examples, assuming one user (or group of users sharing the same
account code) submit a 2 hour job to all three partitions, their one, single
account code should be charged:
- 2 units for the generic CPU partition
- 4 units for the job on the low latency partition
- 8 units for the gpu partition.
- A total of 14 credits removed from their single account code
Is this feasible to achieve without having to allocate credits to each of the
partitions for an account, or creating a QOS variant for each and every
combination of account and partition?
John Snowdon
Senior Research Infrastructure Engineer (HPC)
Research Software Engineering
Catalyst Building, Room 2.01
Newcastle University
3 Science Square
Newcastle Helix
Newcastle upon Tyne
NE4 5TG
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Frse.ncldata.dev%2F&
