We've been using the multifactor plugin to enforce accounting with grpcpumin limits for accounts. That's worked well with a homogeneous node configuration. We've historically allocated time by node-hours, but with all nodes being identical, it's pretty easy to do a uniform conversion from cpu-min to node-hours.
We're thinking about introducing some new nodes into the cluster that will have different characteristics, however, we'd still like to charge node-hours, regardless of the node configuration. That's not going to work with using grpcpumin with a uniform conversion. Looking in the documentation, it looks like I can set a TRESBillingWeights on a partition to charge differently on each partition, or set standard billing weight to "CPU=0,node=1". What's unclear to me is where to look for the amount that gets "billed". Looking at the output of sshare, I can see that the incremental "RawUsage" increases the correct amount based on the TRESBillingWeight, but I couldn't find the "billing" info for individual jobs. I was hoping that there was something in sacct like ConsumedEnergyRaw, or CPUTimeRAW that would show the TRESBilling for a job, but wasn't able to find anything like that. I can certainly calculate that for a job based on what I put in the slurm.conf file for billingweights. If I change the partition TRESBillingWeight values, does that only change the billing for the future, or will the billing of past jobs be updated to match the new billing weights? Also, I can set a grptresmins with sacctmgr, but I'm unsure how to get a measure of what has actually been used. It looks like slurm stores TRES usage information in the assoc_usage file, and I can certainly parse that myself, but I was hoping to find the normal slurm command that would show that information. ----- Gary Skouson
