The problem I have with sshare, is that it won't report raw usage, other than "RawUsage" which is only cpu-seconds. While I can set other GrpTRESMins limits. And slurm will enforce those limits, sshare wasn't sharing the raw usage info for those other limits.
This patch adds an additional output format option to sshare, called GrpTRESRaw. It formats stuff the same way that GrpTRESMins formats the limits. That should allow a user to see more than just the GrpCPUMins limits that might be set and how much utilization there has been. For example: $ sshare -A mscfops -p -o account,user,RawUsage,GrpTRESRaw,GrpTRESMins Account|User|RawUsage|GrpTRESRaw|GrpTRESMins| mscfops||117670512|cpu=1961175,mem=0,energy=0,node=122573|node=3000000| mscfops|skouson|73615920|cpu=1226932,mem=0,energy=0,node=76683|| ----- Gary Skouson -----Original Message----- From: Paddy Doyle [mailto:[email protected]] Sent: Monday, February 13, 2017 6:46 AM To: slurm-dev <[email protected]> Subject: [slurm-dev] Re: Accounting and limits Hi Gary, We wrote some wrapper scripts to pull the limits from the db and the usage from the *_usage files, and give you a ``balance''. For example: [user@headnode01 scripts]$ sbank balance statement -u myuser01 User Usage | Account Usage | Account Limit Available (CPU hrs) ---------- --------- + -------------- --------- + ------------- --------- myuser02 20 | HPC_11_00204 441 | 400,000 399,559 myuser01 * 0 | HPC_11_00204 441 | 400,000 399,559 myuser03 421 | HPC_11_00204 441 | 400,000 399,559 myuser01 * 713 | HPC_13_00778 813 | 300,000 299,187 myuser04 100 | HPC_13_00778 813 | 300,000 299,187 It's available here: https://github.com/paddydoyle/slurm-bank (the docs need a bit of a tidy-up) We use 'AccountingStorageEnforce=safe', and have 'PriorityDecayHalfLife=14-0' and 'PriorityUsageResetPeriod=NONE'; so we can set limits but the usage decays over time and allows people to run again when they have enough balance. Thanks, Paddy On Wed, Feb 08, 2017 at 04:14:50PM -0800, Skouson, Gary B wrote: > > Using sacctmgr you can set limits like GrpCPUMins and other GrpTRESMins. > It's pretty easy to see what the limit is, but I'm not sure how to see how > close to the limit someone is. Is there a normal slurm command that can get > you the internal GrpTRESMins info slurmctld is using to enforce limits? > > Looking at the code, the share request from sshare gets sent all the data, > but there's no option to print it out. > > I made a simple patch below to add a GrpTRESRaw option to sshare to print > this info. It has to convert the usage from double to int, which isn't > ideal, but it was simple and is close enough for what I was looking for. > > Is there a better way to get this information? > > ----- > Gary Skouson > > diff -Naru slurm-16.05.9/src/sshare/process.c > slurm-16.05.9.change/src/sshare/process.c > --- slurm-16.05.9/src/sshare/process.c 2017-01-31 11:55:41.000000000 -0800 > +++ slurm-16.05.9.change/src/sshare/process.c 2017-02-08 15:45:08.019494347 > -0800 > @@ -63,6 +63,7 @@ > {10, "User", print_fields_str, PRINT_USER}, > {30, "GrpTRESMins", _print_tres, PRINT_TRESMINS}, > {30, "TRESRunMins", _print_tres, PRINT_RUNMINS}, > + {30, "GrpTRESRaw", _print_tres, PRINT_GRPTRESRAW}, > {0, NULL, NULL, 0} > }; > > @@ -226,6 +227,7 @@ > char *tmp_char = NULL; > char *local_acct = NULL; > print_field_t *field = NULL; > + uint64_t tres_raw[tres_cnt]; > > if ((options & PRINT_USERS_ONLY) && share->user == 0) > continue; > @@ -342,6 +344,14 @@ > share->tres_grp_mins, > (curr_inx == > field_count)); > break; > + case PRINT_GRPTRESRAW: > + /* convert to ints and minutes */ > + for (i=0; i<tres_cnt; i++) > + tres_raw[i] = (uint64_t) > share->usage_tres_raw[i]/60; > + field->print_routine(field, > + tres_raw, > + (curr_inx == > field_count)); > + break; > case PRINT_RUNMINS: > /* convert to minutes */ > for (i=0; i<tres_cnt; i++) > diff -Naru slurm-16.05.9/src/sshare/sshare.h > slurm-16.05.9.change/src/sshare/sshare.h > --- slurm-16.05.9/src/sshare/sshare.h 2017-01-31 11:55:41.000000000 -0800 > +++ slurm-16.05.9.change/src/sshare/sshare.h 2017-02-08 15:45:08.023488846 > -0800 > @@ -115,7 +115,8 @@ > PRINT_RAWU, > PRINT_RUNMINS, > PRINT_USER, > - PRINT_LEVELFS > + PRINT_LEVELFS, > + PRINT_GRPTRESRAW > }; > > extern int exit_code; /* sshare's exit code, =1 on any error at any time */ -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
