Hi Gary,
We wrote some wrapper scripts to pull the limits from the db and the usage from
the *_usage files, and give you a ``balance''. For example:
[user@headnode01 scripts]$ sbank balance statement -u myuser01
User Usage | Account Usage | Account Limit Available (CPU
hrs)
---------- --------- + -------------- --------- + ------------- ---------
myuser02 20 | HPC_11_00204 441 | 400,000 399,559
myuser01 * 0 | HPC_11_00204 441 | 400,000 399,559
myuser03 421 | HPC_11_00204 441 | 400,000 399,559
myuser01 * 713 | HPC_13_00778 813 | 300,000 299,187
myuser04 100 | HPC_13_00778 813 | 300,000 299,187
It's available here:
https://github.com/paddydoyle/slurm-bank
(the docs need a bit of a tidy-up)
We use 'AccountingStorageEnforce=safe', and have 'PriorityDecayHalfLife=14-0'
and 'PriorityUsageResetPeriod=NONE'; so we can set limits but the usage decays
over time and allows people to run again when they have enough balance.
Thanks,
Paddy
On Wed, Feb 08, 2017 at 04:14:50PM -0800, Skouson, Gary B wrote:
Using sacctmgr you can set limits like GrpCPUMins and other GrpTRESMins. It's
pretty easy to see what the limit is, but I'm not sure how to see how close to
the limit someone is. Is there a normal slurm command that can get you the
internal GrpTRESMins info slurmctld is using to enforce limits?
Looking at the code, the share request from sshare gets sent all the data, but
there's no option to print it out.
I made a simple patch below to add a GrpTRESRaw option to sshare to print this
info. It has to convert the usage from double to int, which isn't ideal, but
it was simple and is close enough for what I was looking for.
Is there a better way to get this information?
-----
Gary Skouson
diff -Naru slurm-16.05.9/src/sshare/process.c
slurm-16.05.9.change/src/sshare/process.c
--- slurm-16.05.9/src/sshare/process.c 2017-01-31 11:55:41.000000000 -0800
+++ slurm-16.05.9.change/src/sshare/process.c 2017-02-08 15:45:08.019494347
-0800
@@ -63,6 +63,7 @@
{10, "User", print_fields_str, PRINT_USER},
{30, "GrpTRESMins", _print_tres, PRINT_TRESMINS},
{30, "TRESRunMins", _print_tres, PRINT_RUNMINS},
+ {30, "GrpTRESRaw", _print_tres, PRINT_GRPTRESRAW},
{0, NULL, NULL, 0}
};
@@ -226,6 +227,7 @@
char *tmp_char = NULL;
char *local_acct = NULL;
print_field_t *field = NULL;
+ uint64_t tres_raw[tres_cnt];
if ((options & PRINT_USERS_ONLY) && share->user == 0)
continue;
@@ -342,6 +344,14 @@
share->tres_grp_mins,
(curr_inx == field_count));
break;
+ case PRINT_GRPTRESRAW:
+ /* convert to ints and minutes */
+ for (i=0; i<tres_cnt; i++)
+ tres_raw[i] = (uint64_t)
share->usage_tres_raw[i]/60;
+ field->print_routine(field,
+ tres_raw,
+ (curr_inx == field_count));
+ break;
case PRINT_RUNMINS:
/* convert to minutes */
for (i=0; i<tres_cnt; i++)
diff -Naru slurm-16.05.9/src/sshare/sshare.h
slurm-16.05.9.change/src/sshare/sshare.h
--- slurm-16.05.9/src/sshare/sshare.h 2017-01-31 11:55:41.000000000 -0800
+++ slurm-16.05.9.change/src/sshare/sshare.h 2017-02-08 15:45:08.023488846
-0800
@@ -115,7 +115,8 @@
PRINT_RAWU,
PRINT_RUNMINS,
PRINT_USER,
- PRINT_LEVELFS
+ PRINT_LEVELFS,
+ PRINT_GRPTRESRAW
};
extern int exit_code; /* sshare's exit code, =1 on any error at any time */
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
