Yes, I agree with you. The TRES stuff is nice, but seems to be
added somewhat quickly, and in particular I do not see anyway to
view that information. I am looking forward to applying your sshare
patch. (I attempted once to write a patch similar to yours but had
issues I could not resolve in the little time I had (maybe I
ommited the long to int conversion you mentioned). And I hope to
also add support for that in the Slurm::Sshare perl module and the
sbalance command.
On Mon, 13 Feb 2017, Skouson, Gary B wrote:
The problem I have with sshare, is that it won't report raw usage, other than
"RawUsage" which is only cpu-seconds. While I can set other GrpTRESMins
limits. And slurm will enforce those limits, sshare wasn't sharing the raw usage info for
those other limits.
This patch adds an additional output format option to sshare, called
GrpTRESRaw. It formats stuff the same way that GrpTRESMins formats the limits.
That should allow a user to see more than just the GrpCPUMins limits that
might be set and how much utilization there has been.
For example:
$ sshare -A mscfops -p -o account,user,RawUsage,GrpTRESRaw,GrpTRESMins
Account|User|RawUsage|GrpTRESRaw|GrpTRESMins|
mscfops||117670512|cpu=1961175,mem=0,energy=0,node=122573|node=3000000|
mscfops|skouson|73615920|cpu=1226932,mem=0,energy=0,node=76683||
-----
Gary Skouson
-----Original Message-----
From: Paddy Doyle [mailto:[email protected]]
Sent: Monday, February 13, 2017 6:46 AM
To: slurm-dev <[email protected]>
Subject: [slurm-dev] Re: Accounting and limits
Hi Gary,
We wrote some wrapper scripts to pull the limits from the db and the usage from
the *_usage files, and give you a ``balance''. For example:
[user@headnode01 scripts]$ sbank balance statement -u myuser01
User Usage | Account Usage | Account Limit Available (CPU
hrs)
---------- --------- + -------------- --------- + ------------- ---------
myuser02 20 | HPC_11_00204 441 | 400,000 399,559
myuser01 * 0 | HPC_11_00204 441 | 400,000 399,559
myuser03 421 | HPC_11_00204 441 | 400,000 399,559
myuser01 * 713 | HPC_13_00778 813 | 300,000 299,187
myuser04 100 | HPC_13_00778 813 | 300,000 299,187
It's available here:
https://github.com/paddydoyle/slurm-bank
(the docs need a bit of a tidy-up)
We use 'AccountingStorageEnforce=safe', and have 'PriorityDecayHalfLife=14-0'
and 'PriorityUsageResetPeriod=NONE'; so we can set limits but the usage decays
over time and allows people to run again when they have enough balance.
Thanks,
Paddy
On Wed, Feb 08, 2017 at 04:14:50PM -0800, Skouson, Gary B wrote:
Using sacctmgr you can set limits like GrpCPUMins and other GrpTRESMins. It's
pretty easy to see what the limit is, but I'm not sure how to see how close to
the limit someone is. Is there a normal slurm command that can get you the
internal GrpTRESMins info slurmctld is using to enforce limits?
Looking at the code, the share request from sshare gets sent all the data, but
there's no option to print it out.
I made a simple patch below to add a GrpTRESRaw option to sshare to print this
info. It has to convert the usage from double to int, which isn't ideal, but
it was simple and is close enough for what I was looking for.
Is there a better way to get this information?
-----
Gary Skouson
diff -Naru slurm-16.05.9/src/sshare/process.c
slurm-16.05.9.change/src/sshare/process.c
--- slurm-16.05.9/src/sshare/process.c 2017-01-31 11:55:41.000000000 -0800
+++ slurm-16.05.9.change/src/sshare/process.c 2017-02-08 15:45:08.019494347
-0800
@@ -63,6 +63,7 @@
{10, "User", print_fields_str, PRINT_USER},
{30, "GrpTRESMins", _print_tres, PRINT_TRESMINS},
{30, "TRESRunMins", _print_tres, PRINT_RUNMINS},
+ {30, "GrpTRESRaw", _print_tres, PRINT_GRPTRESRAW},
{0, NULL, NULL, 0}
};
@@ -226,6 +227,7 @@
char *tmp_char = NULL;
char *local_acct = NULL;
print_field_t *field = NULL;
+ uint64_t tres_raw[tres_cnt];
if ((options & PRINT_USERS_ONLY) && share->user == 0)
continue;
@@ -342,6 +344,14 @@
share->tres_grp_mins,
(curr_inx == field_count));
break;
+ case PRINT_GRPTRESRAW:
+ /* convert to ints and minutes */
+ for (i=0; i<tres_cnt; i++)
+ tres_raw[i] = (uint64_t)
share->usage_tres_raw[i]/60;
+ field->print_routine(field,
+ tres_raw,
+ (curr_inx == field_count));
+ break;
case PRINT_RUNMINS:
/* convert to minutes */
for (i=0; i<tres_cnt; i++)
diff -Naru slurm-16.05.9/src/sshare/sshare.h
slurm-16.05.9.change/src/sshare/sshare.h
--- slurm-16.05.9/src/sshare/sshare.h 2017-01-31 11:55:41.000000000 -0800
+++ slurm-16.05.9.change/src/sshare/sshare.h 2017-02-08 15:45:08.023488846
-0800
@@ -115,7 +115,8 @@
PRINT_RAWU,
PRINT_RUNMINS,
PRINT_USER,
- PRINT_LEVELFS
+ PRINT_LEVELFS,
+ PRINT_GRPTRESRAW
};
extern int exit_code; /* sshare's exit code, =1 on any error at any time */
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Tom Payerle
IT-ETI-EUS [email protected]
4254 Stadium Dr (301) 405-6135
University of Maryland
College Park, MD 20742-4111
