Tim, I believe you have to refer to the gpu as gres/gpu. [root@slurm-login ~]# sreport -T CPU,mem,gres/gpu cluster AccountUtilizationByUser Start=2017-01-01 End=2017-12-31
-------------------------------------------------------------------------------- Cluster/Account/User Utilization 2017-01-01T00:00:00 - 2017-10-06T14:59:59 (24069600 secs) Use reported in TRES Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name TRES Name Used --------- --------------- --------- --------------- -------------- -------- deadpool root cpu 228 deadpool root mem 192008 deadpool root gres/gpu 259 deadpool hpcstaff cpu 228 deadpool hpcstaff mem 192008 deadpool hpcstaff gres/gpu 259 deadpool hpcstaff danbarke Daniel Barker cpu 198 deadpool hpcstaff danbarke Daniel Barker mem 160947 deadpool hpcstaff danbarke Daniel Barker gres/gpu 259 -Dan On Fri, Oct 6, 2017 at 3:12 PM, Tim Carlson <tim.s.carl...@gmail.com> wrote: > Background: Recently installed new cluster which I started with 14.03 but > then upgraded to 17.02 to get better/more gres/tres information. > > In my other clusters I use sreport heavily to do billing and this new > cluster is GPU based and I want to bill off of GPU time consumed. My > assumption was I could use something like > > sreport cluster AccountUtilizationByUser -T cpu,gpu Start=2017-10-06 > End=2017-10-08 > > I think I have slurm.conf configured correctly > > # grep -i gres /etc/slurm/slurm.conf > AccountingStorageTRES=gres/gpu > GresTypes=gpu > NodeName=dl[01-25] Gres=gpu:2 Feature=ml01 Procs=16 State=UNKNOWN > > And sacct seems to report the gres/tres utilization. > > # sacct -X -u tim --format=jobid,elapsed,ReqTRES%30,ReqGRES > --starttime=2017-10-06 | tail > 314 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 315 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 316 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 317 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 318 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 319 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 320 00:05:27 cpu=1,node=1,gres/gpu=1 gpu:1 > 321 00:05:24 cpu=1,node=1,gres/gpu=1 gpu:1 > 322 00:05:24 cpu=1,node=1,gres/gpu=1 gpu:1 > 323 00:05:24 cpu=1,node=1,gres/gpu=1 gpu:1 > > > # sreport cluster AccountUtilizationByUser -T gpu Start=2017-10-06 > End=2017-10-08 > ------------------------------------------------------------ > -------------------- > Cluster/Account/User Utilization 2017-10-06T00:00:00 - 2017-10-06T11:59:59 > (43200 secs) > Use reported in TRES Minutes > ------------------------------------------------------------ > -------------------- > Cluster Account Login Proper Name TRES Name Used > --------- --------------- --------- --------------- -------------- -------- > > Yet if I ask for cpu time from the tres field I get what I want. > > # sreport cluster AccountUtilizationByUser -T cpu,gpu Start=2017-10-06 > End=2017-10-08 > ------------------------------------------------------------ > -------------------- > Cluster/Account/User Utilization 2017-10-06T00:00:00 - 2017-10-06T11:59:59 > (43200 secs) > Use reported in TRES Minutes > ------------------------------------------------------------ > -------------------- > Cluster Account Login Proper Name TRES Name Used > --------- --------------- --------- --------------- -------------- -------- > marianas root cpu 143 > marianas ops cpu 143 > marianas ops tim Tim Carlson cpu 143 > > Bottom line being, what am I missing to get sreport to kick out gpu time? > > -- Dan Barker ARC-TS
