On 5/24/23 11:39 am, Fulton, Ben wrote:
Hi,
Hi Ben,
The release notes for 23.02 say “Added usage gathering for gpu/nvml
(Nvidia) and gpu/rsmi (AMD) plugins”.
How would I go about enabling this?
I can only comment on the nvidia side (as those are the GPUs we have)
but for that you need Slurm built with NVML support and running with
"Autodetect=NVML" in gres.conf and then that information is stored in
slurmdbd as part of the TRES usage data.
For example to grab a job step for a test code I ran the other day:
csamuel@perlmutter:login01:~> sacct -j 9285567.0 -Pno TRESUsageInAve |
tr , \\n | fgrep gpu
gres/gpumem=493120K
gres/gpuutil=76
Hope that helps!
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
