I am confused by the reported amount of Down and PLND Down by sreport.
According to it, our cluster would have had a significant amount of
downtime, which I know didn't happen (or, according to the documentation
"time that slurmctld was not responding", see
https://slurm.schedmd.com/sreport.html)
Could it be my purge settings causing this problem? How can I check (maybe
in some logs, maybe in the future) if actually slurmctld was not
responding? The expected long-term numbers should be less than the ones
reported for last month when we had an issue with a few nodes....
Thanks!
[davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf
#JobPurge=12
#StepPurge=1
PurgeEventAfter=1month
PurgeJobAfter=12month
PurgeResvAfter=1month
PurgeStepAfter=1month
PurgeSuspendAfter=1month
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization
start=2/1/22
--------------------------------------------------------------------------------
Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
Cluster TRES Name Allocated Down PLND Down
Idle Planned Reported
--------- -------------- --------------- --------------- --------------
---------------- --------- ----------------
cluster cpu 19.50% 12.07% 3.92%
64.36% 0.15% 100.03%
cluster mem 16.13% 13.17% 4.56%
66.13% 0.00% 99.99%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization
start=2/1/23
--------------------------------------------------------------------------------
Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
Cluster TRES Name Allocated Down PLND Down
Idle Planned Reported
--------- -------------- --------------- --------------- --------------
--------------- --------- ----------------
cluster cpu 28.74% 18.80% 6.44%
45.77% 0.24% 100.02%
cluster mem 22.52% 20.54% 7.38%
49.55% 0.00% 99.98%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization
start=2/1/24
--------------------------------------------------------------------------------
Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
Cluster TRES Name Allocated Down PLND Down
Idle Planned Reported
--------- -------------- -------------- --------------- --------------
--------------- -------- ---------------
cluster cpu 29.92% 24.88% 17.73%
27.45% 0.02% 100.00%
cluster mem 20.07% 28.60% 19.57%
31.76% 0.00% 100.00%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization
start=8/8/24
--------------------------------------------------------------------------------
Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
Cluster TRES Name Allocated Down PLND Dow Idle
Planned Reported
--------- -------------- ------------- ------------ -------- --------------
-------- --------------
cluster cpu 15.96% 2.53% 0.00% 81.51%
0.00% 100.00%
cluster mem 9.18% 2.22% 0.00% 88.60%
0.00% 100.00%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization
start=7/7/24
--------------------------------------------------------------------------------
Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
Cluster TRES Name Allocated Down PLND Dow
Idle Planned Reported
--------- -------------- -------------- ------------- --------
-------------- -------- --------------
cluster cpu 27.07% 2.57% 0.00%
70.34% 0.02% 100.00%
cluster mem 17.35% 2.26% 0.00%
80.40% 0.00% 100.00%
--
slurm-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
