I'm just catching up on old mailing list messages now. Why not make your
SuspendProgram and ResumePrograms be shell scripts that look at some
node information in Slurm (look at the features as in your example) or
some other source ( use a case statement based on node names) and call
the correct suspend/resume command based on that?
I agree that attaching this metadata in the node definition and have
slurm act on it directly is the best solution, but in the meantime,
having a shell script that can figure out the correct way to
suspend/resume each host should be very doable, if not ideal.
Prentice
On 11/8/22 09:36, Ole Holm Nielsen wrote:
I'm thinking about the best way to configure power saving (see
https://slurm.schedmd.com/power_save.html) when we have different
types of node hardware whose power state have to be managed differently:
1. Nodes with a BMC NIC interface where "ipmitool chassis power ..."
commands can be used.
2. Nodes where the BMC cannot be used for powering up due to the
shared NICs going down when the node is off :-(
3. Cloud nodes where special cloud CLI commands must be used (such as
Azure CLI).
The slurm.conf only permits one SuspendProgram and one ResumeProgram
which then need to figure out the cases listed above and perform
appropriate actions.
I was thinking to add a node feature to indicate the kind of power
control mechanism available, for example along these lines for the 3
above cases:
Nodename=node001 Feature=power_ipmi
Nodename=node002 Feature=power_none
Nodename=node003 Feature=power_azure
The node feature might be inquired in the SuspendProgram and
ResumeProgram and jump to separate branches of the script for power
control commands.
Question: Has anyone thought of a similar or better way to handle
power saving for different types of nodes?
Thanks,
Ole
