On 17/08/16 23:36, Ole Holm Nielsen wrote: > Obviously upgrading slurmd's which are running jobs is quite tricky! I > have some questions: > > 1. Can't you replace the health check by a global scontrol like this? > scontrol update NodeName=<nodelist> State=drain Reason="Upgrading > slurmd"
Yes you could, but as we keep the configs for our healthchecks in git it means we have a historical record of the changes made. > 2. Do you really have to wait for *all* nodes to become drained before > starting to upgrade? This could take weeks! No, that's the opposite of what we do. We update the healthcheck, the nodes drain themselves, then as nodes become idle we upgrade the slurmd on them, the healthcheck sees its the new version and puts itself back into service. So it's a rolling upgrade. Our next outage does require us to have no nodes running jobs, but that's just because we're changing a parameter in slurm.conf that requires there to be no running jobs first. cheers, Chris -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: [email protected] Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci
