On 08/03/2016 03:04 AM, Christopher Samuel wrote:
So you always go in the order of upgrading:
* slurmdbd
* slurmctld
[recompile all plugins, MPI stacks, etc that link against Slurm]
* slurmd
We use a health check script that defines the version of Slurm that is
considered production so we can just bump that number first, wait for
all the compute nodes to be marked as drained and then as nodes become
idle we can start restarting slurmd knowing that we will never get a job
that spans both old and new slurmd's.
Obviously upgrading slurmd's which are running jobs is quite tricky! I
have some questions:
1. Can't you replace the health check by a global scontrol like this?
scontrol update NodeName=<nodelist> State=drain Reason="Upgrading
slurmd"
2. Do you really have to wait for *all* nodes to become drained before
starting to upgrade? This could take weeks!
3. Is it OK to upgrade subsets of nodes after they become drained?
4. I assume that upgraded nodes can be returned to the IDLE state by:
scontrol update NodeName=<nodelist> State=resume
Could you possibly elaborate on the steps which you described?
FYI, I'm trying to capture this advice in my Wiki:
https://wiki.fysik.dtu.dk/niflheim/SLURM#upgrading-on-centos-7
Thanks,
Ole
