1. Part of the communications for slurm is hierarchical. Thus nodes
need to know about other nodes so they can talk to each other and
forward messages to the slurmctld.
2. Yes, this is what we do. We have our slurm.conf shared via NFS from
our slurm master and then we just update that single conf. After that
update we then use salt to issue a global restart to all the slurmd's
and slurmctld to pick up the new config. scontrol reconfigure is not
enough when adding new nodes, you have to issue a global restart.
3. It's pretty straight forward all told. You just need to update the
slurm.conf and do a restart. You need to be careful that the names you
enter into the slurm.conf are resolvable by DNS, else slurmctld may barf
on restart. Sadly no built in sanity checker exists that I am aware of
aside from actually running slurmctld. We got around this by putting
together a gitlab runner which screens our slurm.conf's by running
synthetic slurmctld to sanity check.
-Paul Edmon-
On 4/27/2021 2:35 PM, David Henkemeyer wrote:
Hello,
I'm new to Slurm (coming from PBS), and so I will likely have a few
questions over the next several weeks, as I work to transition my
infrastructure from PBS to Slurm.
My first question has to do with *_adding nodes to Slurm_*. According
to the FAQ (and other articles I've read), you need to basically shut
down slurm, update the slurm.conf file /*on all nodes in the
cluster*/, then restart slurm.
- Why do all nodes need to know about all other nodes? From what I
have read, its Slurm does a checksum comparison of the slurm.conf file
across all nodes. Is this the only reason all nodes need to know
about all other nodes?
- Can I create a symlink that points <sysconfdir>/slurm.conf to a
slurm.conf file on an NFS mount point, which is mounted on all the
nodes? This way, I would only need to update a single file, then
restart Slurm across the entire cluster.
- Any additional help/resources for adding/removing nodes to Slurm
would be much appreciated. Perhaps there is a "toolkit" out there to
automate some of these operations (which is what I already have for
PBS, and will create for Slurm, if something doesn't already exist).
Thank you all,
David
