Re: [slurm-users] Questions about adding new nodes to Slurm

2021-05-04 Thread Ole Holm Nielsen
The task of adding or removing nodes from Slurm is well documented and discussed in SchedMD presentations, please see my Wiki page https://wiki.fysik.dtu.dk/niflheim/SLURM#add-and-remove-nodes /Ole On 04-05-2021 14:47, Tina Friedrich wrote: Not sure if that's changed but aren't there cases wh

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-05-04 Thread Tina Friedrich
Not sure if that's changed but aren't there cases where 'scontrol reconfigure' isn't sufficient? (Like adding nodes?) But yes, that's my point exactly; it is a pretty basic day to day task to update slurm.conf, not some daunting operation that requires a downtime or anything like it. (I rememb

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-05-04 Thread Sid Young
You can push a new conf file and issue an "scontrol reconfigure" on the fly as needed... I do it on our cluster as needed, do the nodes first then login nodes then the slurm controller... you are making a huge issue of a very basic task... Sid On Tue, 4 May 2021, 22:28 Tina Friedrich, wrote: >

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-05-04 Thread Tina Friedrich
Hello, a lot of people already gave very good answer to how to tackle this. Still, I thought it worth pointing this out - you said 'you need to basically shut down slurm, update the slurm.conf file, then restart'. That makes it sound like a major operation with lots of prep required. It's no

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-04-28 Thread Ole Holm Nielsen
On 4/28/21 2:48 AM, Sid Young wrote: I use SaltStack to push out the slurm.conf file to all nodes and do a "scontrol reconfigure" of the slurmd, this makes management much easier across the cluster. You can also do service restarts from one point etc. Avoid NFS mounts for the config, if the mou

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-04-27 Thread Sid Young
Hi David, I use SaltStack to push out the slurm.conf file to all nodes and do a "scontrol reconfigure" of the slurmd, this makes management much easier across the cluster. You can also do service restarts from one point etc. Avoid NFS mounts for the config, if the mount locks up your screwed. htt

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-04-27 Thread Max Voit
On Tue, 27 Apr 2021 11:35:18 -0700 David Henkemeyer wrote: > - Can I create a symlink that points /slurm.conf to a > slurm.conf file on an NFS mount point, which is mounted on all the > nodes? This way, I would only need to update a single file, then > restart Slurm across the entire cluster. Yo

Re: [slurm-users] Questions about adding new nodes to Slurm

2021-04-27 Thread Paul Edmon
1. Part of the communications for slurm is hierarchical.  Thus nodes need to know about other nodes so they can talk to each other and forward messages to the slurmctld. 2. Yes, this is what we do.  We have our slurm.conf shared via NFS from our slurm master and then we just update that single

[slurm-users] Questions about adding new nodes to Slurm

2021-04-27 Thread David Henkemeyer
Hello, I'm new to Slurm (coming from PBS), and so I will likely have a few questions over the next several weeks, as I work to transition my infrastructure from PBS to Slurm. My first question has to do with *adding nodes to Slurm*. According to the FAQ (and other articles I've read), you need t