Hello, Changing slurmctld port should probably wait until all jobs have stopped running. Running jobs won't fail in this case, but there is a good chance they will fail to complete properly, and the compute node operating them might get stuck in the completing state (since the slurmstepd operating the job would fail to communicate properly, owing to the change in port).
Changing a parameter like MaxJobCount should be safe in my opinion, possibly even with just a change to slurm.conf followed by `scontrol reconfigure`; though if that doesn't work you might need to restart slurmctld. -Doug ---- Doug Jacobsen, Ph.D. NERSC Computer Systems Engineer National Energy Research Scientific Computing Center <http://www.nersc.gov> [email protected] ------------- __o ---------- _ '\<,_ ----------(_)/ (_)__________________________ On Tue, May 16, 2017 at 2:22 AM, Mark S. Holliman <[email protected]> wrote: > > Hi everyone, > > Does anyone know if changing the slurmctld settings for MaxJobCount and > SlurmctldPort will cause jobs already running/waiting to fail? My users > have hit the default 10,000 queue limit, and I'd like to increase that, but > not if it's going to kill everything that's running. I know most settings > can be changed (and slurmctld/slurmd restarted) without issue. But given > that scheduler changes can cause existing jobs to get killed I'm uncertain > about these parameters... > > Cheers, > Mark > > --------------------------------------- > Mark Holliman > Wide Field Astronomy Unit > Institute for Astronomy > University of Edinburgh > --------------------------------------- > > >
