Brian, FWIW, we just restart slurmctld when this happens. I’ll be interested to hear if there’s a proper fix.
Andy From: slurm-users [mailto:slurm-users-boun...@lists.schedmd.com] On Behalf Of Brian Andrus Sent: Thursday, July 18, 2019 11:01 AM To: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: [slurm-users] sacct thinks slurmctld is not up All, I have slurmdbd running and everything is (mostly) happy. It's been working well for months, but fairly regularly, when I do 'sacctmgr show runaway jobs', I get: sacctmgr: error: Slurmctld running on cluster orion is not up, can't check running jobs if I do 'sacctmgr show cluster', it lists the cluster but has no IP in the ControlHost field. slurmctld is most definitely running (on the same system even), but the only fix I find is to restart slurmctld. Then I can check and there is an IP in the ControlHost field and I am able to check for runawayjobs. Is this a known issue? Is there a better fix than restarting slurmctld? Brian Andrus
