Hello,
we have an issue with (apparently) slurmdbd->slurmctld communication.
Both daemons are running on the same machine, which got reinstalled last
week and got a new IP address. The new IP address has been written in
`slurm.conf`, and every thing seems to work fine -- except new
associations, created after the change, are not picked up by the
`slurmctld`: jobs are refused with a "wrong account/partition" error.
It looks like a slurmd->slurmctld communication error because:
- new associations, accounts, etc. are created and queried correctly
after creation by `sacctmgr` (and values persist across restart of
`slurmdbd` and I can see them in the MySQL DB);
- however `sacctmgr show cluster` shows the *wrong* error for the
cluster controller address (say, 192.168.0.1) -- this address is
listed nowhere in the configuration file, but it's the first IP
address found by enumerating eth* interfaces on the host (eth0 is
192.168.0.1, eth1 is 10.1.2.3).
- the `slurmctld` is instead listening on another IP address (e.g.,
10.1.2.3 on port 6817)
- everything works fine if I forward traffic directed to
192.168.0.1:6817 to 10.1.2.3:6817 with `iptables`::
iptables -t nat -A PREROUTING -d 192.168.0.1 -p tcp --dport 6817 -j
DNAT --to-destination 10.1.2.3:6817
How can I correct the cluster controller address? I have tried, to no
avail, to stop both slurmctld and slurmdbd, then fix the address in
MySQL, then restart slurm daemons. Every time the cluster control
address is changed back to 192.168.0.1.
What am I doing wrong?
Thanks,
R
--
Riccardo Murri, Schwerzenbacherstrasse 2, CH-8606 Nänikon, Switzerland
