I actually just answered this question the other day. Take a look in the archives. The gist of it appears to be that slurmdbd will fix that address to be whatever address slurmctld uses to connect to it.
There’s also a bug/enhancement related to this a few point releases back on 15.08 that addresses this. You don’t say your SLURM version or OS/version. > On Jan 10, 2017, at 5:04 PM, Riccardo Murri <[email protected]> wrote: > > > Hello, > > we have an issue with (apparently) slurmdbd->slurmctld communication. > > Both daemons are running on the same machine, which got reinstalled last > week and got a new IP address. The new IP address has been written in > `slurm.conf`, and every thing seems to work fine -- except new > associations, created after the change, are not picked up by the > `slurmctld`: jobs are refused with a "wrong account/partition" error. > > It looks like a slurmd->slurmctld communication error because: > > - new associations, accounts, etc. are created and queried correctly > after creation by `sacctmgr` (and values persist across restart of > `slurmdbd` and I can see them in the MySQL DB); > > - however `sacctmgr show cluster` shows the *wrong* error for the > cluster controller address (say, 192.168.0.1) -- this address is > listed nowhere in the configuration file, but it's the first IP > address found by enumerating eth* interfaces on the host (eth0 is > 192.168.0.1, eth1 is 10.1.2.3). > > - the `slurmctld` is instead listening on another IP address (e.g., > 10.1.2.3 on port 6817) > > - everything works fine if I forward traffic directed to > 192.168.0.1:6817 to 10.1.2.3:6817 with `iptables`:: > > iptables -t nat -A PREROUTING -d 192.168.0.1 -p tcp --dport 6817 -j > DNAT --to-destination 10.1.2.3:6817 > > How can I correct the cluster controller address? I have tried, to no > avail, to stop both slurmctld and slurmdbd, then fix the address in > MySQL, then restart slurm daemons. Every time the cluster control > address is changed back to 192.168.0.1. > > What am I doing wrong? > > Thanks, > R > > -- > Riccardo Murri, Schwerzenbacherstrasse 2, CH-8606 Nänikon, Switzerland -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - [email protected] || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'
signature.asc
Description: Message signed with OpenPGP using GPGMail
