You could enable debug logging on your slurm controllers to see if that provides some more useful info. I'd also check your firewall settings to make sure your not blocking some traffic that you shouldn't. iptables -Fâ will clear your local Linux firewall.
I'd also triple check the UID on all the systems and run this on all your compute nodes, slurm controllers, and slurmdb to make sure it is the same! đ id 59999â I'd also restart all the slurm daemons all the systems to make sure that you don't have systems that running a daemon from before you created UID 59999 as running processes often don't pick up changes like that unless they're restarted. Cheers -- Mick Timony Senior DevOps Engineer Harvard Medical School -- ________________________________ From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Craig Stark <cest...@ad.uci.edu> Sent: Monday, January 8, 2024 5:46 PM To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error This ticket with SchedMD implies it's a munged issue: https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__;!!CzAuKJ42GuquVTTmVmPViYEvSg!N2M1a84yfU8mhdQ87LnBMQxye_nBsrTzTow7spIqZaQ2dLevBDZy4oNMT8KzMsmhxdRwchIht3Tgl3p8cMHhFOg9ry546OQ_iA$ Is the munge daemon running on all systems? If it is, are all servers running a network time daemon such chronyd or ntpd and the time is in sync on all hosts? Thanks Mick, munge is seemingly running on all systems (systemctl status munge). I do get a warning about the munge file changing on disk, but I'm pretty sure that's from warewulf sync'ing files every minute. A sha256sum on the munge.key file on the compute nodes and host node says they're the same, so I think I can put that aside. The management node runs chrony and the compute nodes sync to the management node. [root@kirby uber]# chronyc tracking Reference ID : 4A06A849 (t2.time.gq1.yahoo.com) Stratum : 3 Ref time (UTC) : Mon Jan 08 22:26:44 2024 System time : 0.000032525 seconds slow of NTP time Last offset : -0.000021390 seconds RMS offset : 0.000055729 seconds Frequency : 38.797 ppm slow Residual freq : +0.001 ppm Skew : 0.018 ppm Root delay : 0.033342984 seconds Root dispersion : 0.000524800 seconds Update interval : 256.8 seconds Leap status : Normal vs [root@sonic01 ~]# chronyc tracking Reference ID : C0A80102 (warewulf) Stratum : 4 Ref time (UTC) : Mon Jan 08 22:31:02 2024 System time : 0.000000120 seconds slow of NTP time Last offset : -0.000000092 seconds RMS offset : 0.000014737 seconds Frequency : 47.495 ppm slow Residual freq : +0.000 ppm Skew : 0.066 ppm Root delay : 0.033458963 seconds Root dispersion : 0.000283949 seconds Update interval : 64.2 seconds Leap status : Normal So, the compute node is talking to the host and the host is talking to generic NTP sources. "date" shows the same time on the compute nodes
