Check firewall rules or network comms in both directions. We had an issue with asymmetric routing between our slurmdbd and slurmctld and so connections could only be initiated one way. However, restarting slurmdbd would restart the connection and resync the latest state (or something like that, it was a few years ago)
> On 14 Oct 2018, at 21:49, Dave Botsch <bot...@cnf.cornell.edu> wrote: > > This seems to reflect what I am seeing. Someone earlier mentioned > multiple restarts of slurmdbd... those restarts never made data appear > unless right around on the hour. > > It's as if instead of data getting sent right through slurmdbd that > something in slurmdbd is just doing an hourly check of the text based > sacct records (which I don't understand why those are even there if not > configured in slurm.conf). > > Thanks. > >> On Sun, Oct 14, 2018 at 12:08:10PM +0100, Antony Cleave wrote: >> I have noticed on several clusters that sreport can be upto one hour out of >> date i.e. it will update on the hour every hour. >> >> sacct does not behave this way and is always up to date. >> >> I cannot see this stated in the docs or see any config settings to control >> this but it happens on the last 17.02 cluster I checked. >> >> Antony >> >> On 14 Oct 2018 11:58, "Steven Dick" <kg4...@gmail.com> wrote: >> >> It is documented that you need to create the cluster in the database. >> >> It is not documented that the accounting system won't work until you >> restart slurmdbd multiple times before it starts collecting accounting >> records. >> >> Also, none of the necessary restarts are needed on an upgrade -- only >> when slurm is initialized for a new cluster. >> >> >> On Sun, Oct 14, 2018 at 4:12 AM Ole Holm Nielsen >> <ole.h.niel...@fysik.dtu.dk> wrote: >>> Correct, and this is documented in the Slurm accounting setup page: >>> https://slurm.schedmd.com/accounting.html#database-configuration >>> > > -- > ******************************** > David William Botsch > Programmer/Analyst > @CNFComputing > bot...@cnf.cornell.edu > ******************************** >
