So I was going to take a stab at trying to rectify this after taking care of post-holiday matters.
Paste of the $CLUSTER_job_table table where I think I see the issue, and now I just want to sanity check my steps to remediate. https://rentry.co/qhw6mg <https://rentry.co/qhw6mg> (pastebin alternative because markdown is paywalled for pastebin). There are a number of job steps with a timelimit of 4294967295, where as the others of the same job array are 525600. Obviously I want to edit those time limits to sane limits (match them to the others). I don’t see anything in the $CLUSTER_step_table that looks like it would need to be modified to match, though I could be wrong. But then the part of getting slurm to pick it up is where I’m wanting to make sure I’m on the right page. Should I manually update the mod_time timestamp and slurm will catch that at its next rollup? Or will slurm catch the change in the time limit at update the mod_time when it sees it upon rollup? I also don’t see any documentation stating how to manually trigger a rollup, either via slurmdbd.conf or command line flag. Will it automagically perform a rollup at some predefined, non-configurable interval, or when restarting the daemon? Apologies if this is all trivial information, just trying to measure twice and cut once. Appreciate everyone’s help so far. Thanks, Reed > On Dec 23, 2022, at 7:18 PM, Chris Samuel <ch...@csamuel.org> wrote: > > On 20/12/22 6:01 pm, Brian Andrus wrote: > >> You may want to dump the database, find what table/records need updated and >> try updating them. If anything went south, you could restore from the dump. > > +lots to making sure you've got good backups first, and stop slurmdbd before > you start on the backups and don't restart it until you've made the changes, > including setting the rollup times to be before the jobs started to make sure > that the rollups include these changes! > > When you start slurmdbd after making the changes it should see that it needs > to do rollups and kick those off. > > All the best, > Chris > -- > Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
