Re: [slurm-users] Steps to upgrade slurm for a patchlevel change?

2023-09-28 Thread Ryan Novosielski
I started off writing there’s really no particular process for these/just do your changes and start the new software (be mindful of any PATH that might contain data that’s under your software tree, if you have that setup), and that you might need to watch the timeouts, but I figured I’d have a l

Re: [slurm-users] Steps to upgrade slurm for a patchlevel change?

2023-09-28 Thread Ole Holm Nielsen
On 9/28/23 17:58, Groner, Rob wrote: There's 14 steps to upgrading slurm listed on their website, including shutting down and backing up the database.  So far we've only updated slurm during a downtime, and it's been a major version change, so we've taken all the steps indicated. We now want

Re: [slurm-users] Steps to upgrade slurm for a patchlevel change?

2023-09-28 Thread David
A colleague of mine has it scripted out quite well, so I can't speak to *all* of the details. However, we have a user that we submit our jobs as and it does the steps for upgrading (yum, dnf, etc). The jobs are wholenode/exclusive so nothing else can run there, and then a few other steps might be t

Re: [slurm-users] enabling job script archival

2023-09-28 Thread Paul Edmon
Yes it was later than that. If you are 23.02 you are good.  We've been running with storing job_scripts on for years at this point and that part of the database only uses up 8.4G.  Our entire database takes up 29G on disk. So its about 1/3 of the database. We also have database compression whic

Re: [slurm-users] enabling job script archival

2023-09-28 Thread Paul Edmon
No, all the archiving does is remove the pointer.  What slurm does right now is that it creates a hash of the job_script/job_env and then checks and sees if that hash matches one on record. If not then it adds it to the record, if it does match then it adds a pointer to the appropriate record. 

Re: [slurm-users] enabling job script archival

2023-09-28 Thread Ryan Novosielski
Sorry for the duplicate e-mail in a short time: do you know (or anyone) when the hashing was added? Was planning to enable this on 21.08, but we then had to delay our upgrade to it. I’m assuming later than that, as I believe that’s when the feature was added. On Sep 28, 2023, at 13:55, Ryan Nov

Re: [slurm-users] enabling job script archival

2023-09-28 Thread Ryan Novosielski
Thank you; we’ll put in a feature request for improvements in that area, and also thanks for the warning? I thought of that in passing, but the real world experience is really useful. I could easily see wanting that stuff to be retained less often than the main records, which is what I’d ask for

Re: [slurm-users] enabling job script archival

2023-09-28 Thread Paul Edmon
Slurm should take care of it when you add it. So far as horror stories, under previous versions our database size ballooned to be so massive that it actually prevented us from upgrading and we had to drop the columns containing the job_script and job_env.  This was back before slurm started ha

[slurm-users] enabling job script archival

2023-09-28 Thread Davide DelVento
In my current slurm installation, (recently upgraded to slurm v23.02.3), I only have AccountingStoreFlags=job_comment I now intend to add both AccountingStoreFlags=job_script AccountingStoreFlags=job_env leaving the default 4MB value for max_script_size Do I need to do anything on the DB mysel

[slurm-users] Steps to upgrade slurm for a patchlevel change?

2023-09-28 Thread Groner, Rob
There's 14 steps to upgrading slurm listed on their website, including shutting down and backing up the database. So far we've only updated slurm during a downtime, and it's been a major version change, so we've taken all the steps indicated. We now want to upgrade from 23.02.4 to 23.02.5. O