We just use the Archive function built into slurm. That has worked fine
for us for the past 6 years. We keep 6 months of data in the active
archive.
If you have 6 years worth of data and you want to prune down to 2 years,
I recommend going month by month rather than doing it in one go. When
we initially started archiving data several years back our first pass at
archiving (which at that time had 2 years of data in it) took forever
and actually caused issues with the archive process. We worked with
SchedMD, improved the archive script built into Slurm but also decided
to only archive one month at a time which allowed it to get done in a
reasonable amount of time.
The archived data can be pulled into a different slurm database, which
is what we do for importing historic data into our XDMod instance.
-Paul Edmon-
On 7/13/2022 4:55 PM, Timony, Mick wrote:
Hi Slurm Users,
Currently we don't archive our SlurmDB and have 6 years' worth of data
in our SlurmDB. We are looking to start archiving our database as it
starting to get rather large, and we have decided to keep 2 years'
worth of data. I'm wondering what approaches or scripts other groups use.
The docs refer to the ArchiveScript setting at:
https://slurm.schedmd.com/slurmdbd.conf.html#OPT_ArchiveScript
I've seen suggestions to import into another database that will
require keeping the schema up-to-date which seems like a possible
maintenance issue or nightmare if one forgets to update the schema
after updating Slurmdb. We also have most of the information in an
Elasticsearch <https://slurm.schedmd.com/elasticsearch.html> instance,
which will likely suite our needs for long term historical information.
What do you use to archive this information? CSV files, SQL dumps or
something else?
Regards
--
Mick Timony
Senior DevOps Engineer
Harvard Medical School
--
