On 5/28/19 1:51 AM, Alexandre wrote:
> I have a product I need to backup data for.
> It stores data in a "contentstore" which is structured in a way that
> make it very predictable what needs to be backed up every day.
> Let me explain, every time a content is created it is stored under
> this specific hierarchy:
>
> contentstore_root
> |_YYYY (e.g. 2019)
> |_mm (1...12)
> |_dd (1...31)
> |_HH (1...24)
> |_MM (1...59)
> ...
Alexandre,
I do have an idea. I don't know if it will help you or not. It
definitely needs to be experimented with, and vetted.
Here we go:
Use a script to generate the list of directories to backup, and shove
that into the FileSet definition. Your script could just specify the
current date's directories, and those of some number of days before and
after the current date (eg +/- 5 days). Then those would be the only
directories that the bacula FD would traverse, to do the backup on.
Here's an example of the fileset syntax that includes a script that
outputs the newline-separated list of files:
> FileSet {
> Name = "FilesetName"
> Include {
> Options {
> signature = SHA1
> compression = GZIP
> }
> File = "\\|/path/to/script/on/fd"
> }
> }
In this scenario, since FileSet definition doesn't actually change, it
won't upgrade your incremental to a full, for example, as long as an
older full exists. Yes, the output of the script itself does change,
but that doesn't seem to be enough to trigger the automatic level upgrade.
There are a few enormous questions, though:
- What happens when a previously-backed-up directory, is no longer
listed in the output of the script? Do those files remain available to
restore, or are they treated as if they were deleted?
- Does the size of your ongoing VirtualFull keep growing over time? Is
that going to be a problem?
- Is there ever a time when you no longer want to be able to restore old
files? Do you need to have some kind of manual file expiration mechanism?
- Would it make sense to sometimes include old directories, after
they've been emptied, so that your job can figure out that the files are
moved to the trash folder instead? For example, if files are trashed
after, say, 90 days, then your script could output the list of new
directories to backup, and a list of 5 directories that are 90-95 days old.
Good luck,
Lloyd
--
Lloyd Brown
HPC Systems Administrator
Office of Research Computing
Brigham Young University
http://marylou.byu.edu
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
