I totally get it. Gotta test and absolutely don't want to hose up your NAS.
It's been interesting reading your emails to see how you're approaching
your bacula installation.
Something you might not be aware of:
Bacula by default preserves information from pruned backups (backups that
exceeded their relevant retention time period) as long as possible until
they must be overwritten to make space for new data.
My understanding is that bacula will prefer scratch volumes over previously
pruned volumes until it has no other choice than to overwrite previously
pruned volumes. I believe it also uses a first in, first out approach when
overwriting pruned volume data.
You'll have to review the bacula manual for more information and
confirmation.
I do vaguely recall seeing mention of a process for bacula to reclaim disk
space when using hard drive volumes, as you are. I'm not sure when this is
invoked. I think the way hard drive volumes are treated could be slightly
different from the way tape volumes are treated. Basically, a physical tape
volume that has fallen outside its retention period will be pruned if
automatic recycling is turned on, and ultimately reused. A hard drive
volume in the same situation could just be deleted and a new volume with a
new volume number created. Again, double check this behavior in the bacula
manual and confirm it in practice. I have a slight concern that this
process of deleting old hard drive volumes could lead to different behavior
than I described when using tape volumes.
I've only ever seen mention of retention times being used to determine the
longevity of jobs, volumes, and file records. I haven't seen any sort of
decision making like you described. The default behavior of reusing pruned
volumes only when necessary and on a fifo basis does seem to come close,
though not as granular as you mentioned.
You might be able to vary the retention periods of different job levels,
either in the job resource or perhaps by using different pools for
different job levels (relying on pool volume retention settings, which
should override all other retention settings elsewhere).
There could be a way to do what you want in bacula, but I suspect you
might have to write it yourself, which might not be what you'd want to do.
This could be one of those situations where one might have to adapt their
goals to match what's possible in the software.
I think with the way bacula handles reusing pruned volumes, you might have
similar results assuming sufficient free space and backups as scheduled
don't demand so much storage that they immediately overwrite any pruned
volumes. I also wonder if a setting exists that specifies a maximum number
of volumes. I think bacula might regard free space available on a
filesystem as permission to create a new volume instead of deleting an old
volume and creating a new one. Such a situation implies that a maximum
volumes parameter might be useful. With infinite automatically growing
storage like you might have on koofr, bacula might not ever reuse a pruned
volume that has passed it's retention period. Also interesting is that a
finite filesystem full of bacula volumes (many of which are past their
retention period and have been pruned) might stay consistently full,
incorrectly implying that bacula is about to run out of space when in
actuality it might just be retaining data for as long as possible.
I very much agree that manually purging and pruning volumes is undesirable.
Robert Gerber
402-237-8692
r...@craeon.net
On Wed, Jan 31, 2024, 3:56 PM Myles Dear <smd...@hotmail.com> wrote:
> Thanks for the guidance, Rob.
>
> My restrictions are in place because I simply want to kick the tires
> within a limited space. In my case, I'm backing up to Dropbox, where I
> keep a myriad of other data than backup data, and that space is limited.
> I've had situations in which I used 100% of backup space on my QNAP NAS and
> then the drive stopped working (later firmware updates made it easier to
> recover the drive if this happened). Either way, I want a way to ensure
> backups keep rolling while older backups are pruned in an intelligent
> manner.
>
> I'm considering moving my backup space off to a larger volume such as
> Koofr, where I can purchase large amounts of storage space in perpetuity
> and then connect to Bacula via rclone mounts on a Linux system.
>
> As mentioned in other emails to this list, my goal is to eventually ensure
> differing backup densities, while still keeping a minimum number of
> historical backups, for example one backup every other month for backups
> older than a year, one backup per month for backups older than six months
> but newer than a year, two backups per week for backups older than three
> months, and weekly backups for the past three months, for example.
>
> I'll keep chipping away at this and will report back when I find a
> configuration that works better for me. I really don't want to be forced
> to go in manually and remove old backup files and jobs as I wish this to be
> done automatically, as long as my backup retention policy similar to the
> example I gave is followed.
>
> Thanks,
>
> <Myles>
> On 2024-01-28 9:57 p.m., Rob Gerber wrote:
>
> I don't know of a way to make bacula backups always fit within a set
> amount of space. This could be dangerous. Depends on how far back you want
> your backup data to go. If such a thing was possible using bacula, you
> wouldn't want bacula to delete the last full backup or you wouldn't have
> any valid backups for a given host. Even assuming bacula could do this and
> was smart enough to not delete the latest full backup, you could encounter
> a situation where you previously had a certain amount of data retention
> (say, 2 months), but as backup sizes increased that time slowly went down
> until you only had 1 week of retention. Could be very dangerous, esp if
> ransomware struck some of your clients. In the event of ransomware, you
> will absolutely want backups going back further than 2 weeks. Also, a usage
> based purge of backup data could be initiated by the ransomware encrypting
> many files on your customer systems, resulting in a rapidly ballooning
> backup size as your server backed up the encrypted files. In a worst case
> scenario, a system configured the way you described could be prompted to
> throw out its last good backups of a system by the influx of data from a
> new ransomware infection.
>
> Now on to discussion of your current issue:
> If I recall correctly, you first started messaging this list within 2023.
> I suspect that no volume is older than 365 days, and as such, no volume
> will be reused, end of story. The volume, file, and job retention options
> are all time based. If you've hit 100GB of utilization before 365 days was
> up, you will need to change one of several things:
>
> Increase available storage space.
> Decrease storage utilization. For instance, maybe do full backups less
> frequently, relying on differential and incremental backups for a longer
> period of time.
> Decrease your volume/job/file retention.
>
> With the exception of increasing your storage space, as far as I know your
> situation requires that you manually purge older volumes. I strongly advise
> caution when doing this. The purge command is DANGEROUS. Running it without
> parameters will delete all your database records for whichever file daemon
> you select. Take catalog backups and restore those backups to a safe place
> before doing anything related to purging or deleting existing volumes.
> Understand the database restore process BEFORE you need it. If possible set
> up a test environment to test any proposed changes of this scale.
>
> Manually changing the volume retention period on a per volume basis might
> make the volumes valid targets for pruning. Not sure. What I am pretty sure
> about is that changing the default volume retention period in your config
> won't retroactively change that value for your volumes that already exist.
>
> Double check everything I've said here. My knowledge is limited, and I may
> be incorrect on some points. Pretty sure on the first paragraph diagnosing
> your issue and your list of options, though.
>
> You can confirm the validity of my diagnosis by checking the dates
> associated with your volumes.
>
> As a matter of opinion, your 1GB volumes seem small, though that's not
> necessarily something I can back up with anything other than opinion. I do
> imagine that smaller 1GB volumes would be easier to export from cloud
> hosted storage (which I think you are using, iirc). Don't feel pressured to
> change this unless it bothers you or you have other knowledge supporting my
> opinion. Changing maximum volume sizes could impact access to and use of
> existing volumes. Be careful.
>
> Regards,
> Robert Gerber
> 402-237-8692
> r...@craeon.net
>
>
> On Sat, Jan 27, 2024 at 8:33 AM MylesDearBusiness via Bacula-users <
> bacula-users@lists.sourceforge.net> wrote:
>
>> Hello,
>>
>> I'm having some problems with backup file configuration.
>>
>> I want the backup file area to consume 100GB and no more.
>>
>> I decided on file sizes of 1GB
>>
>> Pool {
>>
>> ...
>>
>> Volume Retention = 365 days # one year
>> Maximum Volume Bytes = 1G # Limit Volume size to something
>> reasonable
>> Maximum Volumes = 100 # Limit number of Volumes in Pool
>>
>> Recycle = yes # Bacula can automatically recycle
>> Volumes
>> AutoPrune = yes # Prune expired volumes
>>
>> ...
>>
>> }
>>
>> I enabled the following schedule:
>>
>> #
>> # When to do the backups, full backup on first sunday of the month,
>> # differential (i.e. incremental since full) every other sunday,
>> # and incremental backups other days
>> Schedule {
>> Name = "WeeklyCycle"
>> Run = Full 1st sun at 23:05
>> Run = Differential 2nd-5th sun at 23:05
>> Run = Incremental mon-sat at 23:05
>> }
>>
>> ... and what I saw is 100 files written and filled, and then all backups
>> stopping in their tracks and no further backups produced.
>>
>> Every new job errors out.
>>
>> None of the older backup files were reused.
>>
>> Backups only resumed when I manually deleted the oldest volumes from
>> bconsole, and stopped again when the maximum number of files had been
>> reached
>>
>> I want automatic space management so that when the maximum number of
>> volumes are created, the oldest ones are reused.
>>
>> I don't know how to set the "Volume Retention" accurately (as my backup
>> sizes are not predictable, as I install new software or accumulate business
>> data their size will likely increase),
>> I want the backups to always fit within my prescribed amount of storage
>> space and recycle that space when needed, without manual intervention.
>>
>> How can I achieve this ?
>>
>> Thanks.
>>
>>
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
