Hi Rob, I'm just replying to show you that I read everything you're suggesting, but all this information requires me to think more deeply. Concerning FileRetention and JobRetention, what you say seems perfectly logical to me and that's how I reasoned, but in the bacula guide, I read the rule: file retention <= job retention <= volume retention and that's what troubled me.
For now, I had understood that "The only other alternative is to reduce what is backed up" and I'm looking more precisely at what I can delete from the backups to lighten them. The ideal for me is to fit these backups in the space available, without adding an additional brick. I'll come back when I've made progress. Thanks again! Le jeudi 09 janvier 2025 à 13:24 -0600, Rob Gerber a écrit : > I had another idea. What if you try bacula progressive virtual full > backups? > > I do not know if you have heard of this method. You take a full > backup, then do incremental backups for a period of time, like one > month. Then you consolidate the previous full backup with the > incremental backups. This gives you a new full backup containing the > changes from the incrementals and the history from the original full > backup. I have been told that bacula enterprise can do this > consolidation purely in the catalog, but that bacula community must > read the old backups and write a new full backup. > > I am sure you are thinking that this doesn't solve your problem > because for a period of time you would have the old backups and the > new backups on disk at the same time. You would be correct, except > for one thing: > What if you did the consolidation for only one FD at a time? This way > the most you could have on disk at any one time would be 6 FD backups > + the new consolidated backup being made for one of the FDs. You > would have to use scripts or commands to ensure that the old source > backups were pruned right away. > > I do not have a lot of knowledge about progressive virtual full > backups beside what I just shared here. I may be missing some > important information. > Read more about them here: https://www.bacula.org/whitepapers/PVF.pdf > > Even if you did not do progressive virtual full, and instead went to > Full >> incrementals for each month instead of Full Inc Diff Inc Diff > Inc Diff like you currently plan, this could save space because the > Diffs would not be repeating the same data over and over again. You > would have to analyze to see if this could make a difference. > > In the fileset, you can increase the compression level for GZIP and > ZSTD. The default value gives good compression / performance > tradeoff. If you want maximum compression, you could try setting the > value to maximum compression. Look in the manual for details. Be > aware that this may not give you big improvements and will cost more > CPU. > > If you aren't using deduplication, progressive virtual full backups, > or simply increasing your storage space to allow for the backups, > then I am not sure what else could be done. The only other > alternative is to reduce what is backed up. Maybe do less frequent > backups? > > Additionally, if you were considering s3 compatible cloud backups as > a cheaper alternative to expanding your hosted servers, you may be > interested to know that bacula 15.0.2 now supports encrypting file > volumes. The only thing left unencrypted is the volume header. The > encryption CPU load is applied to the SD since the SD does the > encryption. If your SD cannot afford this CPU load, then you could > consider FD encryption. In this case, the filenames and metadata > would not be encrypted. > > Oh! One thing about s3 compatible cloud backups: you must store the > data collected from the FDs locally for a brief period, at least > until it is uploaded. There is a setting to immediately truncate this > cached data after upload. This could save space because at absolute > worst you could have the complete cache for a full backup for all 6 > FDs at once, but this is still better than 12 full backups at once. I > doubt you would have that much cache on disk at once, especially if > you staggered your full backups. > > Regarding FileRetention and JobRetention: just set them to be >= > VolumeRetention. Bacula will prune a volume with no jobs on it, but > will not prune a job with no file records. In some cases admins might > set a smaller file retention period to delete catalog records for > files in older backups (to improve catalog performance / size), but > the data would still be accessible if the job was restored. Bacula > will automatically remove File and Job records for volumes that are > pruned. > > Retention periods set in the pool override all other retention > periods specified elsewhere. > > Regards, > Robert Gerber > 402-237-8692 > r...@craeon.net > > > On Thu, Jan 9, 2025 at 11:49 AM Rob Gerber <r...@craeon.net> wrote: > > I think there will no problem having one set of pools per FD, but > > maybe it increases complexity for you. Otherwise, no space savings. > > > > I have a couple ideas. > > > > 1. Bacula has the 'aligned' plugin. If you use the aligned plugin, > > it attempts to better align blocks found on disk in the volumes. It > > does this so deduplication filesystems can work on the bacula > > volumes. I think that your full backups are probably very similar > > to each other. So maybe deduplication could help here? > > > > 2. Bacula has a cloud storage plugin. Maybe you can store the > > backups in a lower cost s3 compatible file storage service? > > > > 3. If your FDs are similar to each other or based on a common > > image, maybe you could back up only essential services, with the > > intent to apply the image, then restore backups to "freshen" the > > image. Maybe something like Relax And Recover? > > > > I think idea 2 is probably easier and/or safer safer than the > > others. > > > > Robert Gerber > > 402-237-8692 > > r...@craeon.net > > > > On Thu, Jan 9, 2025, 11:15 AM Christophe PEREZ via Bacula-users > > <bacula-users@lists.sourceforge.net> wrote: > > > Hi Rob, > > > > > > I understood well since our first exchange that the > > > MaximumVolumes at 1 > > > was a heresy. I also understood well that to best manage several > > > volumes, it remained important to separate Full, Diff and Inc in > > > different pools. So, we are returning substantially to what I > > > already > > > have. > > > On the other hand, sincerely, I do not see what it would bring me > > > in > > > terms of occupied space to group the Full of my FDs in a single > > > pool > > > instead of having a pool per FD, the same for Diff and Inc. > > > > > > What I know is that 2 Full Volumes + 3 (or 4) Diff Volumes + 2 > > > Volume > > > of 6 days of Inc, for the 2 FDs that I save on this SD (the 4 > > > other FDs > > > are saved on another SD), it will be very tight with the space > > > that I > > > have. And I cannot unfortunately increase the space because it is > > > a > > > hosted server whose increase in space is too expensive. > > > I would have preferred to avoid getting into something more > > > complicated > > > by forcing the expiration of volumes by admin job or something > > > else. > > > I need to find a concrete solution to save my 30 days of 2 FDs on > > > this > > > SD. > > > I must also admit that I don't know what to put optimally in: > > > File Retention = ?? days > > > Job Retention = ?? days > > > for clients. > > > > > > In any case, thank you for all the efforts you make to help me. > > > Even if > > > I'm not sure I understand everything. > > > I'm still thinking about all this and if anyone else wants to > > > shed some > > > light, don't hesitate. > > > > > > Regarding ZSTD, I confirm that it is already installed, called by > > > many > > > packages. I configured bacula to use it. > > > > > > Le mercredi 08 janvier 2025 à 17:27 -0600, Rob Gerber a écrit : > > > > Replies below. :) > > > > Regards, > > > > Robert Gerber > > > > 402-237-8692 > > > > r...@craeon.net > > > > > > > > > > > > On Wed, Jan 8, 2025 at 4:03 PM Christophe PEREZ via Bacula- > > > > users > > > > <bacula-users@lists.sourceforge.net> wrote: > > > > > Le mercredi 08 janvier 2025 à 14:46 -0600, Rob Gerber a > > > > > écrit : > > > > > > Christophe, > > > > > > > > > > Hi Rob, > > > > > > > > > > > > > > > Not 2nd AND 5th, but 2nd TO 5th. > > > > > > > > > > > > > That makes more sense. I understand now. > > > > > > > > > > > > > > > > MaximumVolumes="2" can only work if you have only 1 FD. > > > > > > Really, > > > > > > for > > > > > > this to work the right way you must have MaximumVolumes=X > > > > > > where X > > > > > > = > > > > > > (total FD x 2). > > > > > > > > > > And save all FD in the same pool ? > > > > > > > > > > > > > I would want to put Full backups in the Full pool, Diff in the > > > > diff > > > > pool, etc. Different pool for different backup type and > > > > different > > > > retention, but not different pools for different FD unless you > > > > have a > > > > very good reason to do it. So, per storage medium, you will > > > > have 3 > > > > total pools, Full, Diff, Inc. You do not want to mix media > > > > types in a > > > > pool, so if you were backing up to cloud, to another SD, or to > > > > tape > > > > you would want a Full, Diff, and Inc pool for each storage > > > > medium or > > > > device. In my 15.0.2 device, I have local backup pools > > > > Synology- > > > > Local-Full, Synology-Local-Diff, and Synology-Local-Inc, and > > > > pools > > > > B2-Full, B2-Diff, B2-Inc for copying jobs from the local pools > > > > to the > > > > Backblaze B2 cloud service. > > > > > It seems to me that this will not gain me any space. > > > > > > > > > > > > > > > > > > I am trying to say that for you to use the MaximumVolumes=X > > > > idea > > > > with MaximumVolumeJobs=1, X must be >= the number of volumes > > > > you > > > > expect to use within your retention period. Let us say you have > > > > 6 FD, > > > > MaximumVolumes=6, MaximumVolumeJobs=1, VolumeRetention=40days, > > > > and > > > > you do 1 full backup each month. In month 1 there is no problem > > > > - > > > > bacula does its backups and creates 6 volumes. In month 2, > > > > Bacula > > > > cannot make any volumes available without deleting the volumes > > > > from > > > > last month BEFORE you have done any new backups! In my > > > > understanding > > > > bacula would not violate the retention period, so it would be > > > > unable > > > > to make the new backups. So in this example you would want at > > > > least > > > > MaximumVolumes=12 with MaximumVolumeJobs=1. You don't want to > > > > run out > > > > of volumes before your retention period expires. > > > > > > > > There are other ways to limit bacula's space usage. I'll talk > > > > about > > > > them. > > > > > > > > > > So, with Maximum Volume Jobs = 0, the volume size will > > > > > continue to > > > > > grow? > > > > > > > > > > > > > I don't know what would happen if we set MaximumVolumeJobs=0 , > > > > but if > > > > we comment it out or remove it, yes, the volume will grow > > > > forever > > > > unless restrained in some other way. > > > > Bacula gives us many ways to restrain volume growth. We can set > > > > MaximumVolumeUseDuration, MaximumVolumeJobs, > > > > MaximumVolumeBytes, > > > > etc. > > > > See the manual for some more discussion of each. > > > > https://www.bacula.org/15.0.x-manuals/en/main/Configuring_Director.html#SECTION00231600000000000000000 > > > > > So I have to find something else... > > > > > I thought I had made my life easier like that. > > > > > > > > > > > > > I think MaximumVolumeJobs is good for your situation. Maybe > > > > MaximumVolumes is difficult if not set correctly. If you > > > > experience > > > > unexpected blockages where a job cannot run because the pool > > > > does not > > > > have any more volumes, then you can experiment with removing > > > > MaximumVolumes and instead relying on the VolumeRetention > > > > period to > > > > remove volumes. > > > > > > > > As a warning if relying on VolumeRetention alone, it is > > > > possible that > > > > bacula will not remove a volume if it does not have to, even if > > > > expired. Bacula's goal is to keep your volumes available as > > > > long as > > > > possible, just in case you need them. If that becomes a > > > > problem, I > > > > believe you can Schedule a job of Type=Admin and tell it to > > > > truncate > > > > expired volumes. I have not set up such a job, so I am not > > > > certain > > > > how it is done. > > > > > > > > I think MaximumVolumes could work for you, as long as it is set > > > > to be > > > > > = the number of volumes you will need during your retention > > > > > period. > > > > > > > > Here is the bacula manual entry all about automatic volume > > > > recycling. > > > > It is very useful. > > > > https://www.bacula.org/15.0.x-manuals/en/main/Automatic_Volume_Recycling.html > > > > > > > > One extract from that link above that explains what bacula will > > > > do if > > > > not restrained in some way: > > > > " A key point mentioned above, that can be a source of > > > > frustration, > > > > is that Bacula will only recycle purged Volumes if there is no > > > > other > > > > appendable Volume available, otherwise, it will always write to > > > > an > > > > appendable Volume before recycling even if there are Volumes > > > > marked > > > > as Purged. This preserves your data as long as possible. So, if > > > > you > > > > wish to “force” Bacula to use a purged Volume, you must first > > > > ensure > > > > that no other Volume in the Pool is marked Append. If > > > > necessary, you > > > > can manually set a volume to Full. The reason for this is that > > > > Bacula > > > > wants to preserve the data on your old Volumes (even though > > > > purged > > > > from the catalog) as long as absolutely possible before > > > > overwriting > > > > it. There are also a number of directives such as Volume Use > > > > Duration > > > > that will automatically mark a volume as Used and thus no > > > > longer > > > > appendable." > > > > > > > > This behavior is why we must restrain bacula. Your suggestion > > > > of > > > > MaximumVolumeJobs=1 and > > > > MaximumVolumes=$expectedNumberOfVolumesNeededDuringRetentionPer > > > > iod > > > > accomplishes this, with the warning that you will not be able > > > > to make > > > > new backups if you run out of volumes because you added more > > > > FDs but > > > > did not increase MaximumVolumes, or something like that. > > > > > > > > I have set a very short retention period on my test system, > > > > with a > > > > job scheduled once an hour. I'll examine the behavior I see and > > > > report back. > > > > > > > > > I have bacula 15.0.2. > > > > > What linux tool does this ZSTD compression format depend on? > > > > > > > > > > > > > I don't know what tool it uses. I am also running bacula > > > > 15.0.2, on > > > > Rocky Linux 9. For me, the ZSTD feature just worked. I think > > > > bacula > > > > made sure the correct libraries are present. > > > > > > > > -- Christophe PEREZ _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
