Re: [Bacula-users] Puring/Pruning doesn't seem to be happening.

[Gabriele Bulfon]

At last, I think I discovered the basic problem behind the Automatic Recycle/Purge/Prune so many people are facing (me included).

I decided to start from a scratch situation, with the latest configuration I came up with, using all the suggestions I received in this list.
As a first step, I just used bconsole to purge all my 7 weekly volumes, assuming that any previous job would have been deleted and removed from the database.
With a lot of surprise, I discovered that some jobs were still in the bconsole list. Don't know why.
So I decided to delete them by hand (delete job).
Finally I had a scratch situation....I thought...
But I wanted to be sure.
So I stopped bacula daemons, ran psql on bacula database (I am using Postgres) and issued
some sql statements to examine the database status:
- a "select count(*) from job" was never returning....(or probably returning  after a long time)
- a "select count(*) from file" was never returining.....(or probably returning  after an even longer time)
So I supposed that both tables should have been very very large and still containing hidden data!
Now, to get my hands on its data, I used pg_dump to dump all the bacula db on a flat file and see its content.
......many minutes of dump........300Mb of flat file was produced.....
...wow.....and there, I found a lot of data inside the job and file tables.

So, I decided to drop the whole bacula db and recreate it.
After doing this, my latest configuration files work perfectly!

So, my last suggestion to anyone going crazy with their weekly backup, is to start from scratch
any time you need to change the bacula-dir.conf file, then "add" the pre-labaled volumes by hand.
Bacula won't prune or recycle some of your tapes, because old jobs are probably still in the db
in some hidden status.

I hope somone can clarify why this situation occurs.
Thanks a lot.
Gabriele.

Gabriele Bulfon - Sonicle S.r.l. Tel +39 028246016 Int. 30 - Fax +39 028243880 Via Felice Cavallotti 16 - 20089, Rozzano - Milano - ITALY http://www.sonicle.com

----------------------------------------------------------------------------------

Da: Arno Lehmann <[EMAIL PROTECTED]>
A: bacula-users@lists.sourceforge.net
Data: 28 dicembre 2005 21.04.50 CET
Oggetto: Re: [Bacula-users] Puring/Pruning doesn't seem to be happening.

Hello,

On 12/28/2005 6:11 PM, Beren Gamble wrote:
> Hi Arno,
>
> Sorry I didn't reply to the list... That's me on autopilot :D - Hope you had a Merry Xmas!

No problem... and thanks, I did have some nice days.

> I have it working perfectly, EXCEPT the only problem i've now got, is when Bacula looks for volumes to use, it chooses Recycled volumes over Purged volumes. In the hope that i'd be able to force Bacula to use the tape in my drive, I've manually set all of my volumes to Used.. even the ones that have never been used, but some of the volumes have never had a successful backup to them, so bacula sees that no jobs are associated with a volume, so it marks it Recycled, then of course, it chooses the Recycled volume to backup to. During this process, Bacula marks the volume in my drive as Purged, as it should, but it ignores it.
>
> The empty volumes are the stick in the mud here.
>
> The reason there are empty volumes is because they may have failed any time within the last 2 weeks (we run a 2 week cycle).

Hmm. Ok, that sounds like some real problem to me :-)

IIRC, you're running something newer than 1.36.something. As far as I
know, with these versions, Bacula should always prefer any volume that
could be recycled (i.e. is purged) and that's already mounted over ones
that need extra intervention. That, at least, is one feature that's
working here since... about... forever ;-)

Now, you might check if you've got the Accept Any Volume directive set
in your config, and you could check what happens in a situatien where
- you've got a recyclable (or even recycled) volume in the tape drive,
- Bacula requests a different volume,
- and you simple issue a mount command.

Usually, in my experience, Bacula should start writing to the tape in
the drive. (I *think* this is a bug but I never investigated it - there
seems to be some potential for improving the recycling / pruning
algorithm. Basically, I suspect that Bacula, when it looks for a new
volume, first look for a pruned volume, then starts job pruning, after
that it prunes volumes, and thinks it's finished. I've seen situations
like this, and I thought a different sequence might be more useful.)

If the above does work, you should closely observe the sequence of
things Bacula does - run the DIR with debug output and see if you can
pinpoint the situation where it expects the wrong volume. I'm sure that,
during the next week, it's possible to prepare a perfecly detailed bug
reports with all sorts of necessary information ;-)

Arno

> Thanks,
>
> Beren
>
>
>
>>>>Arno Lehmann <[EMAIL PROTECTED]> 27/12/2005 22:17 >>>
>
> Hello,
>
> it took me a while to answer, partly due to the christmas days, partly
> due to you writing to me personally :-) anyway...
>
> On 12/23/2005 10:04 AM, Beren Gamble wrote:
>
>>Arno Lehmann Writes-----
>>
>>
>>
>>>>Well, only to repeat myself - it's not that Bacula is designed to
>>>>overwrite valid data, instead it tries to protect that data.
>>
>>
>>I fully appreciate this, but in cases where we have single tape units, I need a setting which overrides this and uses the tape that I supply. After all, if I put a tape in which is outside of the retention period, a retention period which "I" have set, I want it to be overwritten.. I don't want the data preserved. And it doesn't seem possible to do this wich Bacula.
>
>
> Hmm. I admit I still don't understand why anyone would want to overwrite
> a perfect backup volume, but using the right settings for recycling,
> maximum volume use duration and volume retention should allow what you want.
>
> Keep in mind that I haven't tried this, but allowing, for example 12
> hour of volume use time and setting a volume retention of 11 hours
> should have the result you want: If, after, one day, the volume is still
> loaded it can be recycled, and thus re-used.
>
> Hope this helps,
>
> and I guess I'll need to start some experiments when I've got more time
> available...
>
> Arno
>
>
>>
>>
>>
>>>>>Arno Lehmann <[EMAIL PROTECTED]> 23/12/2005 08:14 >>>
>>
>>Hello,
>>
>>On 12/23/2005 2:19 AM, Doug Sampson wrote:
>>
>>
>>>FWIW.
>>>
>>>My experience with using Bacula on tape drives is similar to Beren's. When
>>>used on HD server, Bacula runs like a dream. But when used with a server
>>>using tape drives, I am seeing Bacula wanting to append to volumes that
>>>aren't full when I've specificaly requested that Bacula reuse the tape
>>>currently in the tape drive. Bacula doesn't ask for a different tape but
>>>appends to the tape inserted.
>>
>>
>>Well, only to repeat myself - it's not that Bacula is designed to
>>overwrite valid data, instead it tries to protect that data.
>>
>>
>>
>>>I have been following this thread and making adjustments to the
>>>bacula-dir.conf file as I see fit. I've inserted the 'Volume Use Duration'
>>>parameter and am not seeing any changes that I expect. Volumes marked
>>>"Append" are still marked such as even after the Volume Use Duration time
>>>parameter has passed.
>>
>>
>>I assume you didn't see the following sentence in the manual, Director
>>configuration, in the section dealing with Volume Use Duration.
>>
>>"The use duration is checked and the Used status is set only at the end > of a job that writes to the particular volume, which means that even
>>though the use duration may have expired, the catalog entry will not be > updated until the next job that uses this volume is run."
>>
>>
>>
>>>Currently running 1.38.2.
>>
>>
>>Currenly not running 1.38.3 :-(
>>
>>Arno
>>
>>
>>
>>>~Doug
>>>
>>>
>>>
>>>
>>>>-----Original Message-----
>>>>From: Beren Gamble [mailto:[EMAIL PROTECTED]
>>>>Sent: Wednesday, December 21, 2005 08:34 AM
>>>>To: bacula-users@lists.sourceforge.net; [EMAIL PROTECTED]
>>>>Subject: Re: [Bacula-users] Puring/Pruning doesn't seem to be
>>>>happening.
>>>>
>>>>
>>>>Ok, maybe i'm not picking up a concept here in the
>>>>documentation.... I've now got my Volume Use Duration at 23h,
>>>>which works beautifully, marking the volume as Used.
>>>>
>>>>However, I just tried to run a backup job. All of my volumes
>>>>in the pool were set as Used. (If VolUsen didn't mark them
>>>>used, I did, just for uniformities' sake) But bacula then
>>>>found a volume which had no jobs associated with it and
>>>>decided to recycle that one instead of the volume that was in
>>>>the drive (it definitely is past the 12 day retention period).
>>>>
>>>>Bacula is an absolute dream with my autochanger because tapes
>>>>are only either marked as Full or, Purged or Recycle and I
>>>>never remove a half used tape. But on the single tape
>>>>systems, I still haven't figured out how to make bacula cope
>>>>with rotating half filled volumes.. or volumes where all
>>>>backups failed and no data was written.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>>>Kern Sibbald <[EMAIL PROTECTED]> 17/12/2005 10:56:45 >>>
>>>>
>>>>I think the basic problem that you have had, and it seems
>>>>like you don't quite
>>>>understand though you have found a solution, is that
>>>>recycling for Bacula requires applying two concepts: 1. the
>>>>time Bacula can write on a Volume, 2.
>>>>the time Bacula keeps that Volume before reusing it.
>>>>
>>>>Most people come with the pre-conceived idea that there is
>>>>one "Recycle" time,
>>>>when in fact Bacula is significantly more sophisticated than that.
>>>>
>>>>I've reviewed the chapter you refer to below, and if one
>>>>reads the full
>>>>chapter, all the information *is* there (at least IMO). I have added
>>>>additional text that stresses the above two concept points a
>>>>bit more as well as refer the user to the "Basic Volume
>>>>Management" chapter that gives a
>>>>detailed practical example of why you want to do. The
>>>>Recycle chapter is complete in itself assuming you want to
>>>>fill all your Volumes. If not, you
>>>>must also read the Basic Volume Management chapter as well.
>>>>
>>>>Concerning your comment:
>>>> "Can someone please update the documentation here
>>>> http://www.bacula.org/rel-manual/Automatic_Volume_Recycling.html
>>>> with this information, that should hopefully forestall
>>>>this question
>>>> coming up so much."
>>>>
>>>>I appreciate you pointing out where you got confused.
>>>>However, this is a free
>>>>software project, and I understand recycling, so I don't
>>>>easily see why some
>>>>people don't get the concepts or I would have already changed
>>>>the doc. As a
>>>>consequence it is much more helpful if you would submit a
>>>>suggested change to
>>>>the documentation that clarifies it for you.
>>>>
>>>>I suggest you look at the documentation in:
>>>>
>>>>http://www.bacula.org/dev-manual/Automatic_Volume_Recycling.html
>>>>
>>>>It is usually much more up to date.
>>>>
>>>>On Saturday 17 December 2005 07:00, Harondel J. Sibble wrote:
>>>>
>>>>
>>>>
>>>>>On 16 Dec 2005 at 1:25, Arno Lehmann wrote:
>>>>>
>>>>>
>>>>>
>>>>>>>....so....how does one force it to do it?!
>>>>>>
>>>>>>Apart from the configuration directives like "prune
>>>>
>>>>oldest volume" (or
>>>>
>>>>
>>>>
>>>>>>something) my advice is usually "you don't".
>>>>>
>>>>>Philisophical disagreement, If I as the admin decide that,
>>>>
>>>>that is an
>>>>
>>>>
>>>>
>>>>>appropriate thing to do, I should be able to do it in some
>>>>
>>>>manner. Reminds
>>>>
>>>>
>>>>
>>>>>me of the unix philosophy, build a bunch of single purpose
>>>>
>>>>tools that you
>>>>
>>>>
>>>>
>>>>>can chain together to do anything you want .....
>>>>>
>>>>>Anyhow, on the how to do it, here's what I got from an
>>>>
>>>>unrelated list where
>>>>
>>>>
>>>>
>>>>>I was ranting about the same problem the OP has as I had it
>>>>
>>>>for months
>>>>
>>>>
>>>>
>>>>>also.
>>>>>
>>>>>What you need is this
>>>>>
>>>>>Volume Use Duration =
>>>>>
>>>>>added to your pool definition where the duration is equal
>>>>
>>>>to or less than
>>>>
>>>>
>>>>
>>>>>your Volume Retention setting, in my case the relevant lines are
>>>>>
>>>>>Volume Retention = 6d
>>>>>Volume Use Duration = 6d
>>>>>
>>>>>With this setup, Bacula FINALLY works like I expect and how
>>>>
>>>>pretty much any
>>>>
>>>>
>>>>
>>>>>other backup software I've used on Windows or Unix works. It will
>>>>>overwrite the tape in the drive, assuming 6 days has
>>>>
>>>>passed, it marks the
>>>>
>>>>
>>>>
>>>>>tape as used and then recycles it, however it only does that at the
>>>>>beginning of the next job, so the recycling happens at the
>>>>
>>>>start of the
>>>>
>>>>
>>>>
>>>>>current job. Can someone please update the documentation here
>>>>>
>>>>>http://www.bacula.org/rel-manual/Automatic_Volume_Recycling.html
>>>>>
>>>>>with this information, that should hopefully forestall this
>>>>
>>>>question coming
>>>>
>>>>
>>>>
>>>>>up so much.
>>>>>
>>>>>>From the messages on tonights job
>>>>>
>>>>>16-Dec 21:30 fileserver-dir: Start Backup JobId 70,
>>>>>Job=Company-Fileserver_Full_Back 16-Dec 21:30
>>>>
>>>>fileserver-dir: Recycled
>>>>
>>>>
>>>>
>>>>>current volume "Friday3"
>>>>>16-Dec 21:30 fileserver-sd: Recycled volume "Friday3" on
>>>>
>>>>device "DDS-4"
>>>>
>>>>
>>>>
>>>>>(/dev/ns
>>>>>
>>>>>This is the first week since the beginning of the year
>>>>
>>>>where I've not had
>>>>
>>>>
>>>>
>>>>>to manually purge the tapes each day.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>>Think: if the tape goes full after 3 jobs, and I have 4
>>>>
>>>>each night, my
>>>>
>>>>
>>>>
>>>>>>>last job will not be done.
>>>>>>
>>>>>>Or, in other words, you need enough space available.
>>>>>
>>>>>Yes, that makes perfect sense.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>No, this is,in my opinion, the best way to handle backups: Avoid
>>>>>>overwriting valid data whenever possible.
>>>>>
>>>>>If I setup a rotation, I should be able as the
>>>>
>>>>administrator to decide what
>>>>
>>>>
>>>>
>>>>>is the appropriate retention time
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Well, your assumption is not coherent with Bacula, then.
>>>>
>>>>Bacula treats a
>>>>
>>>>
>>>>
>>>>>>retention like "keep the data for at least this time.
>>>>
>>>>Only after this
>>>>
>>>>
>>>>
>>>>>>time has passed consider recycling that volume." Thus,
>>>>
>>>>you should assume
>>>>
>>>>
>>>>
>>>>>>that your data will not be overwriten until 12 days have
>>>>
>>>>passed after the
>>>>
>>>>
>>>>
>>>>>>last job on a given volume is finished.
>>>>>
>>>>>In my experience, it still doesn't work, even 5 weeks later
>>>>
>>>>when I have a 6
>>>>
>>>>
>>>>
>>>>>day retention period, I still have to manually purge the tape.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Hmm. Isn't that how Bacula works? If a volume is in the
>>>>
>>>>right pool,
>>>>
>>>>
>>>>
>>>>>>and it is flagged as recyclable, and automatic pruning
>>>>
>>>>for the jobs
>>>>
>>>>
>>>>
>>>>>>and volumes is on, AND the retention period has passed, the volume
>>>>>>will be considered when Bacula looks for a recyclable volume.
>>>>>
>>>>>The key thing for the OP to remember is that bacula will
>>>>
>>>>not recycle the
>>>>
>>>>
>>>>
>>>>>current volume if any other appendable volumes exist, seems
>>>>
>>>>odd to me, but
>>>>
>>>>
>>>>
>>>>>that is how it is. However, from a practical pov, using the
>>>>
>>>>volume use
>>>>
>>>>
>>>>
>>>>>duration statement seems to get around that just fine.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>If you want to limit the time Bacula uses a volume, you
>>>>
>>>>can do so by
>>>>
>>>>
>>>>
>>>>>>setting the proper volume use time or number of jobs.
>>>>>
>>>>>Yes, that needs to be added to the docs.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>If I set up a pool to have backups available for a
>>>>
>>>>certain time, and
>>>>
>>>>
>>>>
>>>>>>I've not got enough volumes, Bacula tells me so. Either I
>>>>
>>>>re-think my
>>>>
>>>>
>>>>
>>>>>>backup strategy and modify the setup, or I buy more tapes.
>>>>>
>>>>>Even with that setup, it still doesn't recycle properly.
>>>>
>>>>I've seen this in
>>>>
>>>>
>>>>
>>>>>several instances where I've followed the chapter on
>>>>
>>>>automatic volume
>>>>
>>>>
>>>>
>>>>>recycling in the manual, until I was shown the info at the
>>>>
>>>>beginning of
>>>>
>>>>
>>>>
>>>>>this email, I was looking at dumping Bacula, which is a
>>>>
>>>>GREAT piece of
>>>>
>>>>
>>>>
>>>>>software, but for this one problem.
>>>>>
>>>>>Considering the frequency that this problem is asked about,
>>>>
>>>>I'd say there
>>>>
>>>>
>>>>
>>>>>is definately a conceptual issue at hand.
>>>>
>>>>--
>>>>Best regards,
>>>>
>>>>Kern
>>>>
>>>>(">
>>>>/\
>>>>V_V
>>>>
>>>>
>>>>-------------------------------------------------------
>>>>This SF.net email is sponsored by: Splunk Inc. Do you grep
>>>>through log files
>>>>for problems? Stop! Download the new AJAX search engine that makes
>>>>searching your log files as easy as surfing the web.
>>>>DOWNLOAD SPLUNK!
>>>>http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>>>>_______________________________________________
>>>>Bacula-users mailing list
>>>>Bacula-users@lists.sourceforge.net
>>>>https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>>>
>>>>**************************************************************
>>>>*********************
>>>>Mail FROM London Borough of Harrow:
>>>>Unencrypted electronic mail is not secure and may not be
>>>>authentic, in whole or in part. You are advised to check
>>>>directly with the sender before acting upon any e-mail received.
>>>>
>>>>The information contained in this message and any attachments
>>>>is confidential and is intended for receipt by the above
>>>>named addressee(s) only. If you have otherwise encountered
>>>>this message please notify its originator via +44(0)20 8863
>>>>5611 at LONDON BOROUGH OF HARROW. The unauthorised use,
>>>>disclosure, copying or alteration of this message is strictly
>>>>forbidden. The views expressed within this message are those
>>>>of the individual sender and not necessarily those of Harrow Council.
>>>>
>>>>Mail TO London Borough of Harrow:
>>>>London Borough of Harrow monitors all electronic mail it
>>>>receives for Policy compliance and to protect its systems
>>>>including anti-spam and anti-virus measures.
>>>>
>>>>Electronic mail does not guarantee delivery, nor notification
>>>>of non-delivery. It is suggested you contact your intended
>>>>recipient(s) by other means should confirmation of receipt be
>>>>important.
>>>>**************************************************************
>>>>*********************
>>>>
>>>>
>>>>
>>>>-------------------------------------------------------
>>>>This SF.net email is sponsored by: Splunk Inc. Do you grep
>>>>through log files
>>>>for problems? Stop! Download the new AJAX search engine that makes
>>>>searching your log files as easy as surfing the web.
>>>>DOWNLOAD SPLUNK!
>>>>http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
>>>>_______________________________________________
>>>>Bacula-users mailing list
>>>>Bacula-users@lists.sourceforge.net
>>>>https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>>
>>>
>>>
>>>-------------------------------------------------------
>>>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
>>>for problems? Stop! Download the new AJAX search engine that makes
>>>searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
>>>http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>>>_______________________________________________
>>>Bacula-users mailing list
>>>Bacula-users@lists.sourceforge.net
>>>https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>
>>
>

--
IT-Service Lehmann [EMAIL PROTECTED]
Arno Lehmann http://www.its-lehmann.de


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users