Hello,
Sorry for the long overdue reply. Thanks very much for the tips. Jobs per
day is very less, but having hundreds of clients with different ways of
handling the tapes is giving me a lot of issues. Most of the time the
problem is due to the clients not changing the tape on time (daily basis),
which leaves the jobs pending, which leads to the tape not being able to
get ejected. And sometimes this will cause the tape to get filled up, and
again, jobs will fail to run.
It's a challenge to create a configuration that is fool proof for the
users. 'Users' that I'm refering here are not technical, so I have to
minimize the steps involved for them to handle the backup as much as
possible.
And yes, I should consider taking up the course :D.
On Wed, Dec 7, 2016 at 5:20 PM, Kern Sibbald <k...@sibbald.com> wrote:
> Hello,
>
> Here are a few tips for you.
>
> - You absolutely need to understand what Pruning is. The best way to
> understand it is to read the manual.
>
> - If you are running MySQL as that catalog and you are having catalog
> performance issues, you should consider switching to Postgresql.
>
> - Usually performance issues with the catalog are related to several points
> 1. You may have added additional indexes. Doing so sometimes speeds up
> a particular operation,
> but much more often totally destroys the performance elsewhere. If
> you have added any, start by
> removing them.
> 2. Every database and particularly Postgresql needs some tuning for
> your system and your workload.
> Generally the database vendor has programs that will help you find
> problems.
>
> - If you are running more than 1000 jobs per day, then you clearly need
> Postgresql and you need it to be properly tuned. You may also need to turn
> off automatic pruning and specifically schedule a prune.
>
> - Pruning of a single job should never take an excessive amount of time,
> but if as noted above you have 1000 jobs, it is not always ideal to do it
> while backups are running.
>
> - You should *never* be manually (with SQL commands) be changing the
> Bacula catalog.
>
> - You should almost never need to use a built in command to modify the
> catalog Media records. This does not include running prune or purge
> commands.
>
> - If you are running more than 500 jobs a day, you need to take the Bacula
> Admin I course.
>
> - If you are running more than 500 jobs and not a Bacula expert, you most
> likely need a professional service contract.
>
> Best regards,
> Kern
>
>
> On 12/06/2016 09:55 PM, Gi Dot wrote:
>
> Weird thing is, after the long pruning, I decided to change the volume
> status to recycle. Done that, the backup worked fine. After that I changed
> the status of all volumes to recycle. But last night, the scheduled backup
> didn't run because of pruning vol-F1. I thought when the volume is recycled
> it will skip all the pruning part.
>
>
>
>
> On Wed, Dec 7, 2016 at 4:41 AM, Gi Dot <gadi...@gmail.com> wrote:
>
>> All of the volumes were full at the time, and have past the retention
>> period. Bacula prunes the oldest volume, and stuck at it.
>>
>> Sorry if this question is trivial, but I don't really understand purging.
>> If I were to purge a volume manually, I will definitely lose all of the
>> backups in the volume. And all the jobs associated with the volume will be
>> removed from the catalog. So how does it make me lose other backups that I
>> wanted to keep? Isn't it pretty much the same if I were to recycle the
>> volume?
>>
>> On Tue, Dec 6, 2016 at 3:18 AM, Martin Simmons < <mar...@lispworks.com>
>> mar...@lispworks.com> wrote:
>>
>>> >>>>> On Mon, 5 Dec 2016 17:40:08 +0800, Gi Dot said:
>>> >
>>> > > What exactly do you mean by "too long"? Does bacula encounter a
>>> > > timeout during the pruning from a database error?
>>> > Like it runs overnight and still not done with it. Sample log:
>>> > https://dpaste.de/wetb/raw
>>>
>>> This doesn't look like a database indexing problem to me.
>>>
>>> It looks like it needs an extra volume and fails to find one older than
>>> the
>>> retention periods.
>>>
>>> Did you expect vol-D1 to fill up? If so, which volume should it use
>>> next?
>>> You need to look at the retention period of that volume to see why it
>>> was not
>>> recycled.
>>>
>>> BTW, forcing a volume from Used to Recycle using the update command is a
>>> bad
>>> idea because the database will bloat with unwanted file records. You
>>> could
>>> use the purge command but beware that you might lose backups that you
>>> wanted
>>> to keep.
>>>
>>> __Martin
>>>
>>>
>>> >
>>> > Thanks.
>>> >
>>> >
>>> >
>>> > On Mon, Dec 5, 2016 at 5:02 PM, Uwe Schuerkamp <
>>> uwe.schuerk...@nionex.net>
>>> > wrote:
>>> >
>>> > > On Mon, Dec 05, 2016 at 03:18:45PM +0800, Gi Dot wrote:
>>> > > > Hello,
>>> > > >
>>> > > > I have this problem with one of my client experiencing pruning of a
>>> > > volume
>>> > > > that is taking too long (and in the end I ended up recycling it
>>> manuall=
>>> > y
>>> > > by
>>> > > > updating the volume status). I have googled up on this and from
>>> what I
>>> > > > understand it is mostly due to the database indexing (to be honest
>>> I
>>> > > don't
>>> > > > entirely understand this part).
>>> > > >
>>> > > > My question is, is there any downside or side effect if I were to
>>> > > include a
>>> > > > script that looks up for Used volume and update it to Recycle
>>> before th=
>>> > e
>>> > > > backup runs for the day. I am using this script on another client
>>> and
>>> > > > things are going fine over there, but I'm just worried if there is
>>> any
>>> > > > impact in a long run.
>>> > > >
>>> > > > If anyone would be so kind to explain to me what exactly it means
>>> by
>>> > > > pruning; as in what bacula does when it runs pruning on a volume,
>>> it is
>>> > > > much appreciated as well. I have read somewhere that bacula
>>> removes the
>>> > > > jobs associated with the volume from the catalog.
>>> > > >
>>> > >
>>> > > Hello Gidot,
>>> > >
>>> > > some more info could be useful to help you in analyzing your setup
>>> > > further.
>>> > >
>>> > > - Hardware specs of the director (assuming all components run on a
>>> > > single machine)
>>> > >
>>> > > - Which database are you using for the catalog?
>>> > >
>>> > > - Amount of RAM available to the DB / backend storage (disks, ssds?)
>>> > >
>>> > > - Catalog size (file table rows)
>>> > >
>>> > > - Bacula version
>>> > >
>>> > > What exactly do you mean by "too long"? Does bacula encounter a
>>> > > timeout during the pruning from a database error?
>>> > >
>>> > > All the best,
>>> > >
>>> > > Uwe
>>> > >
>>> > >
>>> > > --
>>> > > Uwe Sch=C3=BCrkamp | email: < <uwe.schuerk...@nionex.net>
>>> uwe.schuerk...@nionex.net>
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> >
>>> > --001a113fc4b4cef4d20542e612a1
>>> > Content-Type: text/html; charset=UTF-8
>>> > Content-Transfer-Encoding: quoted-printable
>>> >
>>> > <div dir=3D"ltr"><div>- Hardware specs of the director (assuming all
>>> compon=
>>> > ents run on a<br>
>>> > single machine)<br>
>>> > </div>>> Backup runs on a database server. There's also one
>>> remot=
>>> > e client being backed up once a week.<br><div><br>
>>> > - Which database are you using for the catalog?<br></div><div>>>
>>> Post=
>>> > greSQL<br></div><div>
>>> > <br>
>>> > - Amount of RAM available to the DB / backend storage (disks,
>>> ssds?)<br></d=
>>> > iv><div>>> 32G RAM on host, 8GB for effective cache size, 8MB
>>> for wor=
>>> > k mem, and 8GB for shared buffers / SAS disks<br></div><div>
>>> > <br>
>>> > - Catalog size (file table rows)<br></div><div>>> The size of
>>> the cat=
>>> > alog is close to 5MB. Nothing huge.<br></div><div>
>>> > <br>
>>> > - Bacula version<br></div><div>>> 5.2.12-3.1<br></div><div>
>>> > <br>
>>> > What exactly do you mean by "too long"? Does bacula
>>> encounter a<b=
>>> > r>
>>> > timeout during the pruning from a database
>>> error?<br></div><div>>> Li=
>>> > ke it runs overnight and still not done with it. Sample log:<br><a
>>> href=3D"=
>>> > https://dpaste.de/wetb/raw";>https://dpaste.de/wetb/raw</a><b
>>> r><br></div><di=
>>> > v>Thanks.<br><br><br></div><div><div class=3D"gmail_extra"><br><div
>>> class=
>>> > =3D"gmail_quote">On Mon, Dec 5, 2016 at 5:02 PM, Uwe Schuerkamp <span
>>> dir=
>>> > =3D"ltr"><<a href=3D"mailto:uwe.schuerk...@nionex.net";
>>> target=3D"_blank"=
>>> > >uwe.schuerk...@nionex.net</a>></span> wrote:<br><blockquote
>>> class=3D"gm=
>>> > ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid
>>> rgb(204,=
>>> > 204,204);padding-left:1ex"><span class=3D"gmail-">On Mon, Dec 05,
>>> 2016 at 0=
>>> > 3:18:45PM +0800, Gi Dot wrote:<br>
>>> > > Hello,<br>
>>> > ><br>
>>> > > I have this problem with one of my client experiencing pruning of
>>> a vo=
>>> > lume<br>
>>> > > that is taking too long (and in the end I ended up recycling it
>>> manual=
>>> > ly by<br>
>>> > > updating the volume status). I have googled up on this and from
>>> what I=
>>> > <br>
>>> > > understand it is mostly due to the database indexing (to be
>>> honest I d=
>>> > on't<br>
>>> > > entirely understand this part).<br>
>>> > ><br>
>>> > > My question is, is there any downside or side effect if I were to
>>> incl=
>>> > ude a<br>
>>> > > script that looks up for Used volume and update it to Recycle
>>> before t=
>>> > he<br>
>>> > > backup runs for the day. I am using this script on another client
>>> and<=
>>> > br>
>>> > > things are going fine over there, but I'm just worried if
>>> there is=
>>> > any<br>
>>> > > impact in a long run.<br>
>>> > ><br>
>>> > > If anyone would be so kind to explain to me what exactly it means
>>> by<b=
>>> > r>
>>> > > pruning; as in what bacula does when it runs pruning on a volume,
>>> it i=
>>> > s<br>
>>> > > much appreciated as well. I have read somewhere that bacula
>>> removes th=
>>> > e<br>
>>> > > jobs associated=C2=A0 with the volume from the catalog.<br>
>>> > ><br>
>>> > <br>
>>> > </span>Hello Gidot,<br>
>>> > <br>
>>> > some more info could be useful to help you in analyzing your setup<br>
>>> > further.<br>
>>> > <br>
>>> > - Hardware specs of the director (assuming all components run on a<br>
>>> > single machine)<br>
>>> > <br>
>>> > - Which database are you using for the catalog?<br>
>>> > <br>
>>> > - Amount of RAM available to the DB / backend storage (disks,
>>> ssds?)<br>
>>> > <br>
>>> > - Catalog size (file table rows)<br>
>>> > <br>
>>> > - Bacula version<br>
>>> > <br>
>>> > What exactly do you mean by "too long"? Does bacula
>>> encounter a<b=
>>> > r>
>>> > timeout during the pruning from a database error?<br>
>>> > <br>
>>> > All the best,<br>
>>> > <br>
>>> > Uwe<br>
>>> > <br>
>>> > <br>
>>> > --<br>
>>> > Uwe Sch=C3=BCrkamp | email: <<a href=3D"mailto:uwe.schuerkamp@
>>> nionex.net=
>>> > ">uwe.schuerk...@nionex.net</a>><br>
>>> > <br>
>>> > <br>
>>> > <br>
>>> > <br>
>>> > <br>
>>> > <br>
>>> > <br>
>>> > </blockquote></div><br></div></div></div>
>>> >
>>> > --001a113fc4b4cef4d20542e612a1--
>>> >
>>> >
>>> > --===============8049073509691406397==
>>> > Content-Type: text/plain; charset="us-ascii"
>>> > MIME-Version: 1.0
>>> > Content-Transfer-Encoding: 7bit
>>> > Content-Disposition: inline
>>> >
>>> > ------------------------------------------------------------
>>> ------------------
>>> >
>>> > --===============8049073509691406397==
>>> > Content-Type: text/plain; charset="us-ascii"
>>> > MIME-Version: 1.0
>>> > Content-Transfer-Encoding: 7bit
>>> > Content-Disposition: inline
>>> >
>>> > _______________________________________________
>>> > Bacula-users mailing list
>>> > Bacula-users@lists.sourceforge.net
>>> > https://lists.sourceforge.net/lists/listinfo/bacula-users
>>> >
>>> > --===============8049073509691406397==--
>>> >
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> _______________________________________________
>>> Bacula-users mailing list
>>> Bacula-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/xeonphi
>
>
>
> _______________________________________________
> Bacula-users mailing
> listBacula-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
>
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
