Hi Kern,
While writing my mail I did begin to wonder if actually the v7 upgrade had the
larger impact, but on balance I think both were worth doing so I’m not going to
go revert the auto pruning changes for the time being. I’m currently working
with the order of about 5,000 volumes so a little way off the numbers you
quote, but I suspect we’ll hit 10,000 within 3 years. For reference we were
running 5.2.13 for the last 12 months.
I am using mysql and I’m aware that’s not the preferred database engine for
Bacula, however there are a couple of other constraints that affected my
decision not to switch to postgres yet. One is that we’re running the Bacula
daemons and catalog on Solaris 11 and mysql is by far the easier system to get
running on that platform for fairly obvious reasons. The other is that we have
a lot more mysql experience than postgres in-house so there would be a
significant overhead in installing and maintaining it just for this purpose.
Our bacula setup needs a bit of love every 6-12 months when we run into
performance limitations due to data growth. So far I’ve managed to avoid
changing the catalog by implementing other optimisations (block-level backups,
additional storage capacity, configuration tweaks). At some point I may have to
bite the bullet and switch, but I’m hoping to hold off for at least the next
few rounds of improvement work! ☺
Regards,
Ben Roberts
From: Kern Sibbald [mailto:k...@sibbald.com]
Sent: 06 December 2014 10:52
To: Roberts, Ben; bacula-users
Subject: Re: [Bacula-users] Mini-Status report
Hello Ben,
Thanks for your feedback on pruning. While I was reading your email, I kept
wondering what version of Bacula you were running, because slowing down of the
console with version 5.2.x and before was a typical symptom, while in 7.0.x
this should largely be fixed even without delaying the pruning. At the end of
your email, you pointed out that you also switched to 7.0.x. Previously
Bacula used a single thread process at a time that was accessing the database,
with version 7.0a.x (if I remember right), we separated the console into a
second DB connection so it runs in its own thread and generally does not need
to wait for pruning or attribute spooling that can go on at the same time.
If you are currently using MySQL as the backend database and you still find
that either pruning or inserting file attributes takes a long time with Bacula
version 7.0.x, I would recommend you to move to PostgreSQL. Out of the box,
PostgreSQL is a bit harder to get to work with Bacula for two reasons: 1.
Getting the login permissions setup correctly for Bacula to access and modify
the database is more complex than MySQL which works out of the box. 2. The
default installation of PostgreSQL is very inefficient for medium to large
Bacula installations. It requires a little bit of tuning of about 10 of the
parameters, then PostgreSQL will perform significantly better than MySQL. I
have heard some good comments about MariaDB, but we have not yet tried it and
thus don't officially support it.
Final note: the current versions of Bacula are designed to handle about 1,000
volumes, if you start getting 20,000 or 100,000 volumes, you will very likely
run into severe performance problems. This is currently on my radar screen,
and I now think I know how to fix it (not trivial) but haven't yet found the
time.
Best regards,
Kern
On 12/06/2014 11:33 AM, Roberts, Ben wrote:
Hi Kern,
Thanks for updating the docs regarding auto prune. I disabled this last week
and it has immediately fixed a long-running performance problem we were having
when Bacula was looking for appendable volumes for a running job. I bring this
up not only to report success but also because our issue was slightly different
than initially described, so worth having on the record in this thread in case
it affects other people in future.
We could experience this issue even when only 2-3 jobs were running at once. If
one job ran out of volumes, the console would become slow to respond to any
queries that touched the catalog (~30-60 seconds). If two jobs ran out at the
same time the console might block for several minutes at a time. When three or
more jobs all ran out the console would become practically unusable, blocking
for so long that you'd have to go away and do something else in the meantime. I
think Bacula rescans for appendable volumes every ~5minutes, so this process
would compound if there was more than 5mins work to do each cycle. It made
resolving an out-of-volumes issue an absolute pain and as a result it was
frequently ignored causing our already long-running jobs to hit the 6 day limit
and fail. This had gotten increasingly worse over the last few months which is
likely because the number of volumes we consume grows over time (~+50/month).
While the end-of-job pruning scans would also have caused this
The reason for this is that while bacula is scanning for appendable volumes it
appears to issue queries against the catalog regarding individual volumes in
sequence (I guess this is to find the list of jobs on each volume one at a
time, then check the job retention periods to see if any can be deleted?) We
have several thousand volumes, so executing these queries takes a while and I
get the feeling bacula will only do these in sequence, due to either a lock or
other single-threaded behaviour. So when there were several of these operations
queued up because multiple jobs were seeking volumes at the same time, it's
understandable why the window to slip a bconsole query in and have a timely
response was very narrow and why most bconsole commands would take a very long
time to return.
Full disclosure, I did also upgrade to 7 at the same time as making the pruning
changes and the separate catalog connection may also have helped with this,
however both together certainly give the desirable performance we used to
experience.
Ben Roberts
> -----Original Message-----
> From: Kern Sibbald [mailto:k...@sibbald.com]
> Sent: 25 November 2014 13:42
> To: bacula-users; bacula-devel
> Subject: [Bacula-users] Mini-Status report
>
> Hello,
>
> This is a mini-status report just to let you know that someone on one of
> these lists said that the new white papers advise using:
>
> Maximum Concurrent Jobs = 1
>
> In the bacula-sd.conf Device resource. I have reviewed the three white
> papers listed on the bacula.org site and I find no such mention. Almost
> all the examples use MaximumConcurrentJobs=5 and several set it to 10.
>
> If someone can point me to the place (if it exists) where 1 in mentioned,
> I will be happy to fix it.
>
> In addition, another person pointed out that manual_prune.pl does not seem
> to be available. That was an oversight on my part, and that file is now
> included as a download item under White Papers. The essential elements of
> what manual_prune.pl does has been integrated directly into the Bacula
> source code, and in the next few days I will update the documentation to
> refer to the new way of doing "manual" pruning (i.e.
> pruning once a day rather than at the end of every Job).
>
> For those of you who are not aware of the pruning issues, you need to know
> that when one has many jobs (say 1000 per day) and many Volumes, the
> normal Job pruning that Bacula does at the end of each Job can become a
> performance problem (not likely with less than 50 jobs per day). To
> resolve it, it is best to turn off automatic pruning but then to schedule
> an Admin job once a day that will do the equivalent. The program (now
> integrated into the code) manual_prune.pl was written by Bacula Systems
> for the Enterprise customers, but is published for your use with the
> community version.
>
> Finally, I have made a few very minor changes to the Best Practices for
> Disk Based Backup document.
>
> Best regards,
> Kern
>
> --------------------------------------------------------------------------
> ----
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from
> Actuate! Instantly Supercharge Your Business Reports and Dashboards with
> Interactivity, Sharing, Native Excel Exports, App Integration & more Get
> technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clkt
> rk
> _______________________________________________
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net<mailto:Bacula-users@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/bacula-users
________________________________
This email and any files transmitted with it contain confidential and
proprietary information and is solely for the use of the intended recipient. If
you are not the intended recipient please return the email to the sender and
delete it from your computer and you must not use, disclose, distribute, copy,
print or rely on this email or its contents. This communication is for
informational purposes only. It is not intended as an offer or solicitation for
the purchase or sale of any financial instrument or as an official confirmation
of any transaction. Any comments or statements made herein do not necessarily
reflect those of GSA Capital. GSA Capital Partners LLP is authorised and
regulated by the Financial Conduct Authority and is registered in England and
Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261.
GSA Capital Services Limited is registered in England and Wales at the same
address, number 5320529.
This email and any files transmitted with it contain confidential and
proprietary information and is solely for the use of the intended recipient.
If you are not the intended recipient please return the email to the sender and
delete it from your computer and you must not use, disclose, distribute, copy,
print or rely on this email or its contents.
This communication is for informational purposes only.
It is not intended as an offer or solicitation for the purchase or sale of any
financial instrument or as an official confirmation of any transaction.
Any comments or statements made herein do not necessarily reflect those of GSA
Capital.
GSA Capital Partners LLP is authorised and regulated by the Financial Conduct
Authority and is registered in England and Wales at Stratton House, 5 Stratton
Street, London W1J 8LA, number OC309261.
GSA Capital Services Limited is registered in England and Wales at the same
address, number 5320529.
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
