Hello,
On Monday 12 February 2007 11:43, Daniel Holtkamp wrote:
> Hi !
>
> My bacula 2.0.1 installation is running quite nicely except for some
> servers. I`ll use only one of these as an example as the others have the
> same problem.
>
> This one server has to backup more than 5 million files that are very
> small (usually less than 2KB). The problem is that the performance
> impact backing up these files is enormous.
>
> Here is a little sniplet from the last (unfinished) backup.
>
> Elapsed time: 23 hours 51 mins 42 secs
> Priority: 10
> FD Files Written: 3,562,070
> SD Files Written: 3,561,858
> FD Bytes Written: 2,507,509,039 (2.507 GB)
> SD Bytes Written: 3,088,552,545 (3.088 GB)
> Rate: 29.2 KB/s
>
> At that time the backup ran for almost a complete day and it still has
> to backup 2+ million files that make up for about 3 GB of data. As you
> can see the rate is VERY slow. I have of course enabled attribute
> spooling to take the database out of the equation. Also the backup goes
> to diskbased-volumes. It only gets this slow when it gets to the loads
> of small files - prior to that the backup rate is perfectly acceptable
> with 2MB/s.
>
> The fileset for this server is this:
>
> FileSet {
> Name = "X400mta"
> Include {
> Options {
> exclude = yes
> wilddir = "/var/tmp"
> regexdir = "/var/[cache/man|catman]/[cat?|X11R6/cat?|local/cat?]"
> compression=GZIP
> signature=SHA1
> }
> File = /
> File = /opt
> File = /usr
> File = /var
> File = /export/home
> }
> Include {
> Options {
> regexdir = "/var/[cache/man|catman]/[cat?|X11R6/cat?|local/cat?]"
> keepatime=yes
> mtimeonly=yes
> compression=GZIP
> signature=SHA1
> }
> File = /var/tmp
> }
> Exclude {
> File = .autofsck
> File = /proc
> File = /tmp
> File = .journal
> File = /opt/rsi/archive
> File = /opt/rsi/spool
> File = /opt/x400/mtadata/logfiles
> }
> }
>
> Any ideas on how to improve performance here ? Can the excludes be a
> problem ? Or the Regex ?
>
> Also what influences the performance on migrating data ?
>
> I`ve had migration processes running nicely at 15MB/s (max for
> tapedrive) and some go at a measily 1 MB/s - from the same disk-array to
> the same tapedrive of course.
Performance is a complicated issue. Judging from everything that you have
written above (especially the variations of the migration speeds), I suspect
that there is nothing terribly slow with your FD. Rather the problem seems
to be in your Catalog.
Catalog performance problems can be due to:
1. the SQL database parameters are not properly configure for handling large
databases. This is an issue with MySQL or PostgreSQL (with backup volumes
like yours you should not be using SQLite). The manual has some points on
how to make sure the database is setup to handle large volumes.
2. You may not have all the proper indexes on your tables. Again, the manual
suggests some solutions.
3. Inserting attributes in the current Bacula code is rather inefficient,
especially if you have large numbers of new files being created each backup
(some mail programs do this). The current code for version 2.1.4 (in the
SVN) has some new code that speeds up insertions by quite a lot (most
improvement is for PostgreSQL, but MySQL also gets a good boost). This code
is not currently turned on, though it has been in use at Eric's site for
quite a long time now. I will be enabling this code by default in the next
few weeks once I have tested it a bit.
If you are interested in testing this new code, I would recommend that you get
in touch with Eric. Some of the table parameters should be modified, and
this is documented only in the patches/testing/batch-insert.readme file, and
you must explicitly turn on a #define in src/version.h) to turn it on.
Please copy the bacula-devel list if you decide to do this so that we can all
benefit from your tests.
Best regards,
Kern
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
