On Fri, 2007-10-05 at 18:57 -0400, Steve Thompson wrote: > On Wed, 26 Sep 2007, Ross Boylan wrote: > > > I've been having really slow backups (13 hours) when I backup a large > > mail spool. I've attached a run report. There are about 1.4M files > > with a compressed size of 4G. I get much better throughput (e.g., > > 2,000KB/s vs 86KB/s for this job!) with other jobs. > > > > First, does it sound as if something is wrong? I suspect the number of > > files is the key thing, and the mail spool has lots of little files > > (it's used by Cyrus). Is this just life when you have lots of little > > files? > > How long does it take just to create a tar file containing those 1.4M > files, with no bacula in the picture? Perhaps the file system type is of > concern here. > > Steve I did some early tests, but wasn't sure what had already been read in. I did another directory with 357k files (roughly 3.6G according to du). time tar cf /dev/null mydir First time: 15 minutes, 19 seconds. Second time: 32 seconds.
Here are some representative vmstat 10 outputs during the initial pass: 3 1 418760 20376 212364 399624 0 0 117 30 499 3385 19 4 36 41 1 1 418760 23924 214612 392436 0 0 123 33 504 3463 19 5 35 40 5 2 418760 25196 215912 389740 0 0 120 33 501 3389 20 5 37 38 2 1 418760 24832 215400 389504 0 75 122 129 510 3497 20 6 32 42 1 1 418760 25384 215304 388572 0 0 109 103 643 6277 26 8 29 37 3 1 418760 29112 217036 388692 0 0 120 33 546 3476 19 6 34 41 This is particularly ironic because I only selected the filesystem (ext3) after extensive performance testings. The tests, however, were oriented toward reads, and probably didn't cover this huge number of files (I think they did cover the "file system almost full" case). I don't know if this is enough to account for the whole problem, but clearly it's a problem, and it doesn't involve bacula. I suppose the candidates are the file system, the way it is tuned, the lvm layers, or the evms layers. Maybe some hdparm type parameters could be involved too. Ross ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
