On Friday 06 October 2006 08:27, Dirk H. Schulz wrote: > Hi folks, > > I have started testing Bacula and do have a peculiar phenomen. > > I am using Bacula 1.36.x because it comes with Debian Sarge stable > (compiling the current version comes after the first testing phase). The > data I use for backup testing is an amount of 550 GB which consists of > nearly 2 million files ranging from many thousands of the smallest > possible to several 100 files with 2-15 GB each. > > The first testing I have done with SQLite as db backend. Everything was > fine and faster than expected (an average of 200 GB in 5 hours over the > network), and there was no slowdown with the trillions of smallest files. > > Now I have switched to MySQL 4.1 as the db backend. The first 400 GB > went strait within 10 hours, but the next 34 GB (consisting of trillions > of the smallest ...) took another 20 hours. The backup is still running. > > There is no error message, nothing unnormal, the backup run has just > become very, very slow. There is no network congestion (it is a GBit > network with just a few machines and easy to overlook), the machines do > nothing else besides backing up, so an external influence can be excluded. > > AFAIF, the questions are now: > - does MySQL 4.1 as db backend slow down the backup process if it comes > to lots of smallest files - compared to sqlite? > - if yes, is this phenomenon restricted to Bacula version 1.36.x (see > below, please)? > - is there any other influence I overlook at the moment? > > Inbetween I had a test running with Debian Testing, MySQL 5 and Bacula > 1.38.x. As far as I remember I did not have a slowdown at the "trillion > small files" part - but since I had a lot of other problems I did not > look on performance intensely. I just remember the test running normally > and smoothly into a crash I accidentally fired (the bacula db had 1.7 GB > when it suddenly died, then - now it has just over 400 MB). > So the next question would be: > - if the slowdown does not appear with MySQL 5 and Bacula 1.38.x, is it > according to MySQL 5 as a better db backend or to Bacula 1.38 having > certain improvements therein? > > I know I can answer all these questions on my own by simply testing on > and on - but since one test (setting up the backup server, optimizing > configs, running at least one backup, evaluating the results) takes at > least 3 days, I would appreciate any comment and hint on this.
For the size of operation you are using, I personally wouldn't even consider using SQLite. I also strongly recommend Bacula 1.38.11, and for MySQL, make sure you read the database maintenance chapter -- I think you need something like mybig.cnf (I forget the exact name) so that MySQL is tuned to use a lot of memory and thus handle a large number of files. The users of this list can surely give you more details better than I can. > > Thanks for your patience, folks. > > Dirk > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
