---- > -----Original Message----- > From: Kern Sibbald [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 11, 2007 3:26 AM > To: Alan Davis; bacula-users; bacula-devel > Subject: Re: [Bacula-users] Optimizing bacula in large filesystems > > On Thursday 11 January 2007 07:36, Alan Davis wrote: > > > > Problem solved - even though we aren't licensed for > > snapshots on the NAS server it still has a hidden snapshot > > directory as well as another hidden directory that seems > > to be part of the filesystem. When I exclude these from > > the fileset the backup starts processing files within > > minutes. > > > > This probably doesn't rise to the level of a bug, but may > > deserve at least a mention in the FAQ? > > I'm happy to hear you find a solution. > > Yes it does deserve to be mentioned. Could you give a bit more details of > what you had to do to turn off the snapshot? > > Also, it sounds like it is taking 36 hours to make the snapshot, which > seems > terribly long. That might be something to discuss with the NAS guys > because > if they could reduce the time, doing a backup from the snapshot might be > preferable (depending on the time it takes to set it up). From our > experience with Windows, a snapshot is always slower, but if it isn't too > much slower, it is worth the effort to have a more consistent image. > > I'm copying the list on this so that it will be documented in the > archives. > > > > > Thanks again, > > > > Alan > > > > On Wed, 10 Jan 2007 17:53:24 -0500 > > "Kern Sibbald" <[EMAIL PROTECTED]> wrote: > > > On Wednesday 10 January 2007 22:48, Alan Davis wrote: > > >> > > >> The servers that I am backing up with bacula have on the > > >>order of 11TB > > >> of data 4.5 million files. > > >> > > >> The data resides on NAS appliances that are incapable of > > >>running the > > >> bacula FD natively so I'm forced to back the data up > > >>over GbE and NFS > > >> mounted on the server that is running DIR, SD, FD and > > >>the db. > > >> > > >> > > >> > > >> The server is a 2cpu 2+GHz Xeon Solaris 10 x86 server > > >>with 4GB of RAM. > > >> > > >> > > >> > > >> It's currently taking 36+ hours from the start of the > > >>backup job to > > > when > > >> the first files are written to tape. > > >> > > >> This makes it difficult to do incremental backups in a > > >>reasonable > > >> timeframe. > > > > > > Even with 4.5 million files, which is rather large, it > > >should not be > > > more than > > > 5 or 10 minutes before Bacula finds the first file in an > > >incremental > > > backup > > > and begins writing it to tape (assuming you are not > > >using spooling). I > > > suspect that you have an *extremely* slow NAS or some > > >other major > > > bottleneck > > > (network, NFS, ...). It seems to me that you not only > > >have something > > > that > > > has a bottleneck, but it has a monsterous bottleneck > > >that I even have > > > difficulties imagining unless you are talking about > > >another software > > > product > > > or trying to do a restore, which for 4.5 million files > > >could be very > > > time > > > consuming. > > > > > > Bacula simply reads the filesystem directories and > > >writes out files that > > > need > > > backing up. There is no preliminary time intensive > > >setup before Bacula > > > begins working as in some other programs. > > > > > >> > > >> > > >> > > >> I've seen mention here on the list of others backing up > > >>multi-terabyte > > >> servers - I'm looking for suggestions on how to optimize > > >>and speed up > > >> the backup process. > > >> > > >> > > >> > > >> Thanks! > > >> > > >> > > >> > > >> Alan > > >> > > >> > > >> > > >> > > >> > > >> ---- > > >> > > >> Alan Davis > > >> > > >> Senior Architect > > >> > > >> Ruckus Network, Inc. > > >> > > >> 703.464.6578 (o) > > >> > > >> 410.365.7175 (m) > > >> > > >> [EMAIL PROTECTED] > > >> > > >> alancdavis AIM > > >> > > >> > > >> > > >> > > >> > > >> > > > > > The system isn't actually taking a snapshot - that functionality is disabled. What seems to be happening is that the contents of the .ifsvar and .snapshot directories in the Isilon filesystem contain things that /aren't/ "files" and when the FD tries to recurse into them it's not succeeding but also not failing with any sort of error message. What I'm seeing is that the FD uses about 20% of the cpu, the debug message output stops and there's no discernable progress on the backup. My solution was to use a FileSet specification similar to this : FileSet { Name = ... Include { ... } File = </local/etc/... Exclude { File = "*/.snapshot" File = "*/.ifsvar" ... } } There are probably other ways of specifying the exclude directives that will work. [Alan Davis] Alan Davis Senior Architect Ruckus Network, Inc. 703.464.6578 (o) 410.365.7175 (m) [EMAIL PROTECTED] alancdavis AIM ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
