> -----Original Message----- > From: Stan Meier [mailto:stan.me...@billigmail.org] > Sent: Saturday, February 27, 2010 10:08 AM > To: bacula-users@lists.sourceforge.net > Subject: Re: [Bacula-users] Backing up > 100 servers > > While you are right and creating a configuration based on scripts is > quite easy (and has added benefits, for example that you dan define > one file pool per server group), we still have to deal with 120 backup > jobs. > > But, since you didn't jump on that part of my question, I presume > there is no solution to that?
You will always have (at least) 120 jobs, but with a script, you can simply autogenerate them in something like a for loop. > I see several things here which we will have to look at. Please > correct me if I'm wrong or if I forgot anything: > > 1. Concurrency: We will need to investigate all the different places > in Bacula where job concurrency, concurrent pool/storage usage and > connection limits are defined and adjust them to "fit together" as > well as optimize them to the I/O operations limit of our raid storage. In order for concurrency to work well, you'd need multiple different storage devices. You could define different directories on your disk as different storage devices. With backing up that many machines, you will be looking at quite a few optimizations; there are quite a few bottlenecks you would need to address. One of the pitfalls with concurrency is that if you set up concurrency on the same storage device, multiple backups will end up on the same storage volume. That may make it later harder to recover the disk space. Also, restores will take longer because bacula has to sort through the jumble of blocks. > 2. Scheduling: Ideally, the copy job would start as soon as all the > backup jobs have finished. But since the Schedule resource does not > allow references to Job names, we are pretty much screwed in that > department and will probably have to resort to a fixed schedule. Not necessarily. You have at least two options: - Don't use a copy job, but instead some shell scripting that happens in a Run After Job script. Not sure how well this works with tape drives, but I use it for an rsync-based setup. - Schedule the copy jobs for one minute later than the backup jobs, and with a lower priority. Bacula will then complete all backup jobs and immediately work on the copy jobs. Pitfall: if the backup jobs overrun, the next backup jobs may get scheduled before the copy job ever executes. - Schedule the copy jobs for one minute later than the corresponding backup job, and with the same priority. Then bacula will execute the backup job, the corresponding copy job, another backup job, another copy job, etc. > 3. Job selection: You already pointed that one out. Would it be as > easy as just selecting all uncopied jobs from a given pool? Since all > volumes in the pool are recycle before backup starts (or while it is > running), naturally, the only uncopied jobs would be those that were > written to disk recently. > > So, for now, our backup plan would be something like: > > 1. Start actual backup > 1.1 Start backup jobs in parallel > 1.2 (Possibly erase all used volumes in a given pool) > 1.3 Get data from clients, write to File pools > 1.4 wait until all backup operations have finished > 2. Copy all recent jobs to tape > 3. Backup catalog > > I don't see how we can synchronize more than 120 backup jobs yet, to > be honest. We could run backup at 9pm and the copy job at 10am to > allow for a large margin of error, but that just doesn't feel like a > proper solution. Also doesn't look like a large margin of error. Depending on how big your backup jobs are and how well tuned your system is, 120 full backups may well take much more than 24 hours - and that's simply due to network bandwidth limitations and the like, nothing bacula could address. ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
