You could take the approach of "append to incremental file every night" but I don't think it would help you, and actually would hurt.
Your idea of using small, 100 MB volumes to encourage rsync to work more efficiently really is redoing something that bacula already does for you. If you simply create a new volume for each job, this volume will contain *exactly* the data that needs to be transferred. As a bonus, it also allows bacula to more efficiently prune/purge volumes. So there really is no need to use 100MB chunks here. I also found that rsync diffs takes an inordinate amount of time figuring out what has changed. With remote transfers of small changes, that is a good tradeoff. My situation was a bit different because I used rsync for local copying. cp -u was a lot faster than rsync. Of course that is for a local copy. I would take a look of using rsync with the -W and -u options. -u only copies updated files - so old backup files won't be re-copied. -W turns off the diff algorithm. If you have one volume per job, there is no need to do a diff because you know that all files that -u finds are completely new. > -----Original Message----- > From: wvoice [mailto:myo...@wildernessvoice.com] > Sent: Friday, November 13, 2009 12:28 PM > To: bacula-users@lists.sourceforge.net > Subject: Re: [Bacula-users] Using bacula for local and cloud backup > > > Hi Gavin, > > I don't know either. That's the big concern for me. I suppose I can > test > that out. > > My other thought was to take my full backups and restore them onto a > USB > drive so I can directly upload the contents to my datacenter. Then I > could > create a job to restore an incremental onto a directory then rsync it > to my > datacenter, then delete the local data. > > Just trying to figure out the best method. > > -Mike > > Gavin McCullagh-2 wrote: > > > > Hi, > > > > On Tue, 10 Nov 2009, Jesper Krogh wrote: > > > >> wvoice wrote: > >> > However, I'd like to be able to backup the backup data offsite. > Right > >> now, > >> > my storage pool is on a file volume located in /data/backup/. I'm > >> trying to > >> > figure out the best way to do this. My usual mechanism is to use > rsync > >> for > >> > my offsite replication. But these files are quite large now. Just > >> backing up > >> > the file will be very costly, unless I can mount it and get access > to > >> the > >> > contents. Then I can copy the diffs. > > > > I was under the impression that rsync was very smart about updating > the > > contents of a file. If (for the sake of argument) you had an > incremental > > volume which you wrote to every night, you should be appending to the > > existing file (like a tape, right?). I would expect (perhaps > naively?) > > that rsync would discover that some large chunk of the file was > unchanged > > and mostly just transfer the newly appended data. > > > > I've never tried though. It would be interesting to test this. > > > >> Isn't the quick'n'dirty solution not just to make the individual > >> volume-files smaller? Say 100MB or similar. The for > >> differential/incremental runs you would eventually end up only > >> transferring the diffs. > > > > That's an option alright. > > > > As we're on the subject, has anyone considered running a bacula-sd in > the > > cloud and running migrate or copy jobs to get the data across? That > > should > > be a pretty targetted approach which transfers exactly the correct > data > > and > > allows simple (if slow) restores to be done from the cloud. I > appreciate > > not every subscription service would allow you install a bacula-sd > but a > > VM > > should. Might a subscription service offer a bacula-sd out of the > box? > > You could of course do this over a VPN if need be. > > > > Would (pragmatic) security people shudder at the thought of this? > > > > Gavin > > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
