Re: Use rsync's checksums to deduplicate across backups

2011-11-20 Thread Dan Stromberg
On Sun, Nov 6, 2011 at 2:29 PM, Dan Stromberg wrote: > > http://stromberg.dnsalias.org/~strombrg/backshift/documentation/comparison/index.html > I've updated the above URL to include a comparison against Lessfs and git wrappers. The table has also become easier to navigate recently, due to usin

Re: Use rsync's checksums to deduplicate across backups

2011-11-19 Thread Dan Stromberg
On Sat, Nov 19, 2011 at 5:43 AM, Andrea Gelmini wrote: > 2011/11/3 Alex Waite : > >Recently I learned that rsync does a checksum of every file > > transferred. I thought it might be interesting to record the path and > > checksum of each file in a table. On future backups, the checksum of >

Re: Use rsync's checksums to deduplicate across backups

2011-11-19 Thread Andrea Gelmini
2011/11/3 Alex Waite : >    Recently I learned that rsync does a checksum of every file > transferred.  I thought it might be interesting to record the path and > checksum of each file in a table.  On future backups, the checksum of I guess you can be interested in these projects: - lessfs:¹ dedup

Re: Use rsync's checksums to deduplicate across backups

2011-11-06 Thread Dan Stromberg
On Thu, Nov 3, 2011 at 7:22 AM, Alex Waite wrote: > > > > Check out http://backuppc.sourceforge.net/, it's perl-based backup tool, > > using rsync and doing exactly what you ask for. > > > I have looked at BackupPC before (and it is a nice piece of > software), and it does hardlink across all

Re: Use rsync's checksums to deduplicate across backups

2011-11-06 Thread Cameron Simpson
On 04Nov2011 10:27, Chris Dunlop wrote: | On Thu, Nov 03, 2011 at 09:34:53AM -0500, Alex Waite wrote: | >> Not a direct answer, but this may do what you want: | >> | >>  http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=link-by-hash.diff | >> | >>  This patch adds the --link-by-hash=DIR option

Re: Use rsync's checksums to deduplicate across backups

2011-11-03 Thread Chris Dunlop
On Thu, Nov 03, 2011 at 09:34:53AM -0500, Alex Waite wrote: >> Not a direct answer, but this may do what you want: >> >>  http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=link-by-hash.diff >> >>  This patch adds the --link-by-hash=DIR option, which hard >> links received >>  files in a link fa

Re: Use rsync's checksums to deduplicate across backups

2011-11-03 Thread Carlos Carvalho
Alex Waite (alexq...@gmail.com) wrote on 2 November 2011 20:09: >Recently I learned that rsync does a checksum of every file >transferred. I thought it might be interesting to record the path and >checksum of each file in a table. On future backups, the checksum of >a file being backed up

Re: Use rsync's checksums to deduplicate across backups

2011-11-03 Thread Alex Waite
> Not a direct answer, but this may do what you want: > >  http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=link-by-hash.diff > >  This patch adds the --link-by-hash=DIR option, which hard links received >  files in a link farm arranged by MD4 file hash.  The result is that the > system >  wi

Re: Use rsync's checksums to deduplicate across backups

2011-11-03 Thread Alex Waite
> > Check out http://backuppc.sourceforge.net/, it's perl-based backup tool, > using rsync and doing exactly what you ask for. > I have looked at BackupPC before (and it is a nice piece of software), and it does hardlink across all backups, but believe it does its own checksum on top of what rs

Re: Use rsync's checksums to deduplicate across backups

2011-11-03 Thread Johannes Totz
On 03/11/2011 01:09, Alex Waite wrote: > I apologize if this has already been discussed before, but as of > yet I have been unable to find any info on the topic. > I have a very simple (and common) disk based backup system using > rsync, hard links, and a little bit of perl to glue it toget

Re: Use rsync's checksums to deduplicate across backups

2011-11-02 Thread Chris Dunlop
On 2011-11-03, Alex Waite wrote: > I apologize if this has already been discussed before, but as of > yet I have been unable to find any info on the topic. > I have a very simple (and common) disk based backup system using > rsync, hard links, and a little bit of perl to glue it together.

Use rsync's checksums to deduplicate across backups

2011-11-02 Thread Alex Waite
I apologize if this has already been discussed before, but as of yet I have been unable to find any info on the topic. I have a very simple (and common) disk based backup system using rsync, hard links, and a little bit of perl to glue it together. Remote machines are backed up regularly us