Re: Caching {filePath,mtime64,checksum} values to speed up execution-time

2014-03-14 Thread Doug Robinson
Wayne: Thank you for responding. On Thu, Mar 13, 2014 at 8:37 PM, Wayne Davison wrote: > On Tue, Mar 11, 2014 at 3:11 PM, Doug Robinson > wrote: > >> I was wondering what folks thought of a proposal to enhance rsync to be >> able to create and maintain a cache of {f

Re: Caching {filePath,mtime64,checksum} values to speed up execution-time

2014-03-12 Thread Doug Robinson
fication by reseting the 64-bit mtime (a trick noted in https://lists.samba.org/archive/rsync/2011-August/026676.html ). That last URL was a find by one of my co-workers. Now I need to go track the "db.diff" patch that Wayne notes and see if I can tweak it to do the 64-bit stuff and so o

Re: Caching {filePath,mtime64,checksum} values to speed up execution-time

2014-03-12 Thread Doug Robinson
AND doing the necessary CPU intensive checksum computation. It would be a huge win. Thank you. Doug >On 03/11/2014 06:11 PM, Doug Robinson wrote: >> Folks: >> >> When using rsync to copy huge amounts of data I've found that a >> significant amount of time is spent

Re: Caching {filePath,mtime64,checksum} values to speed up execution-time

2014-03-11 Thread Doug Robinson
Ah, forgot to mention that we can't preserve time-stamps and matching byte counts are false negatives so we really need to use file checksums as comparisons. Thanks again. Doug -- WANdisco // *Non-Stop Data* t. 925-396-1125 e. doug.robin...@wandisco.com -- Join us in New York and San Franci

Caching {filePath,mtime64,checksum} values to speed up execution-time

2014-03-11 Thread Doug Robinson
Folks: When using rsync to copy huge amounts of data I've found that a significant amount of time is spent computing the checksums. Sometimes hours, ... sometimes days - it depends on the total amount of data checked! And after that sometimes it's only a few files that need to be updated. I've