On Tue 30 Jan 2007, Wayne Davison wrote: > On Tue, Jan 30, 2007 at 02:38:42PM +0100, Paul Slootman wrote: > > It also did the sync in under 2 hours, whereas the non-incremental > > version would take up to 12 hours, and use up to 500MB of memory. > > Awesome! That's an even more dramatic improvement than what I was > anticipating for large hierarchies. Thanks for your help testing
It now only has to keep 1% of the total list of files in memory (there are 100 subdirectories which don't have any further subdirs). However, I may have exaggerated the 12 hours, according to the logs it used to be around 5 hours, so the improvement is still pretty good, and the memory usage reduction is very useful as well. Here's some output from a log that's generated; the "File list generation time" and "File list transfer time" is perhaps less useful nowadays :) 20070130-113432 doing images-17 Number of files: 3766788 Number of files transferred: 194 Total file size: 31635945072 bytes Total transferred file size: 2453786 bytes Literal data: 633820 bytes Matched data: 1819966 bytes File list size: 58092428 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 22425 Total bytes received: 58745190 sent 22425 bytes received 58745190 bytes 9103.50 bytes/sec total size is 31635945072 speedup is 538.32 20070130-132207 doing images-18 Number of files: 3843863 Number of files transferred: 74 Total file size: 32515860221 bytes Total transferred file size: 1017500 bytes Literal data: 381837 bytes Matched data: 635663 bytes File list size: 59415970 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 9884 Total bytes received: 59804946 sent 9884 bytes received 59804946 bytes 9102.16 bytes/sec total size is 32515860221 speedup is 543.61 20070130-151138 doing ... It used to be like this: 20070105-035712 doing images-16 Number of files: 4338185 Number of files transferred: 472 Total file size: 36888804858 bytes Total transferred file size: 3467730 bytes Literal data: 2254171 bytes Matched data: 1213559 bytes File list size: 71031147 File list generation time: 5601.976 seconds File list transfer time: 0.000 seconds Total bytes sent: 33097 Total bytes received: 73313978 sent 33097 bytes received 73313978 bytes 4857.90 bytes/sec total size is 36888804858 speedup is 502.93 20070105-080853 doing images-17 Number of files: 3766442 Number of files transferred: 682 Total file size: 31632304810 bytes Total transferred file size: 5235606 bytes Literal data: 3479744 bytes Matched data: 1755862 bytes File list size: 61852761 File list generation time: 6170.866 seconds File list transfer time: 0.000 seconds Total bytes sent: 52753 Total bytes received: 65373727 sent 52753 bytes received 65373727 bytes 3546.05 bytes/sec total size is 31632304810 speedup is 483.48 20070105-131639 ... > this, and please keep us apprised of how things are working out. Still going strong :) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
