I believe I've figured out why the process was taking so long...or at least have a theory. In the end it appears that much of the data was being sent even though the "true" amount of data change was less than 7% of the filesize.
Exchange uses a database page size of 4K. Many times a page is deleted and then new data is written to that page (delete a message, new message arrives). Exchange will try to keep the data file size constant by reusing freed up space and it will do online "defragmentation" nightly by default. Defragmentation might be the wrong term because online defragmentation really "makes additional database space available by detecting and removing database objects that are no longer being used." Although only 7% of the file is changing, the overall number of data pages would approach 1.5 million. In all likelihood, these pages would be spread throughout the file. So if the usual approach of making the blocksize larger to process the file is used then rsync actually performs worse. This is because a change in a single 4K data page (likely occurrence) will cause the entire block to be sent. This is what I was seeing in the earlier tests, increasing blocksize decreases performance by sending more data. When I changed the blocksize to be close to the default of sqrt(filesize) but rounded down to a function of 4K, rsync performance is much better. The performance of a 4K rounded blocksize is better than the default (in this case, 262144). I'm continuing to test to find the "best" blocksize for these types of files. I'm just sending this info for future reference for those using rsync for large Exchange files or other database files. Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
