I run a rsync job transferring about 45 million files/approximately 1.8
TB data (a Mac OS X Time Machine backup) over a 100 MBit connection.
I use rsync 3.1.1 from MacPorts (I first tried the built in rsync,
version 2.6.9, since it has a Mac OS X specific cache parameter, but it
ran out of memory) with the following parameters
% rsync -HzvhErlptgoDW --stats --progress --out-format="%t %f %b"
/source/ /destination/
The source is an external 3.5" HDD connected with Firewire 800. The
destination is a sparse disk image bundle mounted locally (but its
"source file" is on a network storage). Initially I got good speeds, 7-9
MB/s for reasonably large files but the longer this operation has been
going on (I restarted it three days ago, see below), the slower it gets.
There are also long pauses when nothing happens, like this:
2011-01-22-070305/Macintosh HD/Library/Application
Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents/Resources/Bamboo.mailstationery/Contents/Resources/Mask3.png
1.28K 100% 3.26kB/s 0:00:00 (xfr#48406, ir-chk=1050/4166332)
2016/01/16 18:26:48
Volumes/src/Backups.backupdb/mm/2011-01-22-070305/Macintosh
HD/Library/Application
Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents/Resources/Bamboo.mailstationery/Contents/Resources/Mask3.png
313
2011-01-22-070305/Macintosh HD/Library/Application
Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents/Resources/Bamboo.mailstationery/Contents/Resources/banner-green.jpg
32.26K 100% 0.00kB/s 0:00:00 (xfr#48407, ir-chk=1049/4166332)
2016/01/16 19:17:37
Volumes/2TB/Backups.backupdb/mm/2011-01-22-070305/Macintosh
HD/Library/Application
Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents/Resources/Bamboo.mailstationery/Contents/Resources/banner-green.jpg
31279
As you can see, the first file is finished 18:26, the second file 19:17,
almost an hour for a file that is just 32 kB.
I don't think the transfer is CPU limited. There are some CPU spikes but
generally CPU load is less than 10%. The three rsync processes spawned
by this operation has, all in all, used almost exactly 5h of CPU time in
the 72h the transfer has been going on. The computer itself idles 23h a day.
Nor is memory a problem. Memory pressure has been "green" since the
operation begun.
Kernel task has accumulated quite a bit of CPU time (57h when I write
this), but on the other hand, the uptime is 25 days and all these 57h
can't have been consumed by rsync.
Some final details
* I had had this process running for a couple of days when I restarted
it to get better logging three days ago. It took nine hours before the
first file was transferred.
* I first used Finder to transfer this directory tree from the same
source to the same destination. That took 3 days, all in all. Now I have
spent 6 days and I don't think I even have transferred a third of the tree.
* I have tried transferring files between the same source and
destination outside of this operation and they go at full speed
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
