-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
First, don't use -z on a local copy. It will only make rsync slower
for no reason at all.
Second, 45 million files means 90 million calls to stat(). This will
take a while even if nothing needs copying.
On 01/21/2016 03:20 AM, dbonde+forum+rsync.lists.samba....@gmail.com
wrote:
> I run a rsync job transferring about 45 million files/approximately
> 1.8 TB data (a Mac OS X Time Machine backup) over a 100 MBit
> connection.
>
> I use rsync 3.1.1 from MacPorts (I first tried the built in rsync,
> version 2.6.9, since it has a Mac OS X specific cache parameter,
> but it ran out of memory) with the following parameters
>
> % rsync -HzvhErlptgoDW --stats --progress --out-format="%t %f %b"
> /source/ /destination/
>
> The source is an external 3.5" HDD connected with Firewire 800.
> The destination is a sparse disk image bundle mounted locally (but
> its "source file" is on a network storage). Initially I got good
> speeds, 7-9 MB/s for reasonably large files but the longer this
> operation has been going on (I restarted it three days ago, see
> below), the slower it gets. There are also long pauses when nothing
> happens, like this:
>
> 2011-01-22-070305/Macintosh HD/Library/Application
> Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents
/Resources/Bamboo.mailstationery/Contents/Resources/Mask3.png
>
>
1.28K 100% 3.26kB/s 0:00:00 (xfr#48406, ir-chk=1050/4166332)
>
> 2016/01/16 18:26:48
> Volumes/src/Backups.backupdb/mm/2011-01-22-070305/Macintosh
> HD/Library/Application
> Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents
/Resources/Bamboo.mailstationery/Contents/Resources/Mask3.png
>
>
313
>
> 2011-01-22-070305/Macintosh HD/Library/Application
> Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents
/Resources/Bamboo.mailstationery/Contents/Resources/banner-green.jpg
>
> 32.26K 100% 0.00kB/s 0:00:00 (xfr#48407, ir-chk=1049/4166332)
>
> 2016/01/16 19:17:37
> Volumes/2TB/Backups.backupdb/mm/2011-01-22-070305/Macintosh
> HD/Library/Application
> Support/Apple/Mail/Stationery/Apple/Contents/Resources/Photos/Contents
/Resources/Bamboo.mailstationery/Contents/Resources/banner-green.jpg
>
>
31279
>
> As you can see, the first file is finished 18:26, the second file
> 19:17, almost an hour for a file that is just 32 kB.
>
> I don't think the transfer is CPU limited. There are some CPU
> spikes but generally CPU load is less than 10%. The three rsync
> processes spawned by this operation has, all in all, used almost
> exactly 5h of CPU time in the 72h the transfer has been going on.
> The computer itself idles 23h a day.
>
> Nor is memory a problem. Memory pressure has been "green" since
> the operation begun.
>
> Kernel task has accumulated quite a bit of CPU time (57h when I
> write this), but on the other hand, the uptime is 25 days and all
> these 57h can't have been consumed by rsync.
>
> Some final details
>
> * I had had this process running for a couple of days when I
> restarted it to get better logging three days ago. It took nine
> hours before the first file was transferred.
>
> * I first used Finder to transfer this directory tree from the
> same source to the same destination. That took 3 days, all in all.
> Now I have spent 6 days and I don't think I even have transferred a
> third of the tree.
>
> * I have tried transferring files between the same source and
> destination outside of this operation and they go at full speed
>
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlag5JkACgkQVKC1jlbQAQcTwwCeKKbLa6UXxuiG7TJidqa1PKcT
lh0AnRfDtS90pUJFmDptXmyGEH09G0pS
=E+fZ
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
