Try using major subdirectories one level lower. Example sequence from my
automated home machine backup:
rsync -ax --stats / /rsync_backup/
rsync -ax --stats /usr/ /rsync_backup/usr/
rsync -ax --stats /home/ /rsync_backup/home/
Compare to "rsync -a --stats / /backup/"
Ralf Fassel wrote:
We're using rsync 2.6.3 to sync two DELL PowerEdge servers with both
Redhat-EL4 and otherwise nearly identical hardware (2.8/3GHz, 1GB RAM
each). The source machine has a SCSI-RAID1, the destination a
SATA-RAID1 disk attached.
There are 5 filesystems which are rsynced via ssh. On the smaller
filesystems with ~200.000 files/7GB, rsync takes 1-3 minutes:
lion:/atg/ ========= Tue Aug 22 12:13:29 CEST 2006 ==================
receiving file list ... done
--<snip-snip>--
Number of files: 233780
Number of files transferred: 2
Total file size: 6716087965 bytes
Total transferred file size: 1470 bytes
Literal data: 1470 bytes
Matched data: 0 bytes
File list size: 5015765
Total bytes sent: 68
Total bytes received: 5017339
sent 68 bytes received 5017339 bytes 56060.41 bytes/sec
total size is 6716087965 speedup is 1338.56
lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 ==================
233780 files/89 sec = 2626 files/sec
On the large filesystems with ~1.200.000 files/17GB, rsync takes 30
minutes, even when only few files change (and even when invoked with -n):
lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 ==================
receiving file list ... done
--<snip-snip>--
Number of files: 1232323
Number of files transferred: 124
Total file size: 17103825898 bytes
Total transferred file size: 1416942923 bytes
Literal data: 27658581 bytes
Matched data: 1389311598 bytes
File list size: 95325482
Total bytes sent: 643209
Total bytes received: 123363867
sent 643209 bytes received 123363867 bytes 62963.73 bytes/sec
total size is 17103825898 speedup is 137.93
lion:/homes/atg/ ========= Tue Aug 22 12:47:48 CEST 2006 ==================
1232323 files / 1960 sec = 629 files/sec
In comparison, on the large filesystem, 'find'ing all files on the
source system and 'stat'ing them takes 1.75 minutes:
source# time find /homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l
1232287
real 1m41.987s
user 0m7.075s
sys 0m33.216s
On the destination 4 minutes:
dest# time find /backup/lion-homes -xdev -print0 | xargs -0 stat -c "%s %Y"
| wc -l
1233202
real 4m3.650s
user 0m9.895s
sys 0m41.047s
Receiving the 123363867 bytes should not take longer than 20sec on a
100Mbit LAN, and transferring a GB to disk should not take longer than
say a minute (we have write transfer rates of ~50MB/sec on those disks
on each machine).
So what is rsync doing in addition that it takes >30 minutes to sync
the large filesystem?
Thanks
R'
--
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
