On Thu 21 Jun 2007, Will Smith wrote: > I run several remote rsyncs off a CentOS 4 server and also back up between > two drives on the same server. > > Performance is far better on the remote rsyncs and the local rsync only > manages 10k/sec despite being on modern hardware.
If the two filesystems you're syncing between are on the same disks, you're causing a lot of disk thrashing during the local sync. That's probably the cause of the slowdown (it has to seek to and fro between the source and the destination). > What I have found is that if I empty the destination directory on the local > backup disk, so rsync has to start from scratch and copy all files, > performance is as to be expected but this isn't feasible for a regular > backup routine as there are almost a million files. That's because it doesn't need to check the existing files then. BTW, don't be fooled by the bytes/sec statistic... If nothing has changed, that will be basically 0 excluding any overhead. That doesn't mean it's very slow... More interesting is the wall clock time taken. > I have also noticed that the load on the local rsync is much higher than the > remote rsyncs, with 90%+ cpu usage. I have tried --bwlimit 100 in a vague Is that including or excluding wait for IO time, which actually is CPU idle time that could be used for processor-intensive tasks while the disk is seeking. > hope it might affect disk io but it didn't help. Rsync version is 2.6.9 > 1.el4.rf from rpm. The directory size of the destination backup directory is > around 30gb. There is one directory where the majority of the files are and > they are jpgs between 5k-150kb. > > The command I am running is simple: /usr/bin/rsync -a --bwlimit=100 --stats > /src/dir1 /src/dir2 /backup/dir As you're not preserving hardlinks (-H), you may observe a significant speedup when using version 3.0.0 (currently only available from the CVS or as a daily snapshot, it's not been released yet). Prior versions of rsync first gather all the file meta data from the source, then from the destination, and then start comparing the lists to see what needs to be transferred. 3.0.0 does the comparing as soon as the first couple of directories have been read on both sides, and that can really speed things up. Unfortunately that speedup doesn't work if hardlinks are to be preserved (although I have hope that some day it might :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
