[EMAIL PROTECTED] wrote:
Hello
I have a similar problem. I want to rsync a huge number of files and I'm
running out of memory.
You might give the machine more swap.
As it appeared, it wasn't a freeze - it just took a lot of time to
generate the list. When I came back in the morning, the files were
transferred.
As I remember, the first copy (many many files) took about 14 hours to
complete; the second time could be even slower, as rsync had to compare
two directories, each with over 7 million files, full of hardlinks.
Over 1 GB was on swap, and swap is not the fastest raplacement for RAM...
I'm using Dirvish to backup my servers. For those who haven't heard
about it it's sort of "front end" for rsync making a copy of the files
for the first backup, and then copying modified files and hardlinking
unmodified files every day.
I have 600 000 files, and I'm keeping 120 backups on disk. Files are not
changing too much, I have something like 120 GB of data and 180 GB of
backup.
That is roughly 600 000 files with 100 hardlinks on each, so 60 000 000
files.
Thats nearly 10 times as much as I had :)
From what I've been reading in the FAQ and the ML rsync will use 100
bytes per files (6 GB), and I'll probably get a x3 "bonus" (18 GB) for
using the --delete and -H options.
How could I get these files rsynced ?
Perhaps investing in RAM would be good idea.
By the way what is the memory usage on the receiver side ?
For the first rsync there is no files receiver side, so it's no problem.
But after that first rsync memory usage will be the same on sender and
receiver side, as they'll both have to create their file list. Am I right ?
You could mount the other side (NFS, iSCSI etc., but that depends on the
connection you have), so that there would be only one rsync instance
running?
My servers are running Debian Sarge i.e. rsync 2.6.4
Upgrade to rsync 2.6.8 on both sides?
--
Tomasz Chmielewski
http://wpkg.org
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
