On Wed, Jan 28, 2004 at 02:12:02PM -0500, Eugen Luca wrote:
> server load has been increased so much and the server crashed, as well
> has gone out of memory
The rsync version in CVS has a number of memory-saving optimizations in
it. Just the file list reductions (if we ignore -H for now) would save
~114MB of memory in a list of 5,000,000 files, and that's just for one
of the processes--if the sending and receiving are being done on the
same machine, multiply that savings by 2. The version in CVS also
prevents a copy-on-write memory-duplication between the two processes on
the receiving side, which should cut the file-list memory use in half
(i.e. a savings of ~350MB for 5,000,000 files). The memory use of the
-H option used to cause the entire file list to be duplicated, so that's
another ~350MB savings in memory footprint on the receiving side, though
we do need to add back in a variable amount of memory-use depending on
how many files are actually linked together in the transfer (figure
something like 20 bytes or so per linked file).
So, you might want to give the CVS version a try. There's info here on
how to get it:
http://rsync.samba.org/download.html
You can use CVS, rsync the latest files (from the unpacked/rsync dir),
or grab a nightly snapshot in tar form.
..wayne..
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
