On Sun, Dec 09, 2001 at 03:51:29PM +0100, [EMAIL PROTECTED] wrote:
>
> Hi,
>
> I got stuck within some weird prob concerning my 2-node linux cluster and
> the synchronisation tool at hand (rsync-2.5.1pre1).
>
> I have to copy a structure of 70 directories where the data of these
> directories are hardlinked to the data of the 1st directory. Within this
> "orig data" directory, I have about 30.000 files, so the amount of files
> to sync is approx. 2.100.000. The overall size is about 9.2GB.
... [runs out of memory] ...
In case you weren't aware, rsync uses a lot less memory overall if you can
split up the copies into smaller pieces, because it keeps some memory for
every file it touches in a run. In your case, it may be especially
difficult to break up the copy into smaller pieces because of all the
hardlinks.
Ideas:
1. Would it be possible to use symlinks instead of hardlinks? That
would give you more flexibility to split things up however you
like.
2. Perhaps you could break it up into ~70 copies, where each time you
give it the first directory that contains the data and another
one that contains one of the hardlinks.
- Dave Dykstra
