On Tue, Oct 21, 2003 at 06:25:51PM -0700, Matt Olson wrote:
> I wanted to flag a problem and offer a possible solution.
>
> The problem:
>
> large rsync operation fails on machines with modest amounts of memory.
>
> Proposal:
>
> Add a parameter to rsync recursive to specify a recursion level (See
> example bash wrapper below). (works with recursive file system rsyncs
> only, i.e. -a or -r) The logic goes:
Limiting the depth of recursion is already supported just
not intuitive.
rsync -r --exclude='/*/*/*/'
Your idea for a shell script to automate picking up the
lower levels is good and could compose the --exclude
pattern. The next step would be to set the job partition
level based on path count as in "find $subtree -print|wc -l".
> if recursion switch true and recursion_level > 0
>
> -rsync this directory only
> -call rsync for each subdirectory with a decremented recursion_level and
> pass the same switches along
>
> else (recursion_level really is 0)
>
> -perform the full rsync (from this level)
>
> What this does is help break up the job into smaller pieces. Otherwise
> rsync can consume hundreds of megabyte of memory attempting to perform a
> single operation. In this scenario you'll see one rsync process for each
> level of recursion.
>
> Here's and example bash script that is an attempt at this idea: (it
> supports the -n options so you can see the calls it makes)
>
> My bash scripting skills need some work, but, you get the idea. If
> someone wants to further develop this script, feel free.
>
> Cheers.
>
>
> #!/bin/bash
[snip]
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
