Thanks for some interesting pointers. Unless I'm missing something it seems
that rsync will still have to go through each filename and see if it
matches --include. We're kinda pushing it here, but have around 500,000
files. Files need to be synced at the longest once a minute, and my
experiance so far is that simply building a large filelist (rsync appears to
do its entire search and *then* transfer) is relativly timeconsuming, even
if few of the files match.
What has worked as a stop gap measure is to pass the filelist as a command
line option such as
rsync 'cat filelist' dest::etc
My concern was that there would be some limitation there, but we've managed
all right with a small 800 file test. My feeling is that if rsync
accepts a whole set of files on the command line it would be relativly
straightforward to have it build a filelist from, well, a filelist. I'm
gonna have one of the guys here take a look at patching it to some behavior
like that.
August
> -----Original Message-----
> From: Goswin Brederlow [mailto:[EMAIL PROTECTED]]On
> Behalf Of Goswin Brederlow
> Sent: Tuesday, January 02, 2001 7:21 PM
> To: Bennett Todd
> Cc: Goswin Brederlow; August Zajonc; [EMAIL PROTECTED]
> Subject: Re: Transfering File List
>
>
> >>>>> " " == Bennett Todd <[EMAIL PROTECTED]> writes:
>
> > 2001-01-02-15:32:35 Goswin Brederlow:
> >> rsync $FLAGS $HOST::debian/pool/ --include-from .filelist
> >> --exclude '*' $DEST/pool/
> >>
> >> This will mirror everything from HOST::debian/pool to
> >> DEST/pool, so there you see how to shorten the path.
> >>
> >> But "--exclude '*'" will ignore all files except what has been
> >> allowed by "--include-from .filelist".
>
> > An additional frob worth noting is that this isn't _quite_ as
> > simple as you might hope; as explained in the rsync man page,
> > the matching of files against include/exclude patterns is tried
> > with each directory prefix leading up to the path, which means
> > to sync the file a/b/c you need to include (in .filelist, in
> > the above example):
>
> > a/ a/b/ a/b/c
>
> > The attached perl script will take a list of files and fill in
> > the needed parent directories.
>
> My .filelist only has the files in it, but with the full path. Also
> all directories are already present locally. Works fine, meaning it
> downloads all files in .filelist.
>
> MfG
> Goswin
>
