Re: filelist caching optimization proposal

2005-05-23 Thread Wayne Davison
On Mon, May 23, 2005 at 03:24:07PM +0200, Edwin Eefting wrote: > My idea is to create a patch for something like a --cache option that > will use a cached version of the filelist: Something like that would be fairly easy to write, but only if there are no conflicts between the cache and the live d

Re: filelist caching optimization proposal

2005-05-23 Thread Edwin Eefting
On Monday 23 May 2005 16:09, Christoph Biedl wrote: > > -What are the most likely problems i would run into when i try to > > implement this? > > You can expect a feature request that allows to manipulate certain parts > of the cache only (re-scan or delete a subtree). This would turn the > cache i

Re: filelist caching optimization proposal

2005-05-23 Thread Edwin Eefting
On Monday 23 May 2005 16:29, Carson Gaspar wrote: > --On Monday, May 23, 2005 03:24:07 PM +0200 Edwin Eefting <[EMAIL PROTECTED]> > > wrote: > > My idea is to create a patch for something like a --cache option that > > will use a cached version of the filelist: This way instead of creating > > the

Re: filelist caching optimization proposal

2005-05-23 Thread Carson Gaspar
--On Monday, May 23, 2005 03:24:07 PM +0200 Edwin Eefting <[EMAIL PROTECTED]> wrote: My idea is to create a patch for something like a --cache option that will use a cached version of the filelist: This way instead of creating the filelist every time (100.000's of system calls, diskaccesses),

Re: filelist caching optimization proposal

2005-05-23 Thread Christoph Biedl
Edwin Eefting wrote... > -What are the opinions of other people on this list? Sounds like a great idea for me but I'm just an rsync user. > -Would it be easy to implement, or would it give too much trouble? Without looking into the sources I think it should not be that difficult to dump the l

filelist caching optimization proposal

2005-05-23 Thread Edwin Eefting
Hi, As a gentoo-user i frequently run the emerge sync command, which in turn does a rsync with the mainserver. The 'problem' is that the portage directory tree contains about 19.000 directories and 96.000 files. So building the filelist takes a pretty long time, because of the many disk accesse

Re: Filelist caching

2003-02-18 Thread Andrew J. Schorr
Hi Rogier, On Sat, Feb 15, 2003 at 05:05:16PM +0100, Rogier van Eeten wrote: > On Wed, Feb 12, 2003 at 05:18:11PM -0500, Andrew J. Schorr wrote: > > On Wed, Feb 12, 2003 at 10:51:19AM -0500, Andrew J. Schorr wrote: > > > > I was wondering... is there a way to cache that filelist? Our mirrors > > >

Re: Filelist caching

2003-02-15 Thread Rogier van Eeten
On Wed, Feb 12, 2003 at 05:18:11PM -0500, Andrew J. Schorr wrote: > On Wed, Feb 12, 2003 at 10:51:19AM -0500, Andrew J. Schorr wrote: > > > I was wondering... is there a way to cache that filelist? Our mirrors > > > are updated once, or twice a day, it could speed up downloads when I > > > create a

Re: Filelist caching

2003-02-12 Thread Andrew J. Schorr
On Wed, Feb 12, 2003 at 10:51:19AM -0500, Andrew J. Schorr wrote: > > I've noticed every time someone does an rsync-request on my ftp-site > > (which also provides rsync as mirror method), rsyncd creates a filelist. > > This is a quite IO and CPU intensive procedure, especially for things > > mirro

Re: Filelist caching

2003-02-12 Thread Rogier van Eeten
On Wed, Feb 12, 2003 at 10:51:19AM -0500, Andrew J. Schorr wrote: > Please take a look at the --files-from feature that is now in the CVS tree, > courtesy of Wayne Davison. That should do what you want. This is probably a silly question, but which tree? I've built the tree I got when I did a 'c

Re: Filelist caching

2003-02-12 Thread Andrew J. Schorr
Hi Rogier, > I've noticed every time someone does an rsync-request on my ftp-site > (which also provides rsync as mirror method), rsyncd creates a filelist. > This is a quite IO and CPU intensive procedure, especially for things > mirrors like FreeBSD with lots of little files. > > I was wonderin

Re: Filelist caching

2003-02-11 Thread Greger Cronquist
I too think this would be an interesting feature. --- Rogier van Eeten <[EMAIL PROTECTED]> skrev: > Hi, > > I've noticed every time someone does an > rsync-request on my ftp-site > (which also provides rsync as mirror method), rsyncd > creates a filelist. > This is a quite IO and CPU intensive p

Filelist caching

2003-02-11 Thread Rogier van Eeten
Hi, I've noticed every time someone does an rsync-request on my ftp-site (which also provides rsync as mirror method), rsyncd creates a filelist. This is a quite IO and CPU intensive procedure, especially for things mirrors like FreeBSD with lots of little files. I was wondering... is there a wa