Re: Reducing rsync cost

2010-12-01 Thread Ask Bjørn Hansen
On 11/24/10 1:23, Nicholas Clark wrote: The symlink tree is built by scripts, isn't it? Are they available? Actually; since earlier this year it's not being built anymore. I promised to write a note announcing it but haven't gotten around to it. So inadvertently we got to test if anyone car

Re: Reducing rsync cost

2010-11-27 Thread Johan Vromans
Ask Bjørn Hansen writes: > On 11/24/10 1:23, Nicholas Clark wrote: >> The symlink tree is built by scripts, isn't it? Are they available? > So inadvertently we got to test if anyone cared. The answer is no. :-) I can't say I'm not anyone, but apparently one of very few :( . -- Johan

Re: Reducing rsync cost

2010-11-27 Thread Ask Bjørn Hansen
On 11/24/10 1:23, Nicholas Clark wrote: The symlink tree is built by scripts, isn't it? Are they available? Actually; since earlier this year it's not being built anymore. I promised to write a note announcing it but haven't gotten around to it. So inadvertently we got to test if anyone car

Re: Reducing rsync cost

2010-11-25 Thread David Precious
On Wednesday 24 November 2010 10:51:25 David Golden wrote: > The new "fast" CPAN mirrors use File::Rsync::Mirror::Recent, which > uses the new RECENT.* files to manage the synchronization process. > Those files record recent changes (adds/deletes) to the frequently > changing "authors/" and "module

Re: Reducing rsync cost

2010-11-24 Thread Andreas J. Koenig
> On Tue, 23 Nov 2010 22:24:18 +0100, David Landgren > said: > I shouldn't even bother replying, but I wouldn't want the archives to > think that silence indicates tacit agreement. To give you an update: five tier-1 CPAN sites are pulling a sync every 20 seconds and every sync takes

Re: Reducing rsync cost

2010-11-24 Thread David Golden
On Wed, Nov 24, 2010 at 4:23 AM, Nicholas Clark wrote: > Instead of running rsync over the whole tree, it can change to run a top > level script that runs rsync over the parts that have to be copied, and then > run the symlink generation on the parts that can be recreated locally. The new "fast"

Re: Reducing rsync cost

2010-11-24 Thread Nicholas Clark
On Tue, Nov 23, 2010 at 10:24:18PM +0100, David Landgren wrote: > On 22/11/2010 15:18, David Nicol wrote: > >On Mon, Nov 22, 2010 at 4:37 AM, David Landgren wrote: > >>Yeah, this is the killer. In an ideal world, we would kill the symlinks > >>such > >>as authors/id/*, modules/by-category/*, modu

Re: Reducing rsync cost

2010-11-23 Thread David Landgren
On 22/11/2010 15:18, David Nicol wrote: On Mon, Nov 22, 2010 at 4:37 AM, David Landgren wrote: Yeah, this is the killer. In an ideal world, we would kill the symlinks such as authors/id/*, modules/by-category/*, modules/by-module/* and so on. These could be recreated via shell scripts locally o

Re: Reducing rsync cost (was: Re: Using a better compression than .gz for one's CPAN modules)

2010-11-22 Thread David Nicol
On Mon, Nov 22, 2010 at 4:37 AM, David Landgren wrote: > Yeah, this is the killer. In an ideal world, we would kill the symlinks such > as authors/id/*, modules/by-category/*, modules/by-module/* and so on. These > could be recreated via shell scripts locally on mirrors for people who wish > to ma