On Wed, Nov 24, 2010 at 4:23 AM, Nicholas Clark <n...@ccl4.org> wrote: > Instead of running rsync over the whole tree, it can change to run a top > level script that runs rsync over the parts that have to be copied, and then > run the symlink generation on the parts that can be recreated locally.
The new "fast" CPAN mirrors use File::Rsync::Mirror::Recent, which uses the new RECENT.* files to manage the synchronization process. Those files record recent changes (adds/deletes) to the frequently changing "authors/" and "modules/" directories. The fast mirrors use those files to sync with PAUSE every minute or so with very low overhead. The fast mirrors admins are authorized by Andreas to hit PAUSE directly, but I believe that anyone can use it with open CPAN mirrors offering rsync service. While it needs to run as a daemon, as of version 0.0.8-TRIAL, all the memory intensive work happens in child processes and the main daemon is pretty lightweight. (Mine is holding at about 8.8 MB of memory). The current list of "fast" mirrors are cpan.shadowcatprojects.net, cpan.dagolden.com, cpan.hexten.net and cpan.cpantesters.org. While you probably shouldn't hit those every minute without checking with the admins (e.g. me for cpan.dagolden.com) you can probably use F::R::M::Recent to hit them several times an hour with no problem. See http://tinyurl.com/35t9u3k for instructions on using F::R::M::Recent. Regards, David
