Considering it takes 4.5 minutes for a single instance of the request to complete, I think a real solution lies in figuring out why gitweb/git is consuming so many resources in the first place.
Also, I was using the term 'web crawler' loosely in a way that's inclusive with automated feed readers, the bulk of which don't appear to request the robots.txt file, and some of which don't even stop themselves from issuing simultaneous requests for the same feed (which, I imagine, is because they're not programmed to handle a feed that takes over four minutes to pull). I've added the robots.txt because it certainly can't hurt, and I can analyze the requests after a couple of days. However, I don't think it's real solution, either. Clay On Wed, Nov 25, 2009 at 9:15 AM, Paul Poulain <paul.poul...@biblibre.com> wrote: > Clay Fouts a écrit : >> >> I've discovered that a web crawler was opening up large numbers of >> simultaneous requests for the RSS/Atom feed. > > <snip> >> >> If anyone can suggest a solution that can keep >> the system stable while not removing functionality, please let me >> know. >> > > couldn't a robots.txt do the job ? > > -- > Paul POULAIN > http://www.biblibre.com > Expert en Logiciels Libres pour l'info-doc > Tel : (33) 4 91 81 35 08 > > _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
