Robin Sheat wrote: > It seems you're right. That's a bit annoying of them. I apparently had > it in my head that all the major search engines did.
Would there be interest in a wiki page listing some of the robots.txt-ignoring bots which can give even a tuned Koha indigestion? Or do you know if some other project already keeps such a list? We've collected quite a few over the years with names like ahrefs and majestic21. Regards, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op http://koha-community.org supporter, web and library systems developer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire (including development) at http://www.software.coop/ _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
