At 01:13 PM 6/3/2015 +1200, Robin Sheat wrote:
Paul A schreef op di 02-06-2015 om 19:48 [-0400]:
> How do you "slow it down"? To the best of my knowledge it's an on/off
> choice. See
>
<https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt>.
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive
PS: no need to CC me in replies, I'm on the mailing list.
Robin -- thanks (and apologies for the unintended Cc, end of a long day, etc).
This was specifically for Google, so I had a look at the wiki article and
did some more digging. It seems that at least Bing and Yahoo (neither of
whom appear to abuse our servers) support the Crawl-Delay, but Google does
not, they have a differing method explained at
<https://support.google.com/webmasters/answer/48620?hl=en>.
I'll have to look more closely, but it may not be quite what I'm looking
for as it involves their proprietorial "webmaster tools" rather than robots.txt
Best -- Paul
_______________________________________________
Koha mailing list http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha
