2008/1/3, Don Armstrong <[EMAIL PROTECTED]> wrote:
> On Thu, 03 Jan 2008, Jason Spiro wrote:
> > http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive will
> > help. Yahoo and MSNBot both support it. I bet other major bots
> > support it too. So we can allow Yahoo and MSNBot (plus Googlebot
On Thu, 03 Jan 2008, Jason Spiro wrote:
> http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive will
> help. Yahoo and MSNBot both support it. I bet other major bots
> support it too. So we can allow Yahoo and MSNBot (plus Googlebot, if
> they support it too) and block everyone else.
Googl
2 matches
Mail list logo
