On Thu, 03 Jan 2008, Jason Spiro wrote: > http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive will > help. Yahoo and MSNBot both support it. I bet other major bots > support it too. So we can allow Yahoo and MSNBot (plus Googlebot, if > they support it too) and block everyone else.
Google doesn't, unfortunatly. I've already had a bit of a discussion about this particular issue with them. [The other half of the problem is that the Crawl delay directive is per-bot, and not indexer-wide.] Don Armstrong -- Sentenced to two years hard labor (for sodomy), Oscar Wilde stood handcuffed in driving rain waiting for transport to prison. "If this is the way Queen Victoria treats her prisoners," he remarked, "she doesn't deserve to have any." http://www.donarmstrong.com http://rzlab.ucr.edu
