Michael Jeung wrote:
We currently use DNS round-robin to balance traffic to servers. We've recently run into situations where multiple search engine spiders are crawling our webservers. They appear to be targeting specific webservers by IP address. This defeats DNS round-robin and as a result 1 of our webservers starts responding very slowly since it's full of search engine spiders.
Set up a /robots.txt file in your webserver root which contains appropriate Disallow or Crawl-delay entries to reduce the rate at which spiders make requests, or eliminate such traffic entirely. For example:
http://www.pkix.net/robots.txt -- -Chuck _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
