Hello.

Doesnt robots.txt "Crawl-Delay" directive satisfy your needs? 
Normal spiders should obey robots.txt, if they dont - they can be banned.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,243670,243674#msg-243674

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Reply via email to