On 2/21/2014 7:25 AM, rge3 wrote:
I havne't found any ideas for this and thought I might ask here. We have a fairly straightforward proxy_cache setup with a proxy_pass backend. We cache documents for different lengths of time or go the backend for what's missing. My problem is we're getting overrun with bot and spider requests. MSN in particular started hitting us exceptionally hard yesterday and started bringing our backend servers down. Because they're crawling the site from end to end our cache is missing a lot of those pages and nginx has to pass the request on through.
Are they ignoring your robots.txt? _______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
