On 01/26/2015 09:46 AM, Kevin Fenzi wrote:
I think we added the Crawl-delay several years ago when we were having storage issues. We could definitely try removing it and see if things improve.
10 seconds may be on the high side, but you may still want to keep it and lower it to 1 to 5 seconds. Bing Bot hits my sites like it's running a benchmark.
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
