On Sat, Jul 12, 2014 at 1:25 PM, Miles Fidelman <mfidel...@meetinghouse.net> wrote:
> Hi Folks, > > Ever once in a while, a crawler comes along and starts indexing our site - > and in the process pushes our server's load average through the roof. > > Short of blocking the crawlers, can anybody suggest some quick tuning > adjustments to make, to reduce load (setting the max. number of servers > and/or requests, renicing processes)? > Use robots.txt to block access to dynamically generated resources which are expensive to generate and not necessary for search hits? Is it using a lot of concurrent requests, or is the main load issue due to the cost of the requests it is making? > > Yes - my next step is to go pour through manuals - but I expect others > have done this enough to be able to point me at a few specific config file > lines to change, and specific commands for identifying and renicing > processes. > > Thanks very much, > > Miles Fidelman > > -- > In theory, there is no difference between theory and practice. > In practice, there is. .... Yogi Berra > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > > -- Born in Roswell... married an alien... http://emptyhammock.com/
