On 01.06.2009 18:30, CrystalCracker wrote: > Ok.. adding some info to mod jk logs and doing some thread and heap dumps, I > figured that, during peak traffic hours, the garbage collection gets more > frequest and takes longer. This slows down the overall system and the > threads start piling up. Requests that used to take 2 seconds now take a > couple of hundred seconds. The application isn't using the swap space > though. It takes a long as 15 minutes for the system to recover completely > and sometimes I have to restart. > > Now how can I intelligently use the timeout settings to kill the long > running requests and prevent threads from making it worse? > > In httpd.conf, I have: > Timeout 120 > > In worker.properties, I have: > worker.node1.connection_pool_timeout=600 > > In the tomcat AJP connector, I have: > connectionTimeout="600000" > > How can I ensure that no request is taking longer than 3 minutes?
First of all there is no direct way of "killing" the threads inside Tomcat. Once your requests get there, you can't stop them from processing by any kind of timeout set in Apache or mod_jk. So most important: Analyze and fix your memory problem/sizing. That's the only way that will help you really. Apart from that, the question is, what kind of reaction do you want? a) Apache should not wait longer than xxx seconds for a response and then return an error to the client, thus freeing its own resources, but Tomcat will proceed with the request until it starts sending back the response, then noticing that the web server is gone. b) Like a), but if this happens to often, Apache should send no more requests to the node until it gets fast again, in order to shorten the bad period. The requests which fail over to some other node will likely produce an error, if they use a web session (that does not exist on the other nodes). c) Send all requests without sessions to some other node, once a node goes bad. Ad c): You can use the lb method "Busyness" in mod_jk. It will choose the node with the fewest concurrent requests running for the next request (request without session). Beware that a few users reported a problem with Busyness counters slowly getting out of sync with real concurrent requests over time. Ad a) You can set a reply_timeout. If you do not want the complete node being taken out of balancing after a single reply_timeout, you should also set max_reply_timeouts to a big value. Use a recent mod_jk version, if you are using those attributes. Ad b): Like a, but fine tune max_reply_timeouts. Rule of thumb: max_reply_timeout should be double the value of reply_timeouts you want to accept, before taking a node fully our of balancing. Note: with recent mod_jk you can set reply timeouts also per URL, e.g. in case you know that some URLs run really long (reporting etc.) and you want to have e.g. 30 seconds general timeout, but 2 minutes for some well known special URLs. This can be done by using either the Apache environment variable JK_REPLY_TIMEOUT, or alternatively the so-called rule extensions of uriworkermap.properties: http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions Regards, Rainer --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
