I am following this thread with great interest. I have a similar issue as Vishwajit and have resorted to adding the connectionTimeout to get rid of a large number of RUNNABLE threads.
But mod_jk does not like tomcat threads timing out and logs the message "increase the backend idle connection timeout or the connection_pool_minsize" in the mod_jk logs which leads me to believe that its apache thats not letting go of the threads in my case. ________________________________ From: Christopher Schultz <ch...@christopherschultz.net> To: Tomcat Users List <users@tomcat.apache.org> Sent: Thursday, May 21, 2009 1:05:21 PM Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Vishwajit, On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote: > [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should > not be an issue. The only reason why I thought that was an issue was > that I was observing that the none of the RUNNABLE connections were > not being used to serve new requests, only the WAITING ones were - > and I do know for sure that the RUNNABLE threads were not servicing > any existing requests as I was the only one using the system then. It seems pretty clear that this is what your problem is. See if you can follow the order of events described below: 1. Tomcat and Apache httpd are started. httpd makes one or more (persistent) AJP connections to Tomcat and holds them open (duh). Each connection from httpd->Tomcat puts a Java thread in the RUNNABLE state (though actually blocked on socket read, it's not "really" runnable) 2. Some requests are received by httpd and sent over the AJP connections to Tomcat (or not ... it really doesn't matter) 3. Time passes, your recycle_timeout (300s) or cache_timeout (600s) expires 4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully follows your instructions for closing the connections expired in #3 above (note that Tomcat has no idea that the connection has been closed, and so those threads remain in the RUNNABLE state, not connected to anything, lost forever) 5. A new connection (or multiple new connections... not sure exactly how mod_jks connection expiration-and-reconnect logic is done) is made to Tomcat which allocates a new thread (or threads) which is/are in the RUNNABLE state Rinse, repeat, your server chokes to death when it runs out of threads. The above description accounts for your "loss" of 4 threads at a time: your web browser requests the initial page followed by 3 other assets (image, css, whatever). Each one of them hits step #4 above, causing a new AJP connection to be created, with the old one still hanging around on the Tomcat side just wasting a thread and memory. By setting connectionTimeout on the AJP <Connector>, you are /doing what you should have done in the first place, which is match mod_jk cache_timeout with Connector connectionTimeout/. This allows the threads on the Tomcat side to expire just like those on the httpd side. They should expire at (virtually) the same time and everything works as expected. This problem is compounded by your initial configuration which created 10 connections from httpd->Tomcat for every (prefork) httpd process, resulting in 9 useless AJP connections for every httpd process. I suspect that you were expiring 10 connections at a time instead of just one, meaning that you were running out of threads 10 times faster than you otherwise would. Suggestions: 1. Tell your ops guys we know what we're talking about 2. Upgrade mod_jk 3. Set connection_pool_size=1, or, better yet, remove the config altogether and let mod_jk determine its own value 4. Remove all timeouts unless you know that you have a misbehaving firewall. If you do, enable cping/cpong (the preferred strategy by at least one author or mod_jk) - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe WQCfTNbSLTgNtHcTbTAu5kw5igicNMw= =0EWv -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
