On 18.05.2009 20:33, kvancamp wrote: > My problem seems to be most similar to this post. We are having intermittent > problems with the JBoss/Tomcat AJP 1.3 connector hanging. From searching > the JBoss and Tomcat user forums, other issues that are similar to mine are: > http://marc.info/?l=tomcat-user&m=116231271819840&w=2 > http://www.nabble.com/Problem-with-AJP-connector-td19657959.html#a19657959 > > neither of which really seems to offer a solution. Here are my specifics: > > We are running JBoss 4.2.2 (which uses Tomcat 6) running on Linux (RedHat > 5.3) behind an IIS proxy, which is proxying to the JBoss AJP port. I have > left AJP at its default settings in my server.xml: > <!-- A AJP 1.3 Connector on port 8009 --> > <Connector protocol="AJP/1.3" port="8009" > address="${jboss.bind.address}" > redirectPort="8443" /> > The behavior I’m observing is only occurring about once every 2 weeks, > making it difficult to reproduce. From the user’s perspective, the site is > unreachable. The IIS proxy is logging this when the problem occurs: > [Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1011): > (adastarNode) can't receive the response message from tomcat, network > problems or tomcat (172.17.3.240:8009) is down (errno=54) > [Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1766): > (adastarNode) Tomcat is down or refused connection. No response has been > sent to the client (yet) > [Tue Apr 21 04:13:14.775 2009] [3192:2500] [info] jk_ajp_common.c (2186): > (adastarNode) sending request to tomcat failed (recoverable), (attempt=1) > > My JBoss instance is not logging any errors during this timeframe. As far > as how to solve the problem, in one case the server was left like this for > several hours and seemed to recover on its own, only to hang again a couple > of hours later; otherwise the only solution that’s worked is to restart > JBoss. > > The main difference I can observe in a thread dump is that the AJP acceptor > thread, which is normally in a RUNNABLE state, is in a WAITING state when > the hang occurs: > "ajp-abeitmpr1.andesatpa.com%2F172.17.3.88-8009-Acceptor-0" daemon prio=10 > tid=0x00002aaad7a70400 nid=0x7dae in Object.wait() > [0x0000000044240000..0x0000000044240c10] > java.lang.Thread.State: WAITING (on object monitor) > > Lately I’ve been trying to also use netstat to look at the problem when a > hang occurs, but I’m not sure I’ve caught it during a true hang. It appears > to me that I have a growing number of ESTABLISHED connections prior to the > hang, plus one CLOSE_WAIT connection: > [it...@abeitmpr1 log]$ netstat -vatn |grep 8009 > tcp 0 0 172.17.3.88:8009 0.0.0.0:* > LISTEN > tcp 516 0 172.17.3.88:8009 172.17.5.42:2154 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3690 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2159 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2158 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2144 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3680 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2171 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2170 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1395 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2935 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4724 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2120 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2375 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2119 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2118 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2372 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1114 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2143 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1116 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2131 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3923 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2133 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2132 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2347 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1834 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2093 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1837 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2092 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2348 > ESTABLISHED > tcp 795 0 172.17.3.88:8009 172.17.5.42:2080 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2336 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2086 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2105 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2360 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1592 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2111 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2366 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2099 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2359 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1288 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4610 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2311 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2309 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2308 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4635 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2335 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2079 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4126 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2334 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2323 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2835 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2322 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1809 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4884 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3049 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2286 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2285 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1772 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2529 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2273 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2272 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2277 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2297 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3064 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2294 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2248 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1736 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1224 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1219 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2247 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2246 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2266 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2259 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1233 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2260 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2221 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2220 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1443 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2208 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2214 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2235 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2234 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3002 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3513 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3518 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3260 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4019 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4789 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2184 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3213 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1667 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2183 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2182 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4767 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2207 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2204 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2195 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2196 > ESTABLISHED > tcp 1 0 172.17.3.88:8009 172.17.1.73:4169 > CLOSE_WAIT > > If anyone has any leads on this problem, or suggestions for things to try, > it would be appreciated.
Can you try to get a couple of full thread dump (e.g. 3 dumps, each 3 seconds apart form each other), when the problem happens next time? If the problem lasts for some time, your could e.g. generate those dumps automatically using cron. Regards, Rainer --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
