Hello, We are also haveing exact same issue in our environment. We are using JBoss 4.2.3, Apache 2.2.11 with mod_proxy_ajp using ajp to connect between Apache and JBoss. It happens intermittently and when i happnes all AJP's TCP connections are stayed in ESTABLISHED state on JBoss machine. In thread dump, all AJP threads are in waiting state.
Anyone found solution or work around for this issue? Thanks, GC kvancamp wrote: > > My problem seems to be most similar to this post. We are having > intermittent problems with the JBoss/Tomcat AJP 1.3 connector hanging. > From searching the JBoss and Tomcat user forums, other issues that are > similar to mine are: > http://marc.info/?l=tomcat-user&m=116231271819840&w=2 > http://www.nabble.com/Problem-with-AJP-connector-td19657959.html#a19657959 > > neither of which really seems to offer a solution. Here are my specifics: > > We are running JBoss 4.2.2 (which uses Tomcat 6) running on Linux (RedHat > 5.3) behind an IIS proxy, which is proxying to the JBoss AJP port. I have > left AJP at its default settings in my server.xml: > <!-- A AJP 1.3 Connector on port 8009 --> > <Connector protocol="AJP/1.3" port="8009" > address="${jboss.bind.address}" > redirectPort="8443" /> > The behavior I’m observing is only occurring about once every 2 weeks, > making it difficult to reproduce. From the user’s perspective, the site > is unreachable. The IIS proxy is logging this when the problem occurs: > [Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1011): > (adastarNode) can't receive the response message from tomcat, network > problems or tomcat (172.17.3.240:8009) is down (errno=54) > [Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1766): > (adastarNode) Tomcat is down or refused connection. No response has been > sent to the client (yet) > [Tue Apr 21 04:13:14.775 2009] [3192:2500] [info] jk_ajp_common.c (2186): > (adastarNode) sending request to tomcat failed (recoverable), (attempt=1) > > My JBoss instance is not logging any errors during this timeframe. As far > as how to solve the problem, in one case the server was left like this for > several hours and seemed to recover on its own, only to hang again a > couple of hours later; otherwise the only solution that’s worked is to > restart JBoss. > > The main difference I can observe in a thread dump is that the AJP > acceptor thread, which is normally in a RUNNABLE state, is in a WAITING > state when the hang occurs: > "ajp-abeitmpr1.andesatpa.com%2F172.17.3.88-8009-Acceptor-0" daemon prio=10 > tid=0x00002aaad7a70400 nid=0x7dae in Object.wait() > [0x0000000044240000..0x0000000044240c10] > java.lang.Thread.State: WAITING (on object monitor) > > Lately I’ve been trying to also use netstat to look at the problem when a > hang occurs, but I’m not sure I’ve caught it during a true hang. It > appears to me that I have a growing number of ESTABLISHED connections > prior to the hang, plus one CLOSE_WAIT connection: > [it...@abeitmpr1 log]$ netstat -vatn |grep 8009 > tcp 0 0 172.17.3.88:8009 0.0.0.0:* > LISTEN > tcp 516 0 172.17.3.88:8009 172.17.5.42:2154 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3690 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2159 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2158 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2144 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3680 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2171 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2170 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1395 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2935 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4724 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2120 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2375 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2119 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2118 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2372 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1114 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2143 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1116 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2131 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3923 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2133 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2132 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2347 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1834 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2093 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1837 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2092 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2348 > ESTABLISHED > tcp 795 0 172.17.3.88:8009 172.17.5.42:2080 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2336 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2086 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2105 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2360 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1592 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2111 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2366 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2099 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2359 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1288 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4610 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2311 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2309 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2308 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4635 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2335 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2079 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4126 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2334 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2323 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2835 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2322 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1809 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4884 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3049 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2286 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2285 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1772 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2529 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2273 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2272 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2277 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2297 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3064 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2294 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2248 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1736 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1224 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1219 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2247 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2246 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2266 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2259 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1233 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2260 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2221 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2220 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1443 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2208 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2214 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2235 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2234 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3002 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3513 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3518 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3260 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4019 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4789 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2184 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:3213 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:1667 > ESTABLISHED > tcp 516 0 172.17.3.88:8009 172.17.5.42:2183 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2182 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:4767 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2207 > ESTABLISHED > tcp 0 0 172.17.3.88:8009 172.17.5.42:2204 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2195 > ESTABLISHED > tcp 514 0 172.17.3.88:8009 172.17.5.42:2196 > ESTABLISHED > tcp 1 0 172.17.3.88:8009 172.17.1.73:4169 > CLOSE_WAIT > > If anyone has any leads on this problem, or suggestions for things to try, > it would be appreciated. > > -- View this message in context: http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23737460.html Sent from the Tomcat - User mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
