My problem seems to be most similar to this post. We are having intermittent
problems with the JBoss/Tomcat AJP 1.3 connector hanging. From searching
the JBoss and Tomcat user forums, other issues that are similar to mine are:
http://marc.info/?l=tomcat-user&m=116231271819840&w=2
http://www.nabble.com/Problem-with-AJP-connector-td19657959.html#a19657959
neither of which really seems to offer a solution. Here are my specifics:
We are running JBoss 4.2.2 (which uses Tomcat 6) running on Linux (RedHat
5.3) behind an IIS proxy, which is proxying to the JBoss AJP port. I have
left AJP at its default settings in my server.xml:
<!-- A AJP 1.3 Connector on port 8009 -->
<Connector protocol="AJP/1.3" port="8009"
address="${jboss.bind.address}"
redirectPort="8443" />
The behavior I’m observing is only occurring about once every 2 weeks,
making it difficult to reproduce. From the user’s perspective, the site is
unreachable. The IIS proxy is logging this when the problem occurs:
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1011):
(adastarNode) can't receive the response message from tomcat, network
problems or tomcat (172.17.3.240:8009) is down (errno=54)
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [error] jk_ajp_common.c (1766):
(adastarNode) Tomcat is down or refused connection. No response has been
sent to the client (yet)
[Tue Apr 21 04:13:14.775 2009] [3192:2500] [info] jk_ajp_common.c (2186):
(adastarNode) sending request to tomcat failed (recoverable), (attempt=1)
My JBoss instance is not logging any errors during this timeframe. As far
as how to solve the problem, in one case the server was left like this for
several hours and seemed to recover on its own, only to hang again a couple
of hours later; otherwise the only solution that’s worked is to restart
JBoss.
The main difference I can observe in a thread dump is that the AJP acceptor
thread, which is normally in a RUNNABLE state, is in a WAITING state when
the hang occurs:
"ajp-abeitmpr1.andesatpa.com%2F172.17.3.88-8009-Acceptor-0" daemon prio=10
tid=0x00002aaad7a70400 nid=0x7dae in Object.wait()
[0x0000000044240000..0x0000000044240c10]
java.lang.Thread.State: WAITING (on object monitor)
Lately I’ve been trying to also use netstat to look at the problem when a
hang occurs, but I’m not sure I’ve caught it during a true hang. It appears
to me that I have a growing number of ESTABLISHED connections prior to the
hang, plus one CLOSE_WAIT connection:
[it...@abeitmpr1 log]$ netstat -vatn |grep 8009
tcp 0 0 172.17.3.88:8009 0.0.0.0:*
LISTEN
tcp 516 0 172.17.3.88:8009 172.17.5.42:2154
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3690
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2159
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2158
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2144
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3680
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2171
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2170
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1395
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2935
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4724
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2120
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2375
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2119
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2118
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2372
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1114
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2143
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1116
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2131
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3923
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2133
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2132
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2347
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1834
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2093
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1837
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2092
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2348
ESTABLISHED
tcp 795 0 172.17.3.88:8009 172.17.5.42:2080
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2336
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2086
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2105
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2360
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1592
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2111
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2366
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2099
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2359
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1288
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4610
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2311
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2309
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2308
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4635
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2335
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2079
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4126
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2334
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2323
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2835
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2322
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1809
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4884
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3049
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2286
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2285
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1772
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2529
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2273
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2272
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2277
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2297
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3064
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2294
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2248
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1736
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1224
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1219
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2247
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2246
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2266
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2259
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1233
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2260
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2221
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2220
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1443
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2208
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2214
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2235
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2234
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3002
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3513
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3518
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3260
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4019
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4789
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2184
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:3213
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:1667
ESTABLISHED
tcp 516 0 172.17.3.88:8009 172.17.5.42:2183
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2182
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:4767
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2207
ESTABLISHED
tcp 0 0 172.17.3.88:8009 172.17.5.42:2204
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2195
ESTABLISHED
tcp 514 0 172.17.3.88:8009 172.17.5.42:2196
ESTABLISHED
tcp 1 0 172.17.3.88:8009 172.17.1.73:4169
CLOSE_WAIT
If anyone has any leads on this problem, or suggestions for things to try,
it would be appreciated.
--
View this message in context:
http://www.nabble.com/AJP-connections-just-stop-working-tp1118618p23603003.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
