Hello!
We have exactly the same problem with the following configuration:
OS: Windows Server 2003 Standard Edition
Apache 2.0.54
mod_jk 1.2.14
Tomcat 5.5.9/5.5.12
JBoss 2.4.4
Our website basically serves JSP-Files and custom webapps, using the Tomcat
instance as Servlet/JSP container and JBoss as backend for EJBs and database
connection. In front of the Tomcat we're using the Apache for handling
requests for static content.
We've experienced connection losses beetween Tomcat and Apache with quite
different periods in between (half an hour up to 2-3 weeks). Both, restart
of Tomcat and/or Apache could not bring the system back to health, so the
only solution is a server reboot.
We could not locate the root cause, so we decided to install the load
balancing mechanism provided by the Jk-Module of the Apache httpd. We set up
2 Tomcat nodes, residing on different machines. The Apache served as initial
handler for the request and delivers requests to our primary Tomcat node
(A). We expected a switch to the second Tomcat node (B) in case of a
failover, so that there would be no downtime for the website. After a
restart of the machine where Tomcat A lives we expected the system to resume
normal operation.
Unfortunatly this did not solve the problem, because the balancer did not
switch to Tomcat B. So the problem seems to reside in mod_jk, which let us
finally switch back to an older version of mod_jk (1.2.6). This scenario is
running for about 6 hours now, so there is no statement possible if this
step was successful.
Otherwise it could be possible that it is a problem of the OS, since you
were running Windows Server 2003 too.
Here some log entries from both, Apache and Tomcat Node A from our load
balancer scenario. It seems that the connection is dropped by the mod_jk
module.
Our config is not kind of exotic, so I would be really surprised if it
should be a configuration error.
Apache:
[Thu Jan 19 08:47:52 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53
[Thu Jan 19 08:47:52 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 08:47:52 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 08:47:52 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 08:53:07 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53
[Thu Jan 19 08:53:07 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 08:53:07 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 08:53:07 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 08:54:10 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53
[Thu Jan 19 08:54:10 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 08:54:10 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 08:54:10 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 08:54:31 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53
[Thu Jan 19 08:54:31 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 08:54:31 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 08:54:31 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:41 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 09:05:41 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:41 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:41 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=2
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:41 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=2
[Thu Jan 19 09:05:41 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=3
[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1758): Error connecting
to tomcat. Tomcat is probably not started or is listening on the wrong port.
worker=tomcatA_xxx failed
[Thu Jan 19 09:05:41 2006] [info] jk_lb_worker.c (662): service failed,
worker tomcatA_xxx is in error state
[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53
[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:46 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 09:05:46 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53
[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:46 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0
[Thu Jan 19 09:05:46 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53
[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:46 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:46 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=2
[Thu Jan 19 09:05:47 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53
[Thu Jan 19 09:05:47 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client
[Thu Jan 19 09:05:47 2006] [info] jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=1
[Thu Jan 19 09:05:47 2006] [info] jk_ajp_common.c (1749): Sending request
to tomcat failed, recoverable operation attempt=2
Tomcat:
2006-01-19 08:53:02,622 [TP-Processor1] WARN org.apache.jk.core.MsgContext
- Error sending end packet
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)
at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)
at java.lang.Thread.run(Thread.java:595)
2006-01-19 08:53:02,622 [TP-Processor1] WARN
org.apache.jk.common.ChannelSocket - processCallbacks status 2
2006-01-19 09:05:35,541 [TP-Processor3] WARN org.apache.jk.core.MsgContext
- Error sending end packet
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)
at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:281)
at
org.apache.catalina.connector.Response.finishResponse(Response.java:483)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)
at java.lang.Thread.run(Thread.java:595)
2006-01-19 09:05:35,541 [TP-Processor3] WARN org.apache.jk.core.MsgContext
- Error sending end packet
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)
at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)
at java.lang.Thread.run(Thread.java:595)
2006-01-19 09:05:35,541 [TP-Processor3] WARN
org.apache.jk.common.ChannelSocket - processCallbacks status 2
2006-01-19 09:05:35,885 [TP-Processor3] WARN org.apache.jk.core.MsgContext
- Error sending end packet
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)
at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
at org.apache.coyote.Response.action(Response.java:182)
at org.apache.coyote.Response.finish(Response.java:304)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)
at java.lang.Thread.run(Thread.java:595)
2006-01-19 09:05:35,885 [TP-Processor3] WARN
org.apache.jk.common.ChannelSocket - processCallbacks status 2 .
Maybe anyone out there has an idea or knowlegde according to this problem.
Help is greatly appreciated!
