Hello.
Hope anyone there can help me with this issue. I'm dealing with this for the
past 2 weeks and cannot solve it completely nor locate the root cause.
I have an environment with 2 boxes load balanced with mod_proxy_ajp. Each box
have 1 Apache HTTP and 1 Tomcat. To illustrate this:
Box A:
. Tomcat A (6.0.29, x64)
. Apache A (2.2.22)
Box B:
. Tomcat B (6.0.29, x64)
. Apache B (2.2.22)
Apache A and B have identical setup, each one balances to both Tomcat A and B.
Also, there's a third application in Box C with another application being
proxied by Apache A and Apache B.
Box C:
. Tomcat C (6.0.20, x86)
All boxes runs Windows 2008 R2 x64, have the Windows Firewall started but not
enabled.
Box A and Box B are part of a Windows NLB Domain, but all references in the
proxy configuration are made using the hosts private IP addresses. Users use
the NLB IP address to connecto to Apache.
Apache proxy is configured like:
== begin httpd.conf ==
Timeout 600
LimitRequestFieldSize 20480
ProxyIOBufferSize 21504
ProxyRequests Off
ProxyPreserveHost On
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
<Proxy balancer://wlb>
BalancerMember ajp://10.11.102.224:9109 route=wt1 loadfactor=50 max=85
ttl=120 retry=5 connectiontimeout=5000ms ping=5000ms
BalancerMember ajp://127.0.0.1:9109 route=wt2 loadfactor=50 max=85 ttl=120
retry=5 connectiontimeout=5000ms ping=5000ms
</Proxy>
ProxyPass /app1 balancer://wlb/app1 stickysession=JSESSIONID nofailover=On
ProxyPass /app2 ajp://10.11.102.219:8009/app2
== end httpd.conf ==
Tomcat A and Tomcat B AJP connector is configured like:
<Connector port="9109" protocol="AJP/1.3" redirectPort="8443"
packetSize="22528" maxThreads="200" connectionTimeout="120000"/>
Tomcat C is configured like:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
The load is not high and there are a few users using the applications. This is
the production environment, I could not trace an operation to reproduce this
behavior in a controlled environment.
The "Box C" application "app2" shown above runs fine, not a single error
message.
The timeout parameters for app1 (Tomcat A and B) were configured in an attempt
to solve the problem shown here. When using the default (no connectiontimeout,
no ping, no ttl, no retry) other communications failures were happening.
Now, the problem: the AJP communication between Apache A/B and Tomcat A/B is
bad. The following can be seen in the Apache logs:
== begin apache log ==
[Tue May 29 14:43:59 2012] [error] (OS 10060)A connection attempt failed
because the connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond. :
ajp_ilink_receive() can't receive header
[Tue May 29 14:43:59 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue May 29 14:43:59 2012] [error] (70007)The timeout specified has expired:
proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed
[Tue May 29 14:44:42 2012] [error] [client 10.45.7.78] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 14:45:08 2012] [error] [client 10.45.6.233] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 14:45:17 2012] [error] [client 10.45.6.100] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 14:45:39 2012] [error] (OS 10060)A connection attempt failed
because the connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond. :
ajp_ilink_receive() can't receive header
[Tue May 29 14:45:39 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue May 29 14:45:39 2012] [error] (70007)The timeout specified has expired:
proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed
[Tue May 29 14:45:39 2012] [error] proxy: BALANCER: (balancer://wlb). All
workers are in error state for route (wt1)
[Tue May 29 14:54:40 2012] [error] (OS 10060)A connection attempt failed
because the connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond. :
ajp_ilink_receive() can't receive header
[Tue May 29 14:54:40 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue May 29 14:54:40 2012] [error] (70007)The timeout specified has expired:
proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed
[Tue May 29 15:05:15 2012] [error] [client 200.251.3.133] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 15:07:15 2012] [error] [client 10.45.6.54] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 15:17:17 2012] [error] [client 10.45.8.22] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 15:29:43 2012] [error] (OS 10060)A connection attempt failed
because the connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond. :
ajp_ilink_receive() can't receive header
[Tue May 29 15:29:43 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue May 29 15:29:43 2012] [error] (70007)The timeout specified has expired:
proxy: dialog to 10.11.102.223:9109 (10.11.102.223) failed
[Tue May 29 15:29:44 2012] [error] proxy: BALANCER: (balancer://wlb). All
workers are in error state for route (wt2)
[Tue May 29 15:29:58 2012] [error] [client 10.45.7.225] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 15:48:03 2012] [error] [client 10.45.7.230] File does not exist:
E:/Apache/htdocs/favicon.ico
[Tue May 29 15:48:08 2012] [error] (OS 10060)A connection attempt failed
because the connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond. :
ajp_ilink_receive() can't receive header
[Tue May 29 15:48:08 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue May 29 15:48:08 2012] [error] (70007)The timeout specified has expired:
proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed
[Tue May 29 15:48:09 2012] [error] proxy: BALANCER: (balancer://wlb). All
workers are in error state for route (wt1)
[Tue May 29 15:48:09 2012] [error] proxy: BALANCER: (balancer://wlb). All
workers are in error state for route (wt1)
== end apache log ==
In the Tomcat log we have:
== begin tomcat log ==
29/05/2012 11:11:39 org.apache.jk.common.ChannelSocket processConnection
AVISO: processCallbacks status 2
29/05/2012 11:21:01 org.apache.jk.common.ChannelSocket processConnection
AVISO: processCallbacks status 2
29/05/2012 15:34:09 org.apache.jk.common.ChannelSocket processConnection
AVISO: processCallbacks status 2
29/05/2012 15:39:15 org.apache.jk.core.MsgContext action
AVISO: Unable to send headers
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:537)
at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
at org.apache.jk.core.MsgContext.action(MsgContext.java:266)
at org.apache.coyote.Response.action(Response.java:183)
at org.apache.coyote.Response.sendHeaders(Response.java:379)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)
at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:273)
at
org.apache.catalina.connector.Response.finishResponse(Response.java:493)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:322)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
29/05/2012 15:39:15 org.apache.jk.common.ChannelSocket processConnection
AVISO: processCallbacks status 2
29/05/2012 15:56:58 org.apache.jk.common.ChannelSocket processConnection
GRAVE: Error, processing connection
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:537)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:339)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
29/05/2012 15:56:58 org.apache.jk.common.ChannelSocket processConnection
[...]
== end tomcat log ==
Not sure if they are really related as the error messages from Apache and
Tomcat are not shown at the same time. These log files are from the same host.
Before configuring the Apache "ping" option and the Tomcat "connectionTimeout",
different errors where being shown. Also, I have upgraded from Apache 2.2.21 in
the hope that application timeouts didn't make the node in the "error state"
anymore.
This problem may have a bigger impact when we have more users using the
application.
Hope someone can bring some light to this issue.
Today I'm going to alter the configuration and add the "disablereuse=On" to try
again to isolate the cause.
Thanks!
Roney
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
