I forgot to mention that these errors are infrequent, 6 per day on average. Usually everything works fine, sometimes these errors happens and put the worker in error state. However, system load is still very low.
Thanks, Roney -----Original Message----- From: Roney Duilio Stein [mailto:roney.st...@sondait.com.br] Sent: terça-feira, 29 de maio de 2012 21:36 To: users@tomcat.apache.org Subject: AJP communication failures Hello. Hope anyone there can help me with this issue. I'm dealing with this for the past 2 weeks and cannot solve it completely nor locate the root cause. I have an environment with 2 boxes load balanced with mod_proxy_ajp. Each box have 1 Apache HTTP and 1 Tomcat. To illustrate this: Box A: . Tomcat A (6.0.29, x64) . Apache A (2.2.22) Box B: . Tomcat B (6.0.29, x64) . Apache B (2.2.22) Apache A and B have identical setup, each one balances to both Tomcat A and B. Also, there's a third application in Box C with another application being proxied by Apache A and Apache B. Box C: . Tomcat C (6.0.20, x86) All boxes runs Windows 2008 R2 x64, have the Windows Firewall started but not enabled. Box A and Box B are part of a Windows NLB Domain, but all references in the proxy configuration are made using the hosts private IP addresses. Users use the NLB IP address to connecto to Apache. Apache proxy is configured like: == begin httpd.conf == Timeout 600 LimitRequestFieldSize 20480 ProxyIOBufferSize 21504 ProxyRequests Off ProxyPreserveHost On <Proxy *> Order deny,allow Allow from all </Proxy> <Proxy balancer://wlb> BalancerMember ajp://10.11.102.224:9109 route=wt1 loadfactor=50 max=85 ttl=120 retry=5 connectiontimeout=5000ms ping=5000ms BalancerMember ajp://127.0.0.1:9109 route=wt2 loadfactor=50 max=85 ttl=120 retry=5 connectiontimeout=5000ms ping=5000ms </Proxy> ProxyPass /app1 balancer://wlb/app1 stickysession=JSESSIONID nofailover=On ProxyPass /app2 ajp://10.11.102.219:8009/app2 == end httpd.conf == Tomcat A and Tomcat B AJP connector is configured like: <Connector port="9109" protocol="AJP/1.3" redirectPort="8443" packetSize="22528" maxThreads="200" connectionTimeout="120000"/> Tomcat C is configured like: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> The load is not high and there are a few users using the applications. This is the production environment, I could not trace an operation to reproduce this behavior in a controlled environment. The "Box C" application "app2" shown above runs fine, not a single error message. The timeout parameters for app1 (Tomcat A and B) were configured in an attempt to solve the problem shown here. When using the default (no connectiontimeout, no ping, no ttl, no retry) other communications failures were happening. Now, the problem: the AJP communication between Apache A/B and Tomcat A/B is bad. The following can be seen in the Apache logs: == begin apache log == [Tue May 29 14:43:59 2012] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. : ajp_ilink_receive() can't receive header [Tue May 29 14:43:59 2012] [error] ajp_read_header: ajp_ilink_receive failed [Tue May 29 14:43:59 2012] [error] (70007)The timeout specified has expired: proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed [Tue May 29 14:44:42 2012] [error] [client 10.45.7.78] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 14:45:08 2012] [error] [client 10.45.6.233] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 14:45:17 2012] [error] [client 10.45.6.100] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 14:45:39 2012] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. : ajp_ilink_receive() can't receive header [Tue May 29 14:45:39 2012] [error] ajp_read_header: ajp_ilink_receive failed [Tue May 29 14:45:39 2012] [error] (70007)The timeout specified has expired: proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed [Tue May 29 14:45:39 2012] [error] proxy: BALANCER: (balancer://wlb). All workers are in error state for route (wt1) [Tue May 29 14:54:40 2012] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. : ajp_ilink_receive() can't receive header [Tue May 29 14:54:40 2012] [error] ajp_read_header: ajp_ilink_receive failed [Tue May 29 14:54:40 2012] [error] (70007)The timeout specified has expired: proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed [Tue May 29 15:05:15 2012] [error] [client 200.251.3.133] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 15:07:15 2012] [error] [client 10.45.6.54] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 15:17:17 2012] [error] [client 10.45.8.22] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 15:29:43 2012] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. : ajp_ilink_receive() can't receive header [Tue May 29 15:29:43 2012] [error] ajp_read_header: ajp_ilink_receive failed [Tue May 29 15:29:43 2012] [error] (70007)The timeout specified has expired: proxy: dialog to 10.11.102.223:9109 (10.11.102.223) failed [Tue May 29 15:29:44 2012] [error] proxy: BALANCER: (balancer://wlb). All workers are in error state for route (wt2) [Tue May 29 15:29:58 2012] [error] [client 10.45.7.225] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 15:48:03 2012] [error] [client 10.45.7.230] File does not exist: E:/Apache/htdocs/favicon.ico [Tue May 29 15:48:08 2012] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. : ajp_ilink_receive() can't receive header [Tue May 29 15:48:08 2012] [error] ajp_read_header: ajp_ilink_receive failed [Tue May 29 15:48:08 2012] [error] (70007)The timeout specified has expired: proxy: dialog to 127.0.0.1:9109 (127.0.0.1) failed [Tue May 29 15:48:09 2012] [error] proxy: BALANCER: (balancer://wlb). All workers are in error state for route (wt1) [Tue May 29 15:48:09 2012] [error] proxy: BALANCER: (balancer://wlb). All workers are in error state for route (wt1) == end apache log == In the Tomcat log we have: == begin tomcat log == 29/05/2012 11:11:39 org.apache.jk.common.ChannelSocket processConnection AVISO: processCallbacks status 2 29/05/2012 11:21:01 org.apache.jk.common.ChannelSocket processConnection AVISO: processCallbacks status 2 29/05/2012 15:34:09 org.apache.jk.common.ChannelSocket processConnection AVISO: processCallbacks status 2 29/05/2012 15:39:15 org.apache.jk.core.MsgContext action AVISO: Unable to send headers java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:537) at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326) at org.apache.jk.core.MsgContext.action(MsgContext.java:266) at org.apache.coyote.Response.action(Response.java:183) at org.apache.coyote.Response.sendHeaders(Response.java:379) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:273) at org.apache.catalina.connector.Response.finishResponse(Response.java:493) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:322) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:662) 29/05/2012 15:39:15 org.apache.jk.common.ChannelSocket processConnection AVISO: processCallbacks status 2 29/05/2012 15:56:58 org.apache.jk.common.ChannelSocket processConnection GRAVE: Error, processing connection java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:537) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:339) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:662) 29/05/2012 15:56:58 org.apache.jk.common.ChannelSocket processConnection [...] == end tomcat log == Not sure if they are really related as the error messages from Apache and Tomcat are not shown at the same time. These log files are from the same host. Before configuring the Apache "ping" option and the Tomcat "connectionTimeout", different errors where being shown. Also, I have upgraded from Apache 2.2.21 in the hope that application timeouts didn't make the node in the "error state" anymore. This problem may have a bigger impact when we have more users using the application. Hope someone can bring some light to this issue. Today I'm going to alter the configuration and add the "disablereuse=On" to try again to isolate the cause. Thanks! Roney --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
