On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ranom...@gmail.com> wrote:
> Hi, > > Sometimes, in moments of high traffic for our patterns (170 req/sec), we > have a lot of threads like that: > > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition > [0x000000004ddcb000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000609a53198> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) > at > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. > > Thus, Tomcat hangs (all available threads stay in that state) and we have > to restart the service. > > Server version: Apache Tomcat/7.0.63 > Server built: Jun 30 2015 08:08:33 UTC > Server number: 7.0.63.0 > OS Name: Linux > OS Version: 2.6.18-194.17.1.el5 > Architecture: amd64 > JVM Version: 1.7.0_80-b15 > JVM Vendor: Oracle Corporation > > We are using tomcat-native (APR connector) and Apache (2.2.3) as load > balancer with mod_jk. > > > server.xml > > <?xml version='1.0' encoding='utf-8'?> > <Server port="8005" shutdown="SHUTDOWN"> > <Listener className="org.apache.catalina.core.AprLifecycleListener" > SSLEngine="off" /> > <Listener className="org.apache.catalina.core.JasperListener" /> > <Listener > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> > <Listener > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> > <Listener > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> > <GlobalNamingResources> > <Resource name="UserDatabase" auth="Container" > type="org.apache.catalina.UserDatabase" > description="User database that can be updated and saved" > factory="org.apache.catalina.users.MemoryUserDatabaseFactory" > pathname="conf/tomcat-users.xml" /> > </GlobalNamingResources> > > <Service name="Catalina"> > > <Connector port="8080" protocol="HTTP/1.1" > connectionTimeout="20000" > redirectPort="8443" /> > <Connector port="8009" enableLookups="false" connectionTimeout="60000" > protocol="AJP/1.3" redirectPort="8443" /> > > <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04"> > > <Realm className="org.apache.catalina.realm.LockOutRealm"> > <Realm className="org.apache.catalina.realm.UserDatabaseRealm" > resourceName="UserDatabase" digest="SHA"/> > </Realm> > > <Host name="localhost" appBase="webapps" > unpackWARs="true" autoDeploy="true"> > > <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" > threshold="60" interruptThreadThreshold="120" /> > </Host> > </Engine> > </Service> > </Server> > > > Any suggestions? > This could be an issue with HTTPD talking to Tomcat. - When the problem occurs, check the access logs for both and see if the request is making it from HTTPD to Tomcat. - Also when the problem occurs, try hitting Tomcat directly while the problem is occurring. See if you can access your app when you bypass HTTPD. - Post your HTTPD config - Make sure timeouts are configured appropriately - http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html Do you have a firewall in between HTTPD & Tomcat? Is there anything that could be breaking the connection? Dan
