Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]

[Robert Anderson]

Hi,

Sometimes, in moments of high traffic for our patterns (170 req/sec), we
have a lot of threads like that:

"ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon
prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition
[0x000000004ddcb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000609a53198> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

Thus, Tomcat hangs (all available threads stay in that state) and we have
to restart the service.

Server version: Apache Tomcat/7.0.63
Server built:   Jun 30 2015 08:08:33 UTC
Server number:  7.0.63.0
OS Name:        Linux
OS Version:     2.6.18-194.17.1.el5
Architecture:   amd64
JVM Version:    1.7.0_80-b15
JVM Vendor:     Oracle Corporation

We are using tomcat-native (APR connector) and Apache (2.2.3) as load
balancer with mod_jk.


server.xml

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="off" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <Service name="Catalina">

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <Connector port="8009" enableLookups="false" connectionTimeout="60000"
protocol="AJP/1.3" redirectPort="8443" />

    <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04">

      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase" digest="SHA"/>
      </Realm>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
threshold="60" interruptThreadThreshold="120" />
      </Host>
    </Engine>
  </Service>
</Server>


Any suggestions?


Thanks in advance.


Robert