Interesting. What's the CPU usage like at the time? If it's high, you might want to look at gc logging to see if you've got a lot of GC happening.
CPU is low. You'll probably also want to configure the access logs to log the amount of time the request took to process. I'll analyze this option. Thanks. 2015-07-16 13:44 GMT-03:00 Daniel Mikusa <dmik...@pivotal.io>: > On Thu, Jul 16, 2015 at 12:04 PM, Robert Anderson <ranom...@gmail.com> > wrote: > > > Thanks, Dan. > > > > This is not a problem. The thread is in the pool waiting for work. It's > > what you'd normally see on a server that's not handling any requests. > > > > It's awkward. Everything is going normal and suddenly, many threads start > > waiting on the same monitor. After restart Tomcat, everything works > without > > even make any changes in Apache (service or config files). > > > > The scenario you're describing can happen and it doesn't necessarily mean > there's a problem with Tomcat. If you're seeing lots of threads waiting > for work, it just means that you've got lots of free workers. The server > could still be really busy if you got a couple threads running and working > like mad. > > A full thread dump would be helpful, to see what else is happening at the > time. You can also look at multiple thread dumps 10 - 15 seconds apart to > see what the threads are doing over time (i.e. is thread #X at the same > place across all the threads?) > > > > - When the problem occurs, check the access logs for both and see if the > > request is making it from HTTPD to Tomcat. > > > > OK. I'll check > > > > You'll probably also want to configure the access logs to log the amount of > time the request took to process. That'll give you some understanding of > how much time was spent at each hop. In Tomcat that's done by adding "%D" > to the access log pattern. > > > > https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve/Attributes > > HTTPD is the same, but it logs in micro seconds instead of milliseconds. > > http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats > > > > > > - Also when the problem occurs, try hitting Tomcat directly while the > > problem is occurring. See if you can access your app when you bypass > > HTTPD. > > > > I've already tried this. Tomcat becomes very very very slow when it > > happens. > > > > Interesting. What's the CPU usage like at the time? If it's high, you > might want to look at gc logging to see if you've got a lot of GC > happening. > > Dan > > > > > > > - Post your HTTPD config > > > > worker.properties: > > > > worker.list=balancer > > > > worker.tomsrv01.type=ajp13 > > worker.tomsrv01.host=x.y.z.1 > > worker.tomsrv01.port=8009 > > worker.tomsrv01.lbfactor=1 > > worker.tomsrv01.connect_timeout=10000 > > worker.tomsrv01.prepost_timeout=10000 > > worker.tomsrv01.socket_timeout=1800 > > worker.tomsrv01.connection_pool_timeout=60 > > > > worker.tomsrv02.type=ajp13 > > worker.tomsrv02.host=x.y.z.2 > > worker.tomsrv02.port=8009 > > worker.tomsrv02.lbfactor=1 > > worker.tomsrv02.connect_timeout=10000 > > worker.tomsrv02.prepost_timeout=10000 > > worker.tomsrv02.socket_timeout=1800 > > worker.tomsrv02.connection_pool_timeout=60 > > > > worker.tomsrv04.type=ajp13 > > worker.tomsrv04.host=x.y.z.3 > > worker.tomsrv04.port=8009 > > worker.tomsrv04.lbfactor=1 > > worker.tomsrv04.connect_timeout=10000 > > worker.tomsrv04.prepost_timeout=10000 > > worker.tomsrv04.socket_timeout=1800 > > worker.tomsrv04.connection_pool_timeout=60 > > > > worker.balancer.type=lb > > worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04 > > worker.balancer.sticky_session=1 > > > > - Make sure timeouts are configured appropriately - > > > > Ok > > > > > > Do you have a firewall in between HTTPD & Tomcat? Is there anything that > > could be breaking the connection? > > > > No. > > > > > > Best regards. > > > > > > 2015-07-16 12:44 GMT-03:00 Daniel Mikusa <dmik...@pivotal.io>: > > > > > On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ranom...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > > > > > Sometimes, in moments of high traffic for our patterns (170 req/sec), > > we > > > > have a lot of threads like that: > > > > > > > > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" > daemon > > > > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition > > > > [0x000000004ddcb000] > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > at sun.misc.Unsafe.park(Native Method) > > > > - parking to wait for <0x0000000609a53198> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > > > at > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > > > at > > > > > > > > > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > > > > at > > > > > > > > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) > > > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) > > > > at > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > > > > at > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > > > > at > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > at > > > > > > > > > > > > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > This is not a problem. The thread is in the pool waiting for work. > It's > > > what you'd normally see on a server that's not handling any requests. > > > > > > > > > > > > > > Thus, Tomcat hangs (all available threads stay in that state) and we > > have > > > > to restart the service. > > > > > > > > Server version: Apache Tomcat/7.0.63 > > > > Server built: Jun 30 2015 08:08:33 UTC > > > > Server number: 7.0.63.0 > > > > OS Name: Linux > > > > OS Version: 2.6.18-194.17.1.el5 > > > > Architecture: amd64 > > > > JVM Version: 1.7.0_80-b15 > > > > JVM Vendor: Oracle Corporation > > > > > > > > We are using tomcat-native (APR connector) and Apache (2.2.3) as load > > > > balancer with mod_jk. > > > > > > > > > > > > server.xml > > > > > > > > <?xml version='1.0' encoding='utf-8'?> > > > > <Server port="8005" shutdown="SHUTDOWN"> > > > > <Listener className="org.apache.catalina.core.AprLifecycleListener" > > > > SSLEngine="off" /> > > > > <Listener className="org.apache.catalina.core.JasperListener" /> > > > > <Listener > > > > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" > /> > > > > <Listener > > > > > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" > > > /> > > > > <Listener > > > > > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" > > /> > > > > <GlobalNamingResources> > > > > <Resource name="UserDatabase" auth="Container" > > > > type="org.apache.catalina.UserDatabase" > > > > description="User database that can be updated and > saved" > > > > > > > factory="org.apache.catalina.users.MemoryUserDatabaseFactory" > > > > pathname="conf/tomcat-users.xml" /> > > > > </GlobalNamingResources> > > > > > > > > <Service name="Catalina"> > > > > > > > > <Connector port="8080" protocol="HTTP/1.1" > > > > connectionTimeout="20000" > > > > redirectPort="8443" /> > > > > <Connector port="8009" enableLookups="false" > > > connectionTimeout="60000" > > > > protocol="AJP/1.3" redirectPort="8443" /> > > > > > > > > <Engine name="Catalina" defaultHost="localhost" > > jvmRoute="tomsrv04"> > > > > > > > > <Realm className="org.apache.catalina.realm.LockOutRealm"> > > > > <Realm > className="org.apache.catalina.realm.UserDatabaseRealm" > > > > resourceName="UserDatabase" digest="SHA"/> > > > > </Realm> > > > > > > > > <Host name="localhost" appBase="webapps" > > > > unpackWARs="true" autoDeploy="true"> > > > > > > > > <Valve > className="org.apache.catalina.valves.StuckThreadDetectionValve" > > > > threshold="60" interruptThreadThreshold="120" /> > > > > </Host> > > > > </Engine> > > > > </Service> > > > > </Server> > > > > > > > > > > > > Any suggestions? > > > > > > > > > > This could be an issue with HTTPD talking to Tomcat. > > > - When the problem occurs, check the access logs for both and see if > > the > > > request is making it from HTTPD to Tomcat. > > > - Also when the problem occurs, try hitting Tomcat directly while > the > > > problem is occurring. See if you can access your app when you bypass > > > HTTPD. > > > - Post your HTTPD config > > > - Make sure timeouts are configured appropriately - > > > http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html > > > > > > Do you have a firewall in between HTTPD & Tomcat? Is there anything > that > > > could be breaking the connection? > > > > > > Dan > > > > > >
