On 05/08/2011 15:34, Dante Bell wrote: > Hi, > > I'm running out of ideas on what to try for this customer. Their load > tests show that Tomcat is getting to a point where it no longer services > requests.
Let me guess. It is fine for low loads but as soon as the load goes above a certain number (maybe around 20 concurrent users?) then everything starts going wrong? > Thread dumps show most threads in a wait state, but some are > runnable. I'm suspecting GC is the issue, On what basis? The thread dump clearly shows a different problem. > but in analyzing the logs it > shows the longest pause was 15 seconds, but that only happened once, and > most of the stats look OK to me. That would suggest that it isn't GC then wouldn't it. > Details can be found on my blog: http://wp.me/plPvN-ai All the information needed to diagnose this issue is in the thead dump. If you take a look at the first thread that is blocked: "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in Object.wait() [0x2cd2f000..0x2cd2f870] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854) - locked <0x63a26708> (a java.util.Stack) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) And Bingo! we have found the problem. Now that stack trace might not ring alarm bells with someone unfamiliar with the Tomcat code base but if Tomcat is unresponsive, understanding why *any* thread is blocked would be a good place to start. If you look at line 854 and the surrounding code for StandardWrapper you will see that this is part of the Servlet allocation process. You should then realise that line 854 is part of Servlet allocation for Servlets that implement the SingleThreadModel (and now the alarm bells should be ringing loud and clear). Tomcat allocates a maximum of 20 instances of any STM servlet. Your client has an application that uses STM and requires many more than 20 concurrent instances. Hence most requests are sat waiting for a STM instance to be released. Since that means there must be 20 instances of an STM Servlet already allocated, it is simple enough to grep the thread dump to find out which one. The winner is: com.motorola.nsm.common.ui.servlet.ValidateServlet Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
