On 05/08/2011 16:12, Mark Thomas wrote: > On 05/08/2011 15:34, Dante Bell wrote: >> Hi, >> >> I'm running out of ideas on what to try for this customer. Their load >> tests show that Tomcat is getting to a point where it no longer services >> requests. > > Let me guess. It is fine for low loads but as soon as the load goes > above a certain number (maybe around 20 concurrent users?) then > everything starts going wrong? > >> Thread dumps show most threads in a wait state, but some are >> runnable. I'm suspecting GC is the issue, > > On what basis? The thread dump clearly shows a different problem. > >> but in analyzing the logs it >> shows the longest pause was 15 seconds, but that only happened once, and >> most of the stats look OK to me. > > That would suggest that it isn't GC then wouldn't it. > >> Details can be found on my blog: http://wp.me/plPvN-ai > > All the information needed to diagnose this issue is in the thead dump. > If you take a look at the first thread that is blocked: > "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in > Object.wait() [0x2cd2f000..0x2cd2f870] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854) > - locked <0x63a26708> (a java.util.Stack) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > And Bingo! we have found the problem. > > Now that stack trace might not ring alarm bells with someone unfamiliar > with the Tomcat code base but if Tomcat is unresponsive, understanding > why *any* thread is blocked would be a good place to start. If you look > at line 854 and the surrounding code for StandardWrapper you will see > that this is part of the Servlet allocation process. You should then > realise that line 854 is part of Servlet allocation for Servlets that > implement the SingleThreadModel (and now the alarm bells should be > ringing loud and clear). > > Tomcat allocates a maximum of 20 instances of any STM servlet. Your > client has an application that uses STM and requires many more than 20 > concurrent instances. Hence most requests are sat waiting for a STM > instance to be released. > > Since that means there must be 20 instances of an STM Servlet already > allocated, it is simple enough to grep the thread dump to find out which > one. The winner is: > com.motorola.nsm.common.ui.servlet.ValidateServlet
"By Jove, Holmes!", exclaimed Dr Watson. p --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
