On 05/08/2011 16:56, Dante Bell wrote: > Thanks! > > Like I said, I'm an OS/HW guy, never looked at java b4! > > They are saying that the load test has 20 'connections' so I'm guessing > that's the 20 STMs. > > Now, is this a fixable thing within the Java stack? Or is it an > application limitation?
It looks to be hard-coded within Tomcat. I don't see a way to change that limit without building Tomcat from source. The other option is re-write the STM Servlet(s) as non-STM. Mark > > Danté > > On 08/05/2011 11:12 AM, Mark Thomas wrote: >> On 05/08/2011 15:34, Dante Bell wrote: >>> Hi, >>> >>> I'm running out of ideas on what to try for this customer. Their load >>> tests show that Tomcat is getting to a point where it no longer services >>> requests. >> Let me guess. It is fine for low loads but as soon as the load goes >> above a certain number (maybe around 20 concurrent users?) then >> everything starts going wrong? >> >>> Thread dumps show most threads in a wait state, but some are >>> runnable. I'm suspecting GC is the issue, >> On what basis? The thread dump clearly shows a different problem. >> >>> but in analyzing the logs it >>> shows the longest pause was 15 seconds, but that only happened once, and >>> most of the stats look OK to me. >> That would suggest that it isn't GC then wouldn't it. >> >>> Details can be found on my blog: http://wp.me/plPvN-ai >> All the information needed to diagnose this issue is in the thead dump. >> If you take a look at the first thread that is blocked: >> "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in >> Object.wait() [0x2cd2f000..0x2cd2f870] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> at java.lang.Object.wait(Object.java:485) >> at >> org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854) >> - locked <0x63a26708> (a java.util.Stack) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> >> And Bingo! we have found the problem. >> >> Now that stack trace might not ring alarm bells with someone unfamiliar >> with the Tomcat code base but if Tomcat is unresponsive, understanding >> why *any* thread is blocked would be a good place to start. If you look >> at line 854 and the surrounding code for StandardWrapper you will see >> that this is part of the Servlet allocation process. You should then >> realise that line 854 is part of Servlet allocation for Servlets that >> implement the SingleThreadModel (and now the alarm bells should be >> ringing loud and clear). >> >> Tomcat allocates a maximum of 20 instances of any STM servlet. Your >> client has an application that uses STM and requires many more than 20 >> concurrent instances. Hence most requests are sat waiting for a STM >> instance to be released. >> >> Since that means there must be 20 instances of an STM Servlet already >> allocated, it is simple enough to grep the thread dump to find out which >> one. The winner is: >> com.motorola.nsm.common.ui.servlet.ValidateServlet >> >> Mark >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
