----- Original Message ----- > From: Pid <p...@pidster.com> > To: Tomcat Users List <users@tomcat.apache.org> > Cc: > Sent: Friday, August 5, 2011 8:35 AM > Subject: Re: Urgent: Tomcat 6.0.20 on Solaris 10 Reaches max threads and is > non responsive > > On 05/08/2011 16:12, Mark Thomas wrote: >> On 05/08/2011 15:34, Dante Bell wrote: >>> Hi, >>> >>> I'm running out of ideas on what to try for this customer. Their > load >>> tests show that Tomcat is getting to a point where it no longer > services >>> requests. >> >> Let me guess. It is fine for low loads but as soon as the load goes >> above a certain number (maybe around 20 concurrent users?) then >> everything starts going wrong? >> >>> Thread dumps show most threads in a wait state, but some are >>> runnable. I'm suspecting GC is the issue, >> >> On what basis? The thread dump clearly shows a different problem. >> >>> but in analyzing the logs it >>> shows the longest pause was 15 seconds, but that only happened once, > and >>> most of the stats look OK to me. >> >> That would suggest that it isn't GC then wouldn't it. >> >>> Details can be found on my blog: http://wp.me/plPvN-ai >> >> All the information needed to diagnose this issue is in the thead dump. >> If you take a look at the first thread that is blocked: >> "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in >> Object.wait() [0x2cd2f000..0x2cd2f870] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> at java.lang.Object.wait(Object.java:485) >> at >> org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854) >> - locked <0x63a26708> (a java.util.Stack) >> at >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129) >> at >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> >> And Bingo! we have found the problem. >> >> Now that stack trace might not ring alarm bells with someone unfamiliar >> with the Tomcat code base but if Tomcat is unresponsive, understanding >> why *any* thread is blocked would be a good place to start. If you look >> at line 854 and the surrounding code for StandardWrapper you will see >> that this is part of the Servlet allocation process. You should then >> realise that line 854 is part of Servlet allocation for Servlets that >> implement the SingleThreadModel (and now the alarm bells should be >> ringing loud and clear). >> >> Tomcat allocates a maximum of 20 instances of any STM servlet. Your >> client has an application that uses STM and requires many more than 20 >> concurrent instances. Hence most requests are sat waiting for a STM >> instance to be released. >> >> Since that means there must be 20 instances of an STM Servlet already >> allocated, it is simple enough to grep the thread dump to find out which >> one. The winner is: >> com.motorola.nsm.common.ui.servlet.ValidateServlet > > "By Jove, Holmes!", exclaimed Dr Watson. > > > p
Nice explanation. I hauled out the Tomcat 6 code and followed along. I found the STM code and went back to the top of the class and found the limit of 20 instances. I'm guessing what you grepped on were the service methods (doPost, doGet, etc.) and noticed that there were no doGet methods, only doPost. Then, there are 2 classes in the thread dump that were in the doPost method: com.motorola.nsm.common.ui.servlet.UIJnlpServlet com.motorola.nsm.common.ui.servlet.ValidateServlet Doing: grep com.motorola.nsm.common.ui.servlet.UIJnlpServlet.doPost tomcat_stack_dump.txt | wc -l results in 28, so that's not the culprit. In other words, there are more than 20 instances, so this cannot be a STM servlet which is indicated by the blocking on line 854 of StandardWrapper.java. Doing: grep com.motorola.nsm.common.ui.servlet.ValidateServlet.doPost tomcat_stack_dump.txt | wc -l results in 20, so bingo! We're at the limit here, so the application has to wait until one of these instances is finished. If the ValidateServlet is slow or you get a lot of requests, then the application becomes unresponsive until one or more instances finishes. If the ValidateServlet has problems with certain cases, then you can lock up the entire application. This would serve as a good denial of service attack. Twenty well-crafted requests to the application, and it's essentially locked (or at least this functionality is). . . . . just my two cents. /mde/ --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
