Hi Guys, I hate to pick your brains on this as the customer should know how to do this, but they tasked me to find out ;(
Is there any API or other method they can code that will give them an indication of nearing this threshold? I know it's a crappy solution, but I thought I'd ask anyway :) AFAIK, from the OS side there's really nothing I can think of that would help. Thanks for all your help, it's greatly appreciated by myself and the entire team, Danté On 08/05/2011 12:14 PM, Mark Thomas wrote: > On 05/08/2011 17:10, Dante Bell wrote: >> This is probably a really dumb question, but say they implement >> load-balanced Tomcat on 2 nodes for example. Would that then allow for >> greater than 20 STMs for the servlets? > It will allow them up to 20 concurrent requests per STM Servlet per > Tomcat instance. What that means for actual users is difficult to judge > since a single request may be routed through multiple servlets > (including the same servlet several times). It will certainly increase > capacity. From what to what is hard impossible to tell without knowing > the application code. > > Mark > >> On 08/05/2011 12:00 PM, Mark Thomas wrote: >>> On 05/08/2011 16:56, Dante Bell wrote: >>>> Thanks! >>>> >>>> Like I said, I'm an OS/HW guy, never looked at java b4! >>>> >>>> They are saying that the load test has 20 'connections' so I'm guessing >>>> that's the 20 STMs. >>>> >>>> Now, is this a fixable thing within the Java stack? Or is it an >>>> application limitation? >>> It looks to be hard-coded within Tomcat. I don't see a way to change >>> that limit without building Tomcat from source. >>> >>> The other option is re-write the STM Servlet(s) as non-STM. >>> >>> Mark >>> >>>> Danté >>>> >>>> On 08/05/2011 11:12 AM, Mark Thomas wrote: >>>>> On 05/08/2011 15:34, Dante Bell wrote: >>>>>> Hi, >>>>>> >>>>>> I'm running out of ideas on what to try for this customer. Their load >>>>>> tests show that Tomcat is getting to a point where it no longer services >>>>>> requests. >>>>> Let me guess. It is fine for low loads but as soon as the load goes >>>>> above a certain number (maybe around 20 concurrent users?) then >>>>> everything starts going wrong? >>>>> >>>>>> Thread dumps show most threads in a wait state, but some are >>>>>> runnable. I'm suspecting GC is the issue, >>>>> On what basis? The thread dump clearly shows a different problem. >>>>> >>>>>> but in analyzing the logs it >>>>>> shows the longest pause was 15 seconds, but that only happened once, and >>>>>> most of the stats look OK to me. >>>>> That would suggest that it isn't GC then wouldn't it. >>>>> >>>>>> Details can be found on my blog: http://wp.me/plPvN-ai >>>>> All the information needed to diagnose this issue is in the thead dump. >>>>> If you take a look at the first thread that is blocked: >>>>> "TP-Processor40745" daemon prio=3 tid=0x03831400 nid=0xa888 in >>>>> Object.wait() [0x2cd2f000..0x2cd2f870] >>>>> java.lang.Thread.State: WAITING (on object monitor) >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.lang.Object.wait(Object.java:485) >>>>> at >>>>> org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:854) >>>>> - locked <0x63a26708> (a java.util.Stack) >>>>> at >>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129) >>>>> at >>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >>>>> >>>>> And Bingo! we have found the problem. >>>>> >>>>> Now that stack trace might not ring alarm bells with someone unfamiliar >>>>> with the Tomcat code base but if Tomcat is unresponsive, understanding >>>>> why *any* thread is blocked would be a good place to start. If you look >>>>> at line 854 and the surrounding code for StandardWrapper you will see >>>>> that this is part of the Servlet allocation process. You should then >>>>> realise that line 854 is part of Servlet allocation for Servlets that >>>>> implement the SingleThreadModel (and now the alarm bells should be >>>>> ringing loud and clear). >>>>> >>>>> Tomcat allocates a maximum of 20 instances of any STM servlet. Your >>>>> client has an application that uses STM and requires many more than 20 >>>>> concurrent instances. Hence most requests are sat waiting for a STM >>>>> instance to be released. >>>>> >>>>> Since that means there must be 20 instances of an STM Servlet already >>>>> allocated, it is simple enough to grep the thread dump to find out which >>>>> one. The winner is: >>>>> com.motorola.nsm.common.ui.servlet.ValidateServlet >>>>> >>>>> Mark >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
