Hi Mark Thomas, in image it shows Tomcat Manager screen: Under "ajp-apr-10003" Section in tomcat manager: Max threads: 200 Current thread count:200 Current thread busy :200 Keeped alive socket count :0
*Stage Time BSent BRecv Client VHost Request* S 2884466 ms 0 KB 184063 KB machineip host POST /solr430/update?wt=xml&version=2.2 HTTP/1.1 .......... ............ On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.pa...@highq.com> wrote: > Hi , let me try sending image using attachment , if still its not viewable > then i will find another way. > > On 5 October 2015 at 16:06, Mark Thomas <ma...@apache.org> wrote: > >> On 05/10/2015 11:28, Yogesh Patel wrote: >> > Hi Thomas , >> > Please see this image ...have a look at Time column >> >> The list strips images. If you really want us to look at the image (not >> that I think it will be remotely relevant) upload it somewhere and post >> the URL (make sure it is publicly accessible and doe snot require any >> form of registration to view it). >> >> Mark >> >> >> > >> > >> > >> > >> > On 5 October 2015 at 14:50, Mark Thomas <ma...@apache.org >> > <mailto:ma...@apache.org>> wrote: >> > >> > On 05/10/2015 10:09, Yogesh Patel wrote: >> > > Hi Thomas, >> > > >> > > Connector configuration is like : >> > > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" /> >> > >> > That means you are using the BIO AJP connector. >> > >> > You don't have a problem with long running requests. >> > >> > You have a problem with thread starvation. >> > >> > AJP uses persistent connections. >> > >> > BIO requires one thread per connection, regardless of whether or nor >> > that connection is processing a request or waiting for the next one. >> > >> > httpd will (eventually, assuming mostly default config) create one >> AJP >> > connection for each httpd thread. If you have more httpd threads >> than >> > Tomcat threads you will eventually reach the point where httpd can't >> > create a Tomcat thread it requires and at that point it will appear >> > to hang. >> > >> > There are several possible fixes: >> > a) disable connection re-use in httpd for AJP connections >> > b) use the NIO AJP connector in Tomcat >> > c) increase maxThreads in Tomcat so it is > max threads in httpd >> > >> > For more explanation read this: >> > >> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf >> > < >> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf >> > >> > >> > from slide 29 to 33 >> > >> > Mark >> > >> > >> > >> > > >> > > On 5 October 2015 at 14:17, Mark Thomas <ma...@apache.org >> > <mailto:ma...@apache.org>> wrote: >> > > >> > >> On 05/10/2015 09:07, Yogesh Patel wrote: >> > >>> Thanks Mark Thomas , >> > >>> >> > >>> Our application is access by Apache TO Tomcat using AJP >> > Connector. >> > >> >> > >> OK. That answers one of the questions I asked. However, you >> haven't >> > >> provided the connector configuration. >> > >> >> > >>> Problem : >> > >>> Our application was mostly hanged,after looking at tomcat >> > manager it >> > >>> shown there are so many long running threads shown. >> > >> >> > >> The Tomcat Manager app does not show long running threads. It >> > does show >> > >> you how many requests are busy but again, a busy thread is not >> > >> necessarily processing a request. >> > >> >> > >>> We want to recognize why so many threads are running since >> > long time. >> > >> >> > >> Threads are always "running". The key question is are those >> threads >> > >> 'idle' or 'busy'. An idle thread is waiting to be allocated work. >> > A busy >> > >> thread has been allocated work but the manager application can't >> > >> distinguish if that work is 'wait for a request to process' or >> > >> 'processing a request'. >> > >> >> > >>> We want to detect such thread and want to kill these stucked >> > thread. >> > >> >> > >> You continue to make the (increasingly unlikely) assumption that >> 200 >> > >> busy threads mean you have 200 threads processing long running >> > requests. >> > >> Had you provided the connector configuration I asked for (by that >> > I mean >> > >> the <Connector ... /> elements in server.xml) then I'd be in a >> better >> > >> position to tell you what is wrong. >> > >> >> > >> If you do have 200 long running requests then the >> > >> StuckThreadDetectionValve is how you detect them. That fact that >> that >> > >> valve is not reporting any long running requests should be a big >> clue >> > >> that your assumptions about what is going on are not correct. >> > >> >> > >> If, and it is a big if, you did have 200 concurrent long running >> > >> requests then there is no guaranteed way to stop them. If your >> > >> application checks for interrupts (most don't) then the >> > >> StuckThreadDetectionValve can interrupt them which should >> > terminate the >> > >> processing of that request but the time it would take to code the >> > >> application to handle that properly would normally be better >> spent >> > >> fixing the root cause of the long running request. >> > >> >> > >> Mark >> > >> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> On 5 October 2015 at 13:11, Mark Thomas <ma...@apache.org >> > <mailto:ma...@apache.org>> wrote: >> > >>> >> > >>>> On 05/10/2015 07:54, Yogesh Patel wrote: >> > >>>>> We are facing issues with long running thread in >> > tomcat . we >> > >> are >> > >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy threads >> > : 200, >> > >>>>> application gets stucked due to these long running threads. >> > >>>> >> > >>>> What makes you think you have issues with long running threads? >> > A busy >> > >>>> thread isn't necessarily processing a request. Configuration >> errors >> > >>>> leading to thread starvation are a more typical cause of the >> > symptom you >> > >>>> describe rather than long running threads in the application. >> > >>>> >> > >>>>> We implemented StuckThreadDetectionValve in >> > server.xml( <Valve >> > >>>>> >> > >>>>> >> > className="org.apache.catalina.valves.StuckThreadDetectionValve" >> > >>>>> >> > >>>>> threshold="60" />), but it could not help out. >> > >>>> >> > >>>> Why not? Again, this suggests that long running threads aren't >> the >> > >> problem. >> > >>>> >> > >>>>> So we implemented custom StuckThreadDetectionValve by >> > extending >> > >>>>> StuckThreadDetectionValve from >> > >>>>> catalina, it only goes to "constructor","initInternal",and in >> > >>>> "invoke"(when >> > >>>>> action fires), it does not go to function >> > "getStuckThreadNames()" even >> > >>>>> after threshold time.How to achieve the same.Is there any way >> > to detect >> > >>>>> stucked thread and kill them? >> > >>>> >> > >>>> First of all your invoke() method fails to call super.invoke() >> > so the >> > >>>> Valve is never going to do anything. >> > >>>> >> > >>>> Secondly, if the original valve didn't work adding a bunch of >> > >>>> System.out.println() lines isn't going to magically make it >> work. >> > >>>> >> > >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat >> > code so it >> > >>>> is no surprise that you never see a call to that method. That >> > method is >> > >>>> intended for use via JMX. >> > >>>> >> > >>>> I suggest you start again and tell us more about the problem >> > you are >> > >>>> trying to solve (lack of response) and your configuration. In >> > particular >> > >>>> we need to know your connector configuration and how users >> > access the >> > >>>> application. Do they connect directly to Tomcat or do they go >> via a >> > >>>> reverse proxy? >> > >>>> >> > >>>> Mark >> > >>>> >> > >>>> >> > >>>>> >> > >>>>> My custom Valve is like following : >> > >>>>> >> > >>>>> public class StuckThreadDetection extends >> > StuckThreadDetectionValve >> > >>>>> { >> > >>>>> StuckThreadDetection stuckThreadDetection; >> > >>>>> >> > >>>>> public StuckThreadDetection() >> > >>>>> { >> > >>>>> System.out.println("in StuckThreadDetection constructor"); >> > >>>>> >> > >>>>> } >> > >>>>> >> > >>>>> public void invoke(Request request, Response response) throws >> > >>>> IOException, >> > >>>>> ServletException >> > >>>>> { >> > >>>>> System.out.println("in invoke..."); >> > >>>>> >> > >>>>> getNext().invoke(request, response); >> > >>>>> } >> > >>>>> >> > >>>>> @Override >> > >>>>> protected void initInternal() throws LifecycleException >> > >>>>> { >> > >>>>> System.out.println("In initInternal"); >> > >>>>> super.initInternal(); >> > >>>>> } >> > >>>>> >> > >>>>> @Override >> > >>>>> public String[] getStuckThreadNames() >> > >>>>> { >> > >>>>> System.out.println("in getStuckThreadNames..."); >> > >>>>> String[] listStuckedThread = this.getStuckThreadNames(); >> > >>>>> >> > >>>>> for (String threadName : listStuckedThread) >> > >>>>> { >> > >>>>> System.out.println(threadName); >> > >>>>> } >> > >>>>> >> > >>>>> return listStuckedThread; >> > >>>>> >> > >>>>> } >> > >>>>> >> > >>>>> @Override >> > >>>>> public String getInfo() >> > >>>>> { >> > >>>>> System.out.println("In getInfo"); >> > >>>>> return super.getInfo(); >> > >>>>> } >> > >>>>> } >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>> >> > >>>> >> > >>>> >> > >> --------------------------------------------------------------------- >> > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> > <mailto:users-unsubscr...@tomcat.apache.org> >> > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >> > <mailto:users-h...@tomcat.apache.org> >> > >>>> >> > >>>> >> > >>> >> > >>> >> > >> >> > >> >> > >> >> --------------------------------------------------------------------- >> > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> > <mailto:users-unsubscr...@tomcat.apache.org> >> > >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > <mailto:users-h...@tomcat.apache.org> >> > >> >> > >> >> > > >> > > >> > >> > >> > >> --------------------------------------------------------------------- >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> > <mailto:users-unsubscr...@tomcat.apache.org> >> > For additional commands, e-mail: users-h...@tomcat.apache.org >> > <mailto:users-h...@tomcat.apache.org> >> > >> > >> > >> > >> > -- >> > /Thanks & Regards,/ >> > / / >> > / Yogesh Patel/ >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > > -- > *Thanks & Regards,* > > * Yogesh Patel* > -- *Thanks & Regards,* * Yogesh Patel*
