On 05/10/2015 16:00, Yogesh Patel wrote: > Other 199 process are also for /solr430/update?wt=xml&version=2.2 HTTP/1.1 > only
That is surprising. That could be a Solr issue or a Tomcat issue. Best to take three thread dumps each 10 to 30 seconds apart and then take a look to see what is holding up those threads. Best guess is a deadlock or a database issue. (You still have the potential for thread starvation that will come back to bite you later after you fix this issue unless you fix it) Mark > > On 5 October 2015 at 20:18, Mark Thomas <ma...@apache.org> wrote: > >> On 05/10/2015 12:58, Yogesh Patel wrote: >>> Hi Mark Thomas, >>> >>> in image it shows Tomcat Manager screen: >>> Under "ajp-apr-10003" Section in tomcat manager: >>> Max threads: 200 Current thread count:200 Current thread busy :200 Keeped >>> alive socket count :0 >>> >>> *Stage Time BSent BRecv Client VHost >>> Request* >>> S 2884466 ms 0 KB 184063 KB machineip host >> POST >>> /solr430/update?wt=xml&version=2.2 HTTP/1.1 >> >> And the other 199 processors? >> >> Mark >> >> >>> >>> .......... >>> ............ >>> >>> >>> On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.pa...@highq.com> >> wrote: >>> >>>> Hi , let me try sending image using attachment , if still its not >> viewable >>>> then i will find another way. >>>> >>>> On 5 October 2015 at 16:06, Mark Thomas <ma...@apache.org> wrote: >>>> >>>>> On 05/10/2015 11:28, Yogesh Patel wrote: >>>>>> Hi Thomas , >>>>>> Please see this image ...have a look at Time column >>>>> >>>>> The list strips images. If you really want us to look at the image (not >>>>> that I think it will be remotely relevant) upload it somewhere and post >>>>> the URL (make sure it is publicly accessible and doe snot require any >>>>> form of registration to view it). >>>>> >>>>> Mark >>>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 5 October 2015 at 14:50, Mark Thomas <ma...@apache.org >>>>>> <mailto:ma...@apache.org>> wrote: >>>>>> >>>>>> On 05/10/2015 10:09, Yogesh Patel wrote: >>>>>> > Hi Thomas, >>>>>> > >>>>>> > Connector configuration is like : >>>>>> > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" >> /> >>>>>> >>>>>> That means you are using the BIO AJP connector. >>>>>> >>>>>> You don't have a problem with long running requests. >>>>>> >>>>>> You have a problem with thread starvation. >>>>>> >>>>>> AJP uses persistent connections. >>>>>> >>>>>> BIO requires one thread per connection, regardless of whether or >> nor >>>>>> that connection is processing a request or waiting for the next >> one. >>>>>> >>>>>> httpd will (eventually, assuming mostly default config) create one >>>>> AJP >>>>>> connection for each httpd thread. If you have more httpd threads >>>>> than >>>>>> Tomcat threads you will eventually reach the point where httpd >> can't >>>>>> create a Tomcat thread it requires and at that point it will >> appear >>>>>> to hang. >>>>>> >>>>>> There are several possible fixes: >>>>>> a) disable connection re-use in httpd for AJP connections >>>>>> b) use the NIO AJP connector in Tomcat >>>>>> c) increase maxThreads in Tomcat so it is > max threads in httpd >>>>>> >>>>>> For more explanation read this: >>>>>> >>>>> >> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf >>>>>> < >>>>> >> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf >>>>>> >>>>>> >>>>>> from slide 29 to 33 >>>>>> >>>>>> Mark >>>>>> >>>>>> >>>>>> >>>>>> > >>>>>> > On 5 October 2015 at 14:17, Mark Thomas <ma...@apache.org >>>>>> <mailto:ma...@apache.org>> wrote: >>>>>> > >>>>>> >> On 05/10/2015 09:07, Yogesh Patel wrote: >>>>>> >>> Thanks Mark Thomas , >>>>>> >>> >>>>>> >>> Our application is access by Apache TO Tomcat using AJP >>>>>> Connector. >>>>>> >> >>>>>> >> OK. That answers one of the questions I asked. However, you >>>>> haven't >>>>>> >> provided the connector configuration. >>>>>> >> >>>>>> >>> Problem : >>>>>> >>> Our application was mostly hanged,after looking at tomcat >>>>>> manager it >>>>>> >>> shown there are so many long running threads shown. >>>>>> >> >>>>>> >> The Tomcat Manager app does not show long running threads. It >>>>>> does show >>>>>> >> you how many requests are busy but again, a busy thread is not >>>>>> >> necessarily processing a request. >>>>>> >> >>>>>> >>> We want to recognize why so many threads are running since >>>>>> long time. >>>>>> >> >>>>>> >> Threads are always "running". The key question is are those >>>>> threads >>>>>> >> 'idle' or 'busy'. An idle thread is waiting to be allocated >> work. >>>>>> A busy >>>>>> >> thread has been allocated work but the manager application >> can't >>>>>> >> distinguish if that work is 'wait for a request to process' or >>>>>> >> 'processing a request'. >>>>>> >> >>>>>> >>> We want to detect such thread and want to kill these >> stucked >>>>>> thread. >>>>>> >> >>>>>> >> You continue to make the (increasingly unlikely) assumption >> that >>>>> 200 >>>>>> >> busy threads mean you have 200 threads processing long running >>>>>> requests. >>>>>> >> Had you provided the connector configuration I asked for (by >> that >>>>>> I mean >>>>>> >> the <Connector ... /> elements in server.xml) then I'd be in a >>>>> better >>>>>> >> position to tell you what is wrong. >>>>>> >> >>>>>> >> If you do have 200 long running requests then the >>>>>> >> StuckThreadDetectionValve is how you detect them. That fact >> that >>>>> that >>>>>> >> valve is not reporting any long running requests should be a >> big >>>>> clue >>>>>> >> that your assumptions about what is going on are not correct. >>>>>> >> >>>>>> >> If, and it is a big if, you did have 200 concurrent long >> running >>>>>> >> requests then there is no guaranteed way to stop them. If your >>>>>> >> application checks for interrupts (most don't) then the >>>>>> >> StuckThreadDetectionValve can interrupt them which should >>>>>> terminate the >>>>>> >> processing of that request but the time it would take to code >> the >>>>>> >> application to handle that properly would normally be better >>>>> spent >>>>>> >> fixing the root cause of the long running request. >>>>>> >> >>>>>> >> Mark >>>>>> >> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> On 5 October 2015 at 13:11, Mark Thomas <ma...@apache.org >>>>>> <mailto:ma...@apache.org>> wrote: >>>>>> >>> >>>>>> >>>> On 05/10/2015 07:54, Yogesh Patel wrote: >>>>>> >>>>> We are facing issues with long running thread in >>>>>> tomcat . we >>>>>> >> are >>>>>> >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy >> threads >>>>>> : 200, >>>>>> >>>>> application gets stucked due to these long running threads. >>>>>> >>>> >>>>>> >>>> What makes you think you have issues with long running >> threads? >>>>>> A busy >>>>>> >>>> thread isn't necessarily processing a request. Configuration >>>>> errors >>>>>> >>>> leading to thread starvation are a more typical cause of the >>>>>> symptom you >>>>>> >>>> describe rather than long running threads in the application. >>>>>> >>>> >>>>>> >>>>> We implemented StuckThreadDetectionValve in >>>>>> server.xml( <Valve >>>>>> >>>>> >>>>>> >>>>> >>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve" >>>>>> >>>>> >>>>>> >>>>> threshold="60" />), but it could not help out. >>>>>> >>>> >>>>>> >>>> Why not? Again, this suggests that long running threads >> aren't >>>>> the >>>>>> >> problem. >>>>>> >>>> >>>>>> >>>>> So we implemented custom StuckThreadDetectionValve by >>>>>> extending >>>>>> >>>>> StuckThreadDetectionValve from >>>>>> >>>>> catalina, it only goes to "constructor","initInternal",and >> in >>>>>> >>>> "invoke"(when >>>>>> >>>>> action fires), it does not go to function >>>>>> "getStuckThreadNames()" even >>>>>> >>>>> after threshold time.How to achieve the same.Is there any >> way >>>>>> to detect >>>>>> >>>>> stucked thread and kill them? >>>>>> >>>> >>>>>> >>>> First of all your invoke() method fails to call >> super.invoke() >>>>>> so the >>>>>> >>>> Valve is never going to do anything. >>>>>> >>>> >>>>>> >>>> Secondly, if the original valve didn't work adding a bunch of >>>>>> >>>> System.out.println() lines isn't going to magically make it >>>>> work. >>>>>> >>>> >>>>>> >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat >>>>>> code so it >>>>>> >>>> is no surprise that you never see a call to that method. That >>>>>> method is >>>>>> >>>> intended for use via JMX. >>>>>> >>>> >>>>>> >>>> I suggest you start again and tell us more about the problem >>>>>> you are >>>>>> >>>> trying to solve (lack of response) and your configuration. In >>>>>> particular >>>>>> >>>> we need to know your connector configuration and how users >>>>>> access the >>>>>> >>>> application. Do they connect directly to Tomcat or do they go >>>>> via a >>>>>> >>>> reverse proxy? >>>>>> >>>> >>>>>> >>>> Mark >>>>>> >>>> >>>>>> >>>> >>>>>> >>>>> >>>>>> >>>>> My custom Valve is like following : >>>>>> >>>>> >>>>>> >>>>> public class StuckThreadDetection extends >>>>>> StuckThreadDetectionValve >>>>>> >>>>> { >>>>>> >>>>> StuckThreadDetection stuckThreadDetection; >>>>>> >>>>> >>>>>> >>>>> public StuckThreadDetection() >>>>>> >>>>> { >>>>>> >>>>> System.out.println("in StuckThreadDetection constructor"); >>>>>> >>>>> >>>>>> >>>>> } >>>>>> >>>>> >>>>>> >>>>> public void invoke(Request request, Response response) >> throws >>>>>> >>>> IOException, >>>>>> >>>>> ServletException >>>>>> >>>>> { >>>>>> >>>>> System.out.println("in invoke..."); >>>>>> >>>>> >>>>>> >>>>> getNext().invoke(request, response); >>>>>> >>>>> } >>>>>> >>>>> >>>>>> >>>>> @Override >>>>>> >>>>> protected void initInternal() throws LifecycleException >>>>>> >>>>> { >>>>>> >>>>> System.out.println("In initInternal"); >>>>>> >>>>> super.initInternal(); >>>>>> >>>>> } >>>>>> >>>>> >>>>>> >>>>> @Override >>>>>> >>>>> public String[] getStuckThreadNames() >>>>>> >>>>> { >>>>>> >>>>> System.out.println("in getStuckThreadNames..."); >>>>>> >>>>> String[] listStuckedThread = this.getStuckThreadNames(); >>>>>> >>>>> >>>>>> >>>>> for (String threadName : listStuckedThread) >>>>>> >>>>> { >>>>>> >>>>> System.out.println(threadName); >>>>>> >>>>> } >>>>>> >>>>> >>>>>> >>>>> return listStuckedThread; >>>>>> >>>>> >>>>>> >>>>> } >>>>>> >>>>> >>>>>> >>>>> @Override >>>>>> >>>>> public String getInfo() >>>>>> >>>>> { >>>>>> >>>>> System.out.println("In getInfo"); >>>>>> >>>>> return super.getInfo(); >>>>>> >>>>> } >>>>>> >>>>> } >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> <mailto:users-unsubscr...@tomcat.apache.org> >>>>>> >>>> For additional commands, e-mail: >> users-h...@tomcat.apache.org >>>>>> <mailto:users-h...@tomcat.apache.org> >>>>>> >>>> >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>> --------------------------------------------------------------------- >>>>>> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> <mailto:users-unsubscr...@tomcat.apache.org> >>>>>> >> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> <mailto:users-h...@tomcat.apache.org> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> <mailto:users-unsubscr...@tomcat.apache.org> >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> <mailto:users-h...@tomcat.apache.org> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> /Thanks & Regards,/ >>>>>> / / >>>>>> / Yogesh Patel/ >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Thanks & Regards,* >>>> >>>> * Yogesh Patel* >>>> >>> >>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
