On 13/09/2011 13:19, Darius D. wrote:
>
> Hi,
>
> well the problem is that we have the same problem, real strange NPEs in
> servlets.
>
> It is more pronounced when using dedicated thread executor, moving to
> threads provided by connector itself (with maxThreads="500" option using NIO
> ) seems to be lowering counts of errors. You could try commenting out the
> executor pool and using thread pool provided by connector to see if it helps
> your case.
>
>
> Atm we are seeing 3 modes of failure in our tomcat setup ( nginx reverse
> proxy -> tomcat with NIO connector )
>
>
> 1)
> SEVERE: Error finishing response
> java.lang.NullPointerException
>
> and/or
>
> SEVERE: Error processing request
> java.lang.NullPointerException
>
> in logs. ( could be some app problem, but still shady, as no stack trace is
> given, like if exception object is gone itself??? )
>
> 2) Sometimes we get 502 like Rakesh in servlets that should be fine ( result
> of NPE in real strange places )
>
> 3) And most scary ones are the following:
>
> 2011 09 13 14:18:20.568 WARN Servlet - error response: error=no action
> query: [type=groupsChangedAction&v=1315912033175] content type:null
> protocol: HTTP/1.0 length:-1 character encoding: null GET
> 2011 09 13 14:18:20.937 WARN Servlet - error response: error=no action
> query: [type=groupsChangedAction&v=1315912699092] content type:null
> protocol: HTTP/1.0 length:-1 character encoding: null GET
>
> In tomcat access log they are shown as:
> 127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] "GET
> /ourapp/servlet/Servlet?type=groupsChangedAction&v=1315912033175 HTTP/1.0"
> 200 15
> 127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] "GET
> /ourapp/servlet/Servlet?type=groupsChangedAction&v=1315912699092 HTTP/1.0"
> 200 15
>
> So as you can see, tomcat got that GET reqest with type and v params like it
> gets and processes thousands others.
>
> Servlet code looks like:
> private void doProcess(HttpServletRequest request, HttpServletResponse
> response) throws IOException {
> HttpSession session = request.getSession();
> if (session != null) {
>
> String action = request.getParameter("type");
> logger.debug("action: " + action);
>
> if (action == null) {
>
> printError(request, response,"error=no action");
> ....
>
> private void printError(HttpServletRequest request, HttpServletResponse
> response, String error) throws IOException {
> logger.warn("error response: " + error + " ip: " +
> request.getRemoteAddr() + " query: [" + request.getQueryString() + "]" + "
> content type:" + request.getContentType() + " protocol: " +
> request.getProtocol() + " length:" + request.getContentLength() + "
> character encoding: " + request.getCharacterEncoding() + " " +
> request.getMethod());
>
> response.setContentType("text/plain");
> PrintWriter out = response.getWriter();
> out.print(error);
> out.flush();
> out.close();
> }
>
> It happens several times in a day in bursts ( GC collecting something?).
>
>
> So really puzzling and random error, tomcat got proper request as shown in
> access logs (and even request.getQueryString() is valid :) ) , processed
> request from same user 5s ago, will process one more 5s later, but somehow
> request is NULL??? Now we handle it by retrying, but such random NULL is
> fatal for other servlets and services that throw NPEs and it seems that some
> tomcat code is affected too with (1).
>
>
> I'd grateful if some smart guys could look into this problem and willing to
> provide any other info.
>
> BR,
>
> Darius.
>
>
> P.S. this is kinda thread hijack, but i feel the root cause of these Y
> errors in XXXX requests are the same.
>
>
>
>
>
>
>
>
>
>
> rakesh k-3 wrote:
>>
>> Hi Darius
>>
>> Thanks for reviewing the configurations.
>>
>> in our catalina.out we had the stack trace which you had menitoned.
>>
>> please find the stack trace information which has been requested.
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:194)
>> at
>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:102)
>> at
>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>> at
>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)
>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)
>> at
>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)
>> at
>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>> Darius, if you require more information, i ca provide you the details.
>>
>> Regards
>> Rakesh
>>
>>
>> On Mon, Sep 12, 2011 at 5:49 PM, Darius D. <darius....@gmail.com> wrote:
>>
>>>
>>>
>>> rakesh k-3 wrote:
>>>>
>>>> Hi Christopher
>>>>
>>>> thanks for viewing m configuration ..as mentioned we are using http
>>>> connectors in my configuration not AJP.
>>>>
>>>> and here are my configuration details.
>>>>
>>>> Apache tomcat configuration (server.xml)
>>>>
>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>> maxThreads="150" minSpareThreads="4"/>
>>>> <Connector executor="tomcatThreadPool"
>>>> port="8080" protocol="HTTP/1.1"
>>>> connectionTimeout="20000"
>>>> redirectPort="8443" />
>>>>
>>>> Apache httpd configuration Httpd.conf.
>>>>
>>>> ProxyPass / http://localhost:8080 route=msf1
>>>>
>>>> Regards
>>>> Rakesh
>>>>
>>>>
>>>>
>>>
>>>
>>> Hi,
>>>
>>> can you check your catalina.out to see if there is something like:
>>>
>>> org.apache.coyote.http11.AbstractHttp11Processor endRequest
>>> SEVERE: Error finishing response
>>> java.lang.NullPointerException
>>>
>>> or
>>>
>>> SEVERE: Error processing request
>>> java.lang.NullPointerException
>>>
>>> without any further details / stack traces ?
>>>
>>> Darius.You should start another thread if you also have a problem. p
signature.asc
Description: OpenPGP digital signature
