On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz < ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Jamie, > > On 10/9/15 10:03 AM, Jamie Jackson wrote: > > Here's the stack trace dump: > > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067 > > > > FWIW, I've been trying to come up with a reliable test case to > > trigger the problem, but I haven't nailed it yet. I've suspected > > that it's related to file (large or slow) HTTP file uploads, and > > that's what I was running at the time, which helps to explain the > > java.net.SocketInputStream.socketRead0 traces. > > Your server looks mostly idle. It also looks like it's running both > Tomcat and Jetty in the same instance. What's going on? I'm not sure yet. I'm asking around, and I'll let you know. > > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native) > > java.lang.Thread.State: RUNNABLE at > > java.net.SocketInputStream.socketRead0(Native Method) at > > java.net.SocketInputStream.read(SocketInputStream.java:152) at > > java.net.SocketInputStream.read(SocketInputStream.java:122) at > > > org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535) > > > > > at > > org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) > > at > > > org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566) > > > > > at > > org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137) > > at > > > org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348) > > > > > at org.apache.coyote.Request.doRead(Request.java:422) > > at > > > org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) > > > > > at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390) > > at > > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304) > > > > > at > > org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106) > > at > > > com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123) > > > > > at java.io.InputStream.read(InputStream.java:179) > > at > > > org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) > > > > > at > > org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - > > locked java.io.BufferedInputStream@6e81abfc at > > java.io.FilterInputStream.read(FilterInputStream.java:107) > > Obviously, this thread is reading a file upload. Also these: > > "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE > "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE > > This thread is making an outgoing HTTP request, which could be hanging: > > > "Thread-27550" Id=27623 RUNNABLE (in native) > > java.lang.Thread.State: RUNNABLE at > > java.net.SocketInputStream.socketRead0(Native Method) at > > java.net.SocketInputStream.read(SocketInputStream.java:152) at > > java.net.SocketInputStream.read(SocketInputStream.java:122) at > > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at > > sun.security.ssl.InputRecord.read(InputRecord.java:480) at > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) - > > locked java.lang.Object@659eed19 at > > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) > > > > > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) > > - locked sun.security.ssl.AppInputStream@7c98d0ab at > > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > > > > > at > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > > at > > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > > > > > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > > at > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > > > > > at > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > > at > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > > > > > at > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > > at > > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > > > > > at > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > > at > > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > > > > > at > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > > at > > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > > > > > at > > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > > at > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > > > > > at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416) > > > > > at > > lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252) > > at > > > lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112) > > > > > at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86) > > at > > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108) > > > > > at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58) > > > Same with this one: > > "Thread-27551" Id=27624 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE > > But neither of the above threads making HTTP requests are > request-processing threads: they are independent and shouldn't block > incoming requests. > > You have 12 total request-processing threads: the ones named > "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just > waiting on a new request to come in. > > You have an AJP connector defined, but it looks like it's not being used. > > Initially, you said that you used mod_proxy from httpd. Are you > expecting to use mod_proxy_http or mod_proxy_ajp? mod_proxy_http, as far as I know, but see below... > It certainly looks > like you are using the HTTP flavor of mod_proxy since your > file-uploads are going to the HTTP connector on port 8888 instead of > the AJP connector on port 8009. If you don't need the AJP connector, > you might want to disable it and get a small amount of resources back. > Okay, thanks. When you try to make a connection during these incidents, do you get > any errors on the httpd side? Unfortunately, at the time, I thought this would be easy to reproduce, so I didn't take enough notes (including the time of the problem on my load test box). Here are some errors that I think were just created by the app timing out requests. I don't think these were associated with the failed requests, so take these entries (from httpd's error_log) with a grain of salt: https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt I'll capture this information next time I trigger a crash. > I know mod_jk will complain if it can't > make a connection or if there is a timeout... I suspect mod_proxy_http > will do the same. > > Can you post your <Connector> configuration from conf/server.xml? > Remember to remove any sensitive information that you might have in there. > I think this is what you're after: <Connector port="8888" protocol="HTTP/1.1" connectionTimeout="93000" redirectPort="8443" /> FWIW, I'm also noticing this: <Host name="127.0.0.1" appBase="webapps" unpackWARs="false" autoDeploy="false"> <!-- Access log processes all example. Documentation at: /docs/config/valve.html Note: The pattern used is equivalent to using pattern="common" --> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t "%r" %s %b" resolveHosts="false"/> <!-- visit modcfml.org for details on mod_cfml configuration options --> <Valve className="mod_cfml.core" loggingEnabled="false" waitForContext="3" maxContexts="200" timeBetweenContexts="30000" /> </Host> Also, what do your ProxyPass/ProxyPassReverse configuration directives > look like in httpd.conf? The apache config has: LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_ftp_module modules/mod_proxy_ftp.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_connect_module modules/mod_proxy_connect.so And the virtual host has: https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-virtual-host > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > > iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf > m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0 > Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia > PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv > CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz > Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg > 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j > eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc > o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf > HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc > AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls > +0KkHZDkU8C3MP3MCySA > =5Mun > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
