On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.dere...@thingworx.com> wrote:
> I’m trying to determine if there are any known issues using an Async servlet > with the default BIO connector in Tomcat 7.0.40 on windows server 2008. > > PROBLEM: > We have a situation where we’re occasionally getting a deadlock (according to > Java Melody). When this happens, it’s just a matter of time until all the > threads in the connector’s internal pool are exhausted because browser-based > clients continue to make calls on this async servlet – effectively polling > for work. > > QUESTION: > I’m in the process of going through the stacks and code, but I’m not well > versed in the async servlet code (which is mostly from the Vysper project), > so I’m trying to figure out if the underlying AsyncContext.dispatch usage is > a problem [at least when used with BIO]. > > Given the 3 stacks below, I’d greatly appreciate it if someone who > understands the Async servlet plumbing might be able to take a quick look and > see if anything stands out that could be the cause. If you need any > additional information, please let me know. > > Thanks, > Bob Is there any chance you could get a full thread dump while the problem is occurring? The thread dump below doesn't have any information about what locks are being held by the blocked threads. I think that might help to see why each thread is blocking. Dan > > DEADLOCKED STACK INFORMATION > > Warning, the following threads are deadlocked : > http-bio-192.168.230.133-443-exec-1329, > http-bio-192.168.230.133-443-exec-685, http-bio-192.168.230.133-443-exec-687 > > "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED > > org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:556) > > org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:365) > > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:848) > org.apache.coyote.Request.action(Request.java:344) > > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:222) > > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:181) > > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:175) > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(BoshBackedSessionContext.java:251) > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(BoshHandler.java:237) > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshHandler.java:228) > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:192) > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90) > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78) > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480) > org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154) > > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108) > org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) > > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185) > > org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81) > > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176) > javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > > net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) > > net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279) > > com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150) > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008) > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > java.lang.Thread.run(Unknown Source) > > "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requestExpired(BoshBackedSessionContext.java:479) > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$000(BoshBackedSessionContext.java:49) > > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTimeout(BoshBackedSessionContext.java:669) > > org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:45) > > org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151) > > org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:293) > > org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1568) > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:583) > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > java.lang.Thread.run(Unknown Source) > > "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED > > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:160) > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90) > > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78) > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480) > org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154) > > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108) > org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) > > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185) > > org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81) > > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176) > javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > > net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) > > net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279) > > com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150) > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008) > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > java.lang.Thread.run(Unknown Source) > > > Bob DeRemer > Senior Director, Architecture and Development > > > http://www.thingworx.com > Skype: bob.deremer.thingworx > O: 610.594.6200 x812 > M: 717.881.3986 > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
