On 06/09/2013 13:28, Daniel Mikusa wrote: > On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.dere...@thingworx.com> wrote: > >> I’m trying to determine if there are any known issues using an Async servlet >> with the default BIO connector in Tomcat 7.0.40 on windows server 2008. >> >> PROBLEM: >> We have a situation where we’re occasionally getting a deadlock (according >> to Java Melody). When this happens, it’s just a matter of time until all >> the threads in the connector’s internal pool are exhausted because >> browser-based clients continue to make calls on this async servlet – >> effectively polling for work. >> >> QUESTION: >> I’m in the process of going through the stacks and code, but I’m not well >> versed in the async servlet code (which is mostly from the Vysper project), >> so I’m trying to figure out if the underlying AsyncContext.dispatch usage is >> a problem [at least when used with BIO]. >> >> Given the 3 stacks below, I’d greatly appreciate it if someone who >> understands the Async servlet plumbing might be able to take a quick look >> and see if anything stands out that could be the cause. If you need any >> additional information, please let me know. >> >> Thanks, >> Bob > > Is there any chance you could get a full thread dump while the problem is > occurring? The thread dump below doesn't have any information about what > locks are being held by the blocked threads. I think that might help to see > why each thread is blocking.
+1. What I would say is that there is no async code involved in the http-bio-192.168.230.133-443-exec-1329 thread which makes me suspect the vysper code. The only locks I'd expect Tomcat to be holding are on the socket to ensure that only one thread processes a connection at a time and I'm pretty such there is no opportunity for deadlock there. A closer look at BoshBackedSessionContext is probably in order. Mark > > Dan > > >> >> DEADLOCKED STACK INFORMATION >> >> Warning, the following threads are deadlocked : >> http-bio-192.168.230.133-443-exec-1329, >> http-bio-192.168.230.133-443-exec-685, http-bio-192.168.230.133-443-exec-687 >> >> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED >> >> org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:556) >> >> org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:365) >> >> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:848) >> org.apache.coyote.Request.action(Request.java:344) >> >> org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:222) >> >> org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:181) >> >> org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:175) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0(BoshBackedSessionContext.java:251) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(BoshHandler.java:237) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshHandler.java:228) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:192) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78) >> >> org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480) >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) >> >> org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154) >> >> org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108) >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) >> >> org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81) >> >> com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:647) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:728) >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) >> >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> >> com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279) >> >> com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150) >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >> >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) >> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >> >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) >> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >> >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) >> >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008) >> >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >> >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> java.lang.Thread.run(Unknown Source) >> >> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED >> >> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.requestExpired(BoshBackedSessionContext.java:479) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$000(BoshBackedSessionContext.java:49) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTimeout(BoshBackedSessionContext.java:669) >> >> org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:45) >> >> org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151) >> >> org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:293) >> >> org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp11Processor.java:1568) >> >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:583) >> >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> java.lang.Thread.run(Unknown Source) >> >> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED >> >> org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler.java:160) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanza(BoshSaxContentHandler.java:90) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endElement(BoshSaxContentHandler.java:78) >> >> org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480) >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) >> >> org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154) >> >> org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108) >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) >> >> org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultNonBlockingXMLReader.java:185) >> >> org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder.java:81) >> >> com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost(ThingworxBoshServlet.java:176) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:647) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:728) >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) >> >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> >> com.thingworx.security.filter.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:279) >> >> com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:150) >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >> >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) >> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >> >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) >> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >> >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) >> >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008) >> >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >> >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> java.lang.Thread.run(Unknown Source) >> >> >> Bob DeRemer >> Senior Director, Architecture and Development >> >> >> http://www.thingworx.com >> Skype: bob.deremer.thingworx >> O: 610.594.6200 x812 >> M: 717.881.3986 >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
