> -----Original Message----- > From: Mark Thomas [mailto:ma...@apache.org] > Sent: Friday, September 06, 2013 9:25 AM > To: users@tomcat.apache.org > Subject: Re: (7.0.40) Seeing deadlock followed by maxThread usage using a > Async Servlet AsyncContext.dispatch with BIO > > On 06/09/2013 13:28, Daniel Mikusa wrote: > > On Sep 5, 2013, at 3:30 PM, Bob DeRemer <bob.dere...@thingworx.com> > wrote: > > > >> I'm trying to determine if there are any known issues using an Async > >> servlet > with the default BIO connector in Tomcat 7.0.40 on windows server 2008. > >> > >> PROBLEM: > >> We have a situation where we're occasionally getting a deadlock (according > to Java Melody). When this happens, it's just a matter of time until all the > threads in the connector's internal pool are exhausted because browser-based > clients continue to make calls on this async servlet - effectively polling for > work. > >> > >> QUESTION: > >> I'm in the process of going through the stacks and code, but I'm not well > versed in the async servlet code (which is mostly from the Vysper project), so > I'm trying to figure out if the underlying AsyncContext.dispatch usage is a > problem [at least when used with BIO]. > >> > >> Given the 3 stacks below, I'd greatly appreciate it if someone who > understands the Async servlet plumbing might be able to take a quick look and > see if anything stands out that could be the cause. If you need any > additional > information, please let me know. > >> > >> Thanks, > >> Bob > > > > Is there any chance you could get a full thread dump while the problem is > occurring? The thread dump below doesn't have any information about what > locks are being held by the blocked threads. I think that might help to see > why > each thread is blocking. > > +1. > > What I would say is that there is no async code involved in the > http-bio-192.168.230.133-443-exec-1329 thread which makes me suspect the > vysper code. > > The only locks I'd expect Tomcat to be holding are on the socket to ensure > that > only one thread processes a connection at a time and I'm pretty such there is > no opportunity for deadlock there. > > A closer look at BoshBackedSessionContext is probably in order. > > Mark >
Thanks for the suggestions! I have already found the problem, which (as you suspected), is in the BoshBackedSessionContext. There is a potential deadlock between the BoshBackedSessionContext and the underlying JIoEndpoint. It's happening because the Vypser code has 2 methods using "synchronized": - 1 thd is doing an AsyncContext.dispatch from a synchronized method of BoshBackedSessionContext which hangs waiting for the JIoEndpoint synchronized socket wrapper in processSocketAsync - the other thd is the async timeout in JIoEndpoint firing back into the BoshBackedSessionContext; he acquired the socket wrapper sync and is attempting to a different synchronized method on the same BoshBackedSessionContext So, I'm going to remove the synchronized usage from the BoshBackedSessionContext and replace with a Lock/Condition using a tryLock with timeout and 1 retry. I've used this approach before and it's typically a lot more performant and provides better control over situations like this. Thx, bob > > > > Dan > > > > > >> > >> DEADLOCKED STACK INFORMATION > >> > >> Warning, the following threads are deadlocked : > >> http-bio-192.168.230.133-443-exec-1329, > >> http-bio-192.168.230.133-443-exec-685, > >> http-bio-192.168.230.133-443-exec-687 > >> > >> "http-bio-192.168.230.133-443-exec-685" daemon prio=5 BLOCKED > >> > org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:55 > 6) > >> > org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java: > 365) > >> > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce > ssor.java:848) > >> org.apache.coyote.Request.action(Request.java:344) > >> > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:22 > 2) > >> > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:18 > 1) > >> > org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:17 > 5) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.write0( > BoshBackedSessionContext.java:251) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.terminateSession(Bos > hHandler.java:237) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.processSession(BoshH > andler.java:228) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler. > java:192) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanz > a(BoshSaxContentHandler.java:90) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endEleme > nt(BoshSaxContentHandler.java:78) > >> > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480) > >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) > >> > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154) > >> > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108) > >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) > >> > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultN > onBlockingXMLReader.java:185) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder > .java:81) > >> > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost( > ThingworxBoshServlet.java:176) > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt > erChain.java:305) > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > java:210) > >> > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) > >> > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt > erChain.java:243) > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > java:210) > >> > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Authentica > tionFilter.java:279) > >> > com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.j > ava:150) > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt > erChain.java:243) > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > java:210) > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv > e.java:222) > >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j > ava:123) > >> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase > .java:581) > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:17 > 1) > >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > >> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav > a:118) > >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > >> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Proc > essor.java:1008) > >> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra > ctProtocol.java:589) > >> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:3 > 12) > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> java.lang.Thread.run(Unknown Source) > >> > >> "http-bio-192.168.230.133-443-exec-687" daemon prio=5 BLOCKED > >> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.request > Expired(BoshBackedSessionContext.java:479) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext.access$ > 000(BoshBackedSessionContext.java:49) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshBackedSessionContext$1.onTi > meout(BoshBackedSessionContext.java:669) > >> > org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListener > Wrapper.java:45) > >> > org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:151 > ) > >> > org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.ja > va:293) > >> > org.apache.coyote.http11.AbstractHttp11Processor.asyncDispatch(AbstractHttp > 11Processor.java:1568) > >> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra > ctProtocol.java:583) > >> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:3 > 12) > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> java.lang.Thread.run(Unknown Source) > >> > >> "http-bio-192.168.230.133-443-exec-1329" daemon prio=5 BLOCKED > >> > org.apache.vysper.xmpp.extension.xep0124.BoshHandler.process(BoshHandler. > java:160) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.emitStanz > a(BoshSaxContentHandler.java:90) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshSaxContentHandler.endEleme > nt(BoshSaxContentHandler.java:78) > >> > org.apache.vysper.xml.sax.impl.XMLParser.endElement(XMLParser.java:480) > >> org.apache.vysper.xml.sax.impl.XMLParser.token(XMLParser.java:227) > >> > org.apache.vysper.xml.sax.impl.XMLTokenizer.emit(XMLTokenizer.java:154) > >> > org.apache.vysper.xml.sax.impl.XMLTokenizer.parse(XMLTokenizer.java:108) > >> org.apache.vysper.xml.sax.impl.XMLParser.parse(XMLParser.java:121) > >> > org.apache.vysper.xml.sax.impl.DefaultNonBlockingXMLReader.parse(DefaultN > onBlockingXMLReader.java:185) > >> > org.apache.vysper.xmpp.extension.xep0124.BoshDecoder.decode(BoshDecoder > .java:81) > >> > com.thingworx.xmpp.vysper.extensions.xep0124.ThingworxBoshServlet.doPost( > ThingworxBoshServlet.java:176) > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt > erChain.java:305) > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > java:210) > >> > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) > >> > >> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt > erChain.java:243) > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > java:210) > >> > com.thingworx.security.filter.AuthenticationFilter.propagateRequest(Authentica > tionFilter.java:279) > >> > com.thingworx.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.j > ava:150) > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilt > erChain.java:243) > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. > java:210) > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv > e.java:222) > >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j > ava:123) > >> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase > .java:581) > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:17 > 1) > >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > >> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav > a:118) > >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > >> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Proc > essor.java:1008) > >> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra > ctProtocol.java:589) > >> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:3 > 10) > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> java.lang.Thread.run(Unknown Source) > >> > >> > >> Bob DeRemer > >> Senior Director, Architecture and Development > >> > >> > >> http://www.thingworx.com > >> Skype: bob.deremer.thingworx > >> O: 610.594.6200 x812 > >> M: 717.881.3986 > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
