Yep, AMQ-6275 does look like it might be the fix for this particular issue. If the issue occurs again after upgrading to 5.13.4 or 5.14.0 it can be looked at more closely. I said earlier that it could be tricky because there could be things such as destination interceptors, various optional plugins configured, etc involved that don't always make it obvious the exact code path the other thread might be taking to cause the deadlock.
On Wed, Aug 17, 2016 at 9:38 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > If so, an upgrade to 5.13.4 or 5.14.0 would be all that's needed. > > On Aug 17, 2016 7:08 AM, "Timothy Bish" <tabish...@gmail.com> wrote: > > > > > Could be this. > > https://issues.apache.org/jira/browse/AMQ-6275 > > > > On 08/17/2016 09:03 AM, Tim Bain wrote: > > > >> The stack trace tells us one of the locks, so there should be a pretty > >> limited number of other locks that the thread in question acquires, so > the > >> search space shouldn't be too large. It's not guaranteed that someone > >> will > >> be able to figure it out, but I think the odds are good. > >> > >> On Aug 17, 2016 6:55 AM, "Christopher Shannon" < > >> christopher.l.shan...@gmail.com> wrote: > >> > >> This will be kind of tricky to figure out because your jstack only > printed > >>> out the stack trace for one of the threads. Without the other thread > is > >>> will be hard to know exactly what caused the deadlock. > >>> > >>> On Wed, Aug 17, 2016 at 8:48 AM, Tim Bain <tb...@alumni.duke.edu> > wrote: > >>> > >>> Please submit a bug in JIRA; it should be possible for someone to > figure > >>>> out the root cause based on the one stack trace you did get. > >>>> > >>>> Tim > >>>> > >>>> On Aug 16, 2016 9:45 PM, "RuralHunter" <ruralhun...@gmail.com> wrote: > >>>> > >>>> I got a dead lock in 5.13.3. But the jstack failed to get the stack of > >>>>> > >>>> the > >>>> > >>>>> locking threads: > >>>>> Deadlock Detection: > >>>>> > >>>>> Found one Java-level deadlock: > >>>>> ============================= > >>>>> > >>>>> "ActiveMQ NIO Worker 93810": > >>>>> waiting for ownable synchronizer 0x00007fdafc09e548, (a > >>>>> java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync), > >>>>> which is held by "ActiveMQ NIO Worker 93322" > >>>>> "ActiveMQ NIO Worker 93322": > >>>>> waiting for ownable synchronizer 0x00007fdab28706b0, (a > >>>>> java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync), > >>>>> which is held by "ActiveMQ NIO Worker 93810" > >>>>> > >>>>> Found a total of 1 deadlock. > >>>>> > >>>>> Thread 26319: (state = BLOCKED) > >>>>> Error occurred during stack walking: > >>>>> > >>>>> > >>>>> Thread 26318: (state = BLOCKED) > >>>>> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; > >>>>> > >>>> information > >>>> > >>>>> may be imprecise) > >>>>> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) > >>>>> > >>>> @bci=14, > >>>> > >>>>> line=186 (Compiled frame) > >>>>> - > >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer. > >>>>> parkAndCheckInterrupt() > >>>>> @bci=1, line=834 (Compiled frame) > >>>>> - > >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer. > >>>>> > >>>> doAcquireShared(int) > >>>> > >>>>> @bci=83, line=964 (Compiled frame) > >>>>> - java.util.concurrent.locks.AbstractQueuedSynchronizer. > >>>>> acquireShared(int) > >>>>> @bci=10, line=1282 (Compiled frame) > >>>>> - java.util.concurrent.locks.ReentrantReadWriteLock$ > ReadLock.lock() > >>>>> @bci=5, > >>>>> line=731 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.broker.region.AbstractRegion.lookup( > >>>>> org.apache.activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ActiveMQDestination, boolean) @bci=10, > >>>>> line=541 > >>>>> (Interpreted frame) > >>>>> - > >>>>> org.apache.activemq.broker.region.AbstractRegion. > >>>>> > >>>> addConsumer(org.apache. > >>> > >>>> activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ConsumerInfo) @bci=65, line=342 > (Compiled > >>>>> frame) > >>>>> - > >>>>> org.apache.activemq.broker.region.RegionBroker. > addConsumer(org.apache. > >>>>> activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ConsumerInfo) @bci=41, line=427 > (Compiled > >>>>> frame) > >>>>> - > >>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker. > >>>>> > >>>> addConsumer(org.apache. > >>>> > >>>>> activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ConsumerInfo) @bci=3, line=240 (Compiled > >>>>> frame) > >>>>> - > >>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(org. > >>>>> apache.activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ConsumerInfo) @bci=6, line=103 (Compiled > >>>>> frame) > >>>>> - > >>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(org. > >>>>> apache.activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ConsumerInfo) @bci=6, line=103 (Compiled > >>>>> frame) > >>>>> - > >>>>> org.apache.activemq.broker.MutableBrokerFilter. > addConsumer(org.apache. > >>>>> activemq.broker.ConnectionContext, > >>>>> org.apache.activemq.command.ConsumerInfo) @bci=6, line=108 (Compiled > >>>>> frame) > >>>>> - > >>>>> org.apache.activemq.broker.TransportConnection. > >>>>> processAddConsumer(org.apache.activemq.command.ConsumerInfo) > >>>>> @bci=207, line=671 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.command.ConsumerInfo.visit(org.apache. > >>>>> activemq.state.CommandVisitor) > >>>>> @bci=2, line=351 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.broker.TransportConnection.service( > >>>>> org.apache.activemq.command.Command) > >>>>> @bci=41, line=338 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.broker.TransportConnection$1. > >>>>> onCommand(java.lang.Object) > >>>>> @bci=70, line=188 (Compiled frame) > >>>>> - org.apache.activemq.transport.MutexTransport.onCommand(java. > >>>>> lang.Object) > >>>>> @bci=52, line=50 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.transport.WireFormatNegotiator. > >>>>> onCommand(java.lang.Object) > >>>>> @bci=29, line=125 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.transport.AbstractInactivityMonitor. > >>>>> onCommand(java.lang.Object) > >>>>> @bci=156, line=300 (Compiled frame) > >>>>> - > >>>>> org.apache.activemq.transport.TransportSupport.doConsume( > >>>>> > >>>> java.lang.Object) > >>>> > >>>>> @bci=16, line=83 (Compiled frame) > >>>>> - org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=7, > >>>>> > >>>> line=233 > >>>> > >>>>> (Compiled frame) > >>>>> - org.apache.activemq.transport.tcp.TcpTransport.run() @bci=47, > >>>>> > >>>> line=215 > >>>> > >>>>> (Compiled frame) > >>>>> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) > >>>>> > >>>>> The jstack output seems is not complete(Don't know why). I grep 93810 > >>>>> > >>>> or > >>> > >>>> 93322 in the stack log but couldn't find anything except the dead lock > >>>>> > >>>> info > >>>> > >>>>> above. > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> View this message in context: http://activemq.2283324.n4. > >>>>> nabble.com/Dead-lock-in-5-13-3-tp4715561.html > >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >>>>> > >>>>> > > > > -- > > Tim Bish > > twitter: @tabish121 > > blog: http://timbish.blogspot.com/ > > > > >
