We are using ActiveMQ, using Network of Brokers in Hub-Spoke topology.
Active MQ is deployed in Karaf Container
Software Stack Used
Active MQ Version 5.10.0
Container - Karaf 3.0.3
Camel 2.14.1
Hub is having 2 brokers- Master Active MQ and Slave Active MQ, configured
using shared file system (GFS) and Kaha DB Data Store. Operate as failover.
Spokes or Site connect to Hub using network connectors configuration below-
<networkConnectors>
<networkConnector
name="T:broker1->broker2"
uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616)
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
</networkConnector>
<networkConnector
name="Q:broker1->broker2"
uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616)
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
</networkConnector>
</networkConnectors>
*Issue*
*
We have topic on HUB AMQ broker called Shop.Canonical.WorkOrder.Notify, and
the producers are at the spoke level. The producers write to the local topic
Shop.Canonical.WorkOrder.Notify (This is the same topic AMQ creates when
using network of brokers).
We have Durable Topic Subscribers on Central (HUB ) broker, camel client
routes subscribed to this topic.
Apparently due to network connections lost or force shutdowns of servers
hosting these karaf containers at spoke level, we are ending up having
offline durable subscribers.
It looks like we had network outage and network failover tried to stop the
network bridge and at the same time active mq tried to start another
connection causing deadlock. I see the following exception from log file.
Also the details of the threads causing deadlock towards the end of this
message.
*
javax.jms.InvalidClientIDException: Broker: taxxx-amq-broker - Client:
T:broker1->broker2_central-amq-broker_inbound_taxxx-amq-broker already
connected from vm://taxxx-amq-broker#4548
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:246)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:89)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.security.JaasAuthenticationBroker.addConnection(JaasAuthenticationBroker.java:87)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:764)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:453)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.network.DemandForwardingBridgeSupport.doStartLocalAndRemoteBridges(DemandForwardingBridgeSupport.java:420)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:105)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:335)[83:org.apache.activemq.activemq-osgi:5.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_75]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_75]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_75]
*Above scenario also creates a dead lock condition - 3 threads *
*Thread 1 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-17676*
State: WAITING on
java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by:
ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
Total blocked: 5 Total waited: 10
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:970)
org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:206)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
- locked java.util.concurrent.atomic.AtomicBoolean@1e5f8fb
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:935)
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
* Thread 2 -
Name: ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480*
State: WAITING on
java.util.concurrent.locks.ReentrantLock$NonfairSync@1c83e84 owned by:
ActiveMQ BrokerService[taxxx-amq-broker] Task-17676
Total blocked: 0 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:849)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
- locked java.util.concurrent.atomic.AtomicBoolean@12796af
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:905)
org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1178)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:763)
- locked java.net.URI@15e6d6
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:614)
org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.failover.FailoverTransport$3.onCommand(FailoverTransport.java:208)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
java.lang.Thread.run(Thread.java:745)
*Thread 3 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-18436*
State: WAITING on
java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by:
ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
Total blocked: 0 Total waited: 44
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:288)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Durable-Subscribers-going-offline-Deadlock-Network-Of-Brokers-tp4716206.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
