Hi everyone
We are using ActiveMQ 4.1.1 over JRE 1.5.0_06 and Solaris 10 as JMS
provider. Our broker is embedded inside the application, and we use it,
not only to hold destinations for incoming external messages, but to
hold destinations for internal communication (producers and consumers
executing within the same JVM, in different threads).
Under heavy load, sometimes the broker hangs on a socketWrite0 call:
"Session(recv,TaskManagerQueue)#56" prio=10 tid=0x00b4a6d8 nid=0x6a
runnable [0xeb9fe000..0xeb9ffaa8]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
at java.io.DataOutputStream.flush(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:145)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:80)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:93)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:392)
- locked <0x2abab308> (a java.lang.Object)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)
- locked <0x2aba90f0> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1155)
at
org.apache.activemq.TransactionContext.begin(TransactionContext.java:201)
at
org.apache.activemq.ActiveMQSession.doStartTransaction(ActiveMQSession.java:1564)
at
org.apache.activemq.ActiveMQMessageConsumer.ackLater(ActiveMQMessageConsumer.java:699)
at
org.apache.activemq.ActiveMQMessageConsumer.beforeMessageIsConsumed(ActiveMQMessageConsumer.java:651)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:487)
...
MutexTransport is locking 0x2aba90f0, and thus making other threads to stop:
"Session(recv,TaskManagerQueue)#49" prio=10 tid=0x00b177f0 nid=0x63
waiting for monitor entry [0xec0ff000..0xec0ffa28]
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:46)
- waiting to lock <0x2aba90f0> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1155)
at
org.apache.activemq.TransactionContext.begin(TransactionContext.java:201)
at
org.apache.activemq.ActiveMQSession.doStartTransaction(ActiveMQSession.java:1564)
at
org.apache.activemq.ActiveMQMessageConsumer.ackLater(ActiveMQMessageConsumer.java:699)
at
org.apache.activemq.ActiveMQMessageConsumer.beforeMessageIsConsumed(ActiveMQMessageConsumer.java:651)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:487)
...
socketWrite0 would never stop, and, eventually, all consuming threads
will block waiting for the same object. This thread is consuming from a
local destination (within the embedded broker).
Don't know if it's related, but another thread is stuck in the same call
(IP changed, but it's a non-localhost IP):
"ActiveMQ Transport: tcp:///XXX.XXX.XXX.XXX:32469" daemon prio=10
tid=0x01a0be60 nid=0x25e23 runnable [0x1217e000..0x1217fa28]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
at java.io.DataOutputStream.flush(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:145)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:80)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:93)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)
- locked <0x2abbd550> (a java.lang.Object)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1138)
at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:805)
at
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:770)
at
org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:360)
at
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:83)
at
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:51)
at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:417)
at org.apache.activemq.broker.region.Topic.send(Topic.java:264)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:292)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:385)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:193)
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:277)
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:242)
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:237)
at
org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(AdvisoryBroker.java:190)
at
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221)
at
org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(MutableBrokerFilter.java:229)
at
org.apache.activemq.broker.TransportConnection.processRemoveDestination(TransportConnection.java:530)
- locked <0x2e5d8a38> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
at
org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:123)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:124)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
at java.lang.Thread.run(Unknown Source)
Any ideas about this? Searching about this I found it could be a TCP/IP
related problem: the socketWrite0 could be locked because the socket's
send buffer is full, and the other end is no longer reading (whichever
the reason: dead reader, slow reader). Should failover take care of this?
Any info would be highly appreciated.
Best regards
Eduardo Corral
