Hi:
We have been using ActiveMQ in our production environment now for a
while without too many problems however in the last several days our
client application has frozen and requires a restart to get processing
to resume. On performing a thread dump almost all the threads are
waiting for access to the transport.
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070203b5e8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1308)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1302)
at
org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:255)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:338)
There is a thread that is attempting to use the transport, it is
currently waiting
"Service Thread Pool" prio=10 tid=0x00007f1ed8b69800 nid=0x7c53 in
Object.wait() [0x00007f1e602ef000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
- locked <0x000000070202b678> (a java.lang.Object)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1308)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1302)
at
org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:2070)
at
org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2236)
at
org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2231)
at
org.apache.activemq.ActiveMQMessageConsumer.acknowledge(ActiveMQMessageConsumer.java:1121)
- locked <0x0000000762ac6a28> (a java.util.LinkedList)
at
org.apache.activemq.ActiveMQSession.acknowledge(ActiveMQSession.java:1778)
at
org.apache.activemq.ActiveMQMessageConsumer$1.execute(ActiveMQMessageConsumer.java:606)
at
org.apache.activemq.command.ActiveMQMessage.acknowledge(ActiveMQMessage.java:102)
On the ActiveMQ server there is the following that occurred shortly
prior to the locked up state.
Transport Connection to: tcp://172.16.101.20:58460 failed:
java.net.SocketException: Connection res
et | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///172.16.101.20:58460@61616
We are using the failover transport with the following configuration
failover:(tcp://mqarms01p:61616,tcp://mqarms02p:61616)?randomize=true
Does anyone have some hints as to what we could look at to diagnose this
issue further?
Thanks!
Steve.
