Hello, We were able to reproduce this issue with just ActiveMQ (minus our application). It seems that when the consumers throw a exceptions frequently, something goes wrong trying to rollback the transaction and the LevelDB. It seems that a LevelDB thread is locking it up.
Attached is a test which simulates the behavior. For us it failed after 50000 messages but the number may vary on the machine. We suspect the following thread to be the culprit as it continuously seems to be blocked. Thread 43139: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=20, line=186 (Compiled frame [deoptimized]) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=7, line=834 (Compiled frame [deoptimized]) - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=89, line=994 (Compiled frame [deoptimized]) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=47, line=1303 (Interpreted frame) - java.util.concurrent.CountDownLatch.await() @bci=11, line=236 (Interpreted frame) - *org.apache.activemq.leveldb.LevelDBStore.rollback*(org.apache.activemq.command.TransactionId) @bci=152, line=498 (Interpreted frame) - *org.apache.activemq.transaction.LocalTransaction.rollback*() @bci=111, line=94 (Interpreted frame) - org.apache.activemq.broker.TransactionBroker.rollbackTransaction(org.apache.activemq.broker.ConnectionContext, org.apache.activemq.command.TransactionId) @bci=15, line=258 (Interpreted frame) - org.apache.activemq.broker.BrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext, org.apache.activemq.command.TransactionId) @bci=12, line=142 (Interpreted frame) - org.apache.activemq.broker.MutableBrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext, org.apache.activemq.command.TransactionId) @bci=12, line=147 (Interpreted frame) - org.apache.activemq.broker.TransportConnection.processRollbackTransaction(org.apache.activemq.command.TransactionInfo) @bci=44, line=442 (Interpreted frame) - org.apache.activemq.command.TransactionInfo.visit(org.apache.activemq.state.CommandVisitor) @bci=146, line=104 (Compiled frame [deoptimized]) - org.apache.activemq.broker.TransportConnection.service(org.apache.activemq.command.Command) @bci=52, line=292 (Compiled frame [deoptimized]) - org.apache.activemq.broker.TransportConnection$1.onCommand(java.lang.Object) @bci=95, line=149 (Compiled frame [deoptimized]) - org.apache.activemq.transport.MutexTransport.onCommand(java.lang.Object) @bci=58, line=50 (Compiled frame [deoptimized]) - org.apache.activemq.transport.WireFormatNegotiator.onCommand(java.lang.Object) @bci=35, line=113 (Compiled frame [deoptimized]) - org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(java.lang.Object) @bci=162, line=270 (Compiled frame [deoptimized]) - org.apache.activemq.transport.TransportSupport.doConsume(java.lang.Object) @bci=27, line=83 (Compiled frame [deoptimized]) - org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=13, line=214 (Compiled frame [deoptimized]) - org.apache.activemq.transport.tcp.TcpTransport.run() @bci=70, line=196 (Compiled frame [deoptimized]) - java.lang.Thread.run() @bci=17, line=724 (Interpreted frame) Thanks. Saket On Thu, May 29, 2014 at 5:03 PM, Saket Jha <skate...@gmail.com> wrote: > I looked at the thread dump but it does not point to something obvious. > Attached are the thread dumps. > > I am trying to simulate this on a pure vanilla ActiveMQ instance with some > random data. > > > On Tue, May 27, 2014 at 5:08 PM, ceposta [via ActiveMQ] < > ml-node+s2283324n4681454...@n4.nabble.com> wrote: > >> Take thread dumps while you attach consumers to see what's going on. >> Any way to recreate it? >> >> >> On Fri, May 23, 2014 at 9:07 AM, skate056 <[hidden email] >> <http://user/SendEmail.jtp?type=node&node=4681454&i=0>> wrote: >> >> > Hello all, >> > >> > We have about 145517 messages on the queue but after dequeueing about >> 1510 >> > messages the dequeueing stops. ActiveMQ is using about 28% of memory >> and >> > the web ui responds. But the messages are not getting dequeued. >> > >> > After restarting ActiveMQ it resumes dequeueing but then again >> processes >> > about ~1500 messages and then hangs again. >> > >> > We are running 5.9.0. Any suggestions what the issue could be? >> > >> > Thanks, >> > Saket >> > >> > >> > >> > -- >> > View this message in context: >> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366.html >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > >> >> >> >> -- >> *Christian Posta* >> http://www.christianposta.com/blog >> twitter: @christianposta >> http://www.christianposta.com/blog >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the >> discussion below: >> >> http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4681454.html >> To unsubscribe from ActiveMQ message dequeuing hangs, click here >> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4681366&code=c2thdGUwNTZAZ21haWwuY29tfDQ2ODEzNjZ8MzM0NzIxODA=> >> . >> NAML >> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > activeMqLoadTest.tar.gz (24K) <http://activemq.2283324.n4.nabble.com/attachment/4681579/0/activeMqLoadTest.tar.gz> -- View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4681579.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
