For the latest release of our product we switched from Apache DerbyDB to
KahaDB for ActiveMQ persistence. We also upgraded ActiveMQ to 5.6.0. Now two
of the three production installations, that have been updated to our latest
version, are experiencing issues with ActiveMQ persistence causing ActiveMQ
messaging to be flaky at best.
The issues are currently causing our server to be sometimes unable to send
messages to clients, since we are getting IndexOutOfBoundsExceptions and
IllegalStateExceptions trying to access inactive pages in KahaDB on our logs
(detailed stacktraces below).
The product is a client-server product with fat clients. The server and
clients each have their own internal (as in running in the same JVM) brokers
with persistence running as a network of brokers. The reason the setup is
this is that these systems are running on ships, which might have really bad
internal networks and we are using ActiveMQ to take care that no data is
lost even if a client is first disconnected from a server and then shutdown.
The reason why we switched from DerbyDB was that we were having performance
issues and thought the KahaDB being purpose built would function better.
Since the installations are on ships, we have no direct access to them, so
we cannot debug them. We are trying to get our hands on the KahaDB files in
order to see, if we can reproduce the issues in our development environment.
In the mean time I would like to ask, if anyone has bumped in to similar
issues with ActiveMQ (5.6.0) and KahaDB? And if yes, were you able to solve
them in any other way than clearing the perisistence store and hoping for
the best?
-Pauli
An example of an IndexOutOfBoundsException (there are quite many of these in
the logs):
Caused by: javax.jms.JMSException: Index: 3, Size: 2
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74)
~[activemq-pool-5.6.0.jar:5.6.0]
at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59)
~[activemq-pool-5.6.0.jar:5.6.0]
at <product>.messaging.Messenger.send(Messenger.java:168)
~[<product>-impl-2012.3.1.jar:2012.3.1]
... 18 common frames omitted
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 2
at java.util.ArrayList.rangeCheck(Unknown Source) ~[na:1.7.0_07]
at java.util.ArrayList.get(Unknown Source) ~[na:1.7.0_07]
at
org.apache.activemq.kaha.impl.index.hash.HashPage.getHashEntry(HashPage.java:181)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashPageInfo.getHashEntry(HashPageInfo.java:78)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:247)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.broker.region.Topic.send(Topic.java:417)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
~[activemq-core-5.6.0.jar:5.6.0]
... 3 common frames omitted
An example of the IllegalStateException (many of these as well):
Caused by: javax.jms.JMSException: Trying to access an inactive page: 969728
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74)
~[activemq-pool-5.6.0.jar:5.6.0]
at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59)
~[activemq-pool-5.6.0.jar:5.6.0]
at <product>.messaging.Messenger.send(Messenger.java:168)
~[<product>-impl-2012.3.1.jar:2012.3.1]
... 10 common frames omitted
Caused by: java.lang.IllegalStateException: Trying to access an inactive
page: 969728
at
org.apache.activemq.kaha.impl.index.hash.HashIndex.lookupPage(HashIndex.java:347)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashPageInfo.begin(HashPageInfo.java:96)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashBin.getRetrievePage(HashBin.java:269)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:244)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.broker.region.Topic.send(Topic.java:417)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
~[activemq-core-5.6.0.jar:5.6.0]
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
~[activemq-core-5.6.0.jar:5.6.0]
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
~[activemq-core-5.6.0.jar:5.6.0]
... 3 common frames omitted
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Issues-with-KahaDB-persistence-in-ActiveMQ-5-6-0-IndexOutOfBoundsException-IllegalStateException-tp4658031.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
