from the stack trace, it appears you are using the old kaha persistence adapter. you should be using <kahadb ..>
On 22 October 2012 08:33, Pauli Kaila <pauli.ka...@napa.fi> wrote: > For the latest release of our product we switched from Apache DerbyDB to > KahaDB for ActiveMQ persistence. We also upgraded ActiveMQ to 5.6.0. Now two > of the three production installations, that have been updated to our latest > version, are experiencing issues with ActiveMQ persistence causing ActiveMQ > messaging to be flaky at best. > > The issues are currently causing our server to be sometimes unable to send > messages to clients, since we are getting IndexOutOfBoundsExceptions and > IllegalStateExceptions trying to access inactive pages in KahaDB on our logs > (detailed stacktraces below). > > The product is a client-server product with fat clients. The server and > clients each have their own internal (as in running in the same JVM) brokers > with persistence running as a network of brokers. The reason the setup is > this is that these systems are running on ships, which might have really bad > internal networks and we are using ActiveMQ to take care that no data is > lost even if a client is first disconnected from a server and then shutdown. > The reason why we switched from DerbyDB was that we were having performance > issues and thought the KahaDB being purpose built would function better. > > Since the installations are on ships, we have no direct access to them, so > we cannot debug them. We are trying to get our hands on the KahaDB files in > order to see, if we can reproduce the issues in our development environment. > > In the mean time I would like to ask, if anyone has bumped in to similar > issues with ActiveMQ (5.6.0) and KahaDB? And if yes, were you able to solve > them in any other way than clearing the perisistence store and hoping for > the best? > > -Pauli > > An example of an IndexOutOfBoundsException (there are quite many of these in > the logs): > > Caused by: javax.jms.JMSException: Index: 3, Size: 2 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) > ~[activemq-pool-5.6.0.jar:5.6.0] > at > org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) > ~[activemq-pool-5.6.0.jar:5.6.0] > at <product>.messaging.Messenger.send(Messenger.java:168) > ~[<product>-impl-2012.3.1.jar:2012.3.1] > ... 18 common frames omitted > Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 2 > at java.util.ArrayList.rangeCheck(Unknown Source) ~[na:1.7.0_07] > at java.util.ArrayList.get(Unknown Source) ~[na:1.7.0_07] > at > org.apache.activemq.kaha.impl.index.hash.HashPage.getHashEntry(HashPage.java:181) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashPageInfo.getHashEntry(HashPageInfo.java:78) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:247) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.region.Topic.send(Topic.java:417) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > ~[activemq-core-5.6.0.jar:5.6.0] > ... 3 common frames omitted > > An example of the IllegalStateException (many of these as well): > > Caused by: javax.jms.JMSException: Trying to access an inactive page: 969728 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) > ~[activemq-pool-5.6.0.jar:5.6.0] > at > org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) > ~[activemq-pool-5.6.0.jar:5.6.0] > at <product>.messaging.Messenger.send(Messenger.java:168) > ~[<product>-impl-2012.3.1.jar:2012.3.1] > ... 10 common frames omitted > Caused by: java.lang.IllegalStateException: Trying to access an inactive > page: 969728 > at > org.apache.activemq.kaha.impl.index.hash.HashIndex.lookupPage(HashIndex.java:347) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashPageInfo.begin(HashPageInfo.java:96) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.getRetrievePage(HashBin.java:269) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:244) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.region.Topic.send(Topic.java:417) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > ~[activemq-core-5.6.0.jar:5.6.0] > ... 3 common frames omitted > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Issues-with-KahaDB-persistence-in-ActiveMQ-5-6-0-IndexOutOfBoundsException-IllegalStateException-tp4658031.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. -- http://redhat.com http://blog.garytully.com
