I'm not sure what caused the exception when reading the index but the behavior of the broker on IOException is defined by the IOExceptionHandler that is configured on the broker. Take a look at DefaultIOExceptionHandler and all of the available options. By default on IOException it will shut down the broker and try and restart it. It can be configured to ignore IOExceptions or to disable the restart and just shutdown, or even to kill the entire JVM, etc.
http://activemq.apache.org/configurable-ioexception-handling.html As far as why the restart did not work I'm guessing the lock wasn't released properly on shutdown for some reason. 5.10.0 is pretty old so I'm not sure there have been any fixes for this already. I personally set up my brokers to exit the JVM on IOException and then I have some other process restart the broker (puppet, etc) so that there is always a clean JVM on broker start. On Wed, Feb 22, 2017 at 7:45 AM, Christian Schneider < ch...@die-schneider.net> wrote: > Has anyone seen this before? > > After this exception the broker shuts down and is restarted by the wrapper > but then we get "kahadb\lock could not be locked as lock is already held > for this jvm.". > So there are actually two problems. Why does the broker shut down at all > and why is the restart not working? > > I would be happy about any feedback. > > Christian > > ------ > 2017-01-30 03:56:26,910 | ERROR | KahaDB failed to store to Journal | > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport: > tcp:///10.218.8.223:49837@61616 > java.io.IOException: The parameter is incorrect > at java.io.RandomAccessFile.readBytes(Native Method)[:1.7.0_13] > at java.io.RandomAccessFile.read(Unknown Source)[:1.7.0_13] > at java.io.RandomAccessFile.readFully(Unknown Source)[:1.7.0_13] > at java.io.RandomAccessFile.readFully(Unknown Source)[:1.7.0_13] > at org.apache.activemq.util.RecoverableRandomAccessFile.readFul > ly(RecoverableRandomAccessFile.java:75)[activemq-kahadb-stor > e-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.PageFile.readPage > (PageFile.java:878)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.Transaction$2. > readPage(Transaction.java:456)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.Transaction$2.< > init>(Transaction.java:447)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.Transaction.openI > nputStream(Transaction.java:444)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.Transaction.load( > Transaction.java:420)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.Transaction.load( > Transaction.java:377)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadN > ode(BTreeIndex.java:262)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.index.BTreeNode.getChi > ld(BTreeNode.java:225)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.index.BTreeNode.getLea > fNode(BTreeNode.java:676)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.index.BTreeNode.put( > BTreeNode.java:369)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.put( > BTreeIndex.java:189)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex > (MessageDatabase.java:1304)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase$11.execute( > MessageDatabase.java:1140)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.disk.page.Transaction.execu > te(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase.process( > MessageDatabase.java:1137)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase$10.visit( > MessageDatabase.java:1074)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand. > visit(KahaAddMessageCommand.java:241)[activemq-kahadb- > store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase.process( > MessageDatabase.java:1071)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase.store(Messa > geDatabase.java:978)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.MessageDatabase.store(Messa > geDatabase.java:958)[activemq-kahadb-store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageSt > ore.addMessage(KahaDBStore.java:424)[activemq-kahadb- > store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addM > essage(KahaDBTransactionStore.java:388)[activemq-kahadb- > store-5.10.0.jar:5.10.0] > at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1. > addMessage(KahaDBTransactionStore.java:160)[activemq-kahadb- > store-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.doMessageSend(Queue. > java:913)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.send(Queue.java:733) > [activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.AbstractRegion.send(Abstra > ctRegion.java:424)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.RegionBroker.send(RegionBr > oker.java:445)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(Mana > gedRegionBroker.java:297)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter. > java:147)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.CompositeDestinationBroker.send(C > ompositeDestinationBroker.java:96)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.TransactionBroker.send(Transactio > nBroker.java:307)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.MutableBrokerFilter.send(MutableB > rokerFilter.java:152)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.TransportConnection.processMessag > e(TransportConnection.java:496)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMe > ssage.java:756)[activemq-client-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.TransportConnection.service(Trans > portConnection.java:294)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.TransportConnection$1.onCommand( > TransportConnection.java:148)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50)[activemq-client-5.10.0.jar:5.10.0] > at org.apache.activemq.transport.WireFormatNegotiator.onCommand > (WireFormatNegotiator.java:113)[activemq-client-5.10.0.jar:5.10.0] > at org.apache.activemq.transport.AbstractInactivityMonitor.onCo > mmand(AbstractInactivityMonitor.java:270)[activemq-client-5. > 10.0.jar:5.10.0] > at org.apache.activemq.transport.TransportSupport.doConsume(Tra > nsportSupport.java:83)[activemq-client-5.10.0.jar:5.10.0] > at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTran > sport.java:214)[activemq-client-5.10.0.jar:5.10.0] > at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransp > ort.java:196)[activemq-client-5.10.0.jar:5.10.0] > at java.lang.Thread.run(Unknown Source)[:1.7.0_13] > > -- > Christian Schneider > http://www.liquid-reality.de > > Open Source Architect > http://www.talend.com >
