I'm using an embedded broker (version 5.4.2) with persistence enabled. We're
pumping hundreds of millions of messages per day through this thing. Every
once in a while, all of a sudden the KahaDB directory starts growing
uncontrollably, and these errors start spewing out in the log over and
over...
--------------------------
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
AbstractStoreCursor - Failed to fill batch
java.lang.RuntimeException: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
at org.apache.kahadb.journal.Journal.read(Journal.java:592)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 10 more
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue -
Failed to page in more queue messages
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate
data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
... 9 more
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
at org.apache.kahadb.journal.Journal.read(Journal.java:592)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 10 more
--------------------------
It doesn't happen consistently, but it has happened numerous times. The
only way to "fix" it is to kill the app, blow away the KahaDB data
directory, and restart (ugh).
Has anybody else seen this issue? Any idea why it happens?
Here's the embedded broker config, fwiw:
--------------------------
<broker xmlns="http://activemq.apache.org/schema/core";
id="embeddedActivemqBroker"
useJmx="false"
persistent="true"
dataDirectory="/usr/local/embedded/activemq-data"
schedulerSupport="false"
advisorySupport="false"
enableStatistics="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="20mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
</broker>
--------------------------
Help?!
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context:
http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
