Yeah, we're using whatever is the default persistence provider in 5.4.2. I opened a fresh ticket for it:
https://issues.apache.org/jira/browse/AMQ-3120 Thanks, Dan On Thu, Jan 6, 2011 at 10:42 AM, Gary Tully [via ActiveMQ] < ml-node+3177622-1458695149-140...@n4.nabble.com<ml-node%2b3177622-1458695149-140...@n4.nabble.com> > wrote: > AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems > to be different from your use case? > > On 6 January 2011 15:14, dcheckoway <[hidden > email]<http://user/SendEmail.jtp?type=node&node=3177622&i=0>> > wrote: > > > > > Gary, > > > > Thanks for the reply and advice. I'll definitely try disabling > > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an > IO > > exception handler. > > > > Unfortunately the issue can't reliably be reproduced. We have a cluster > of > > servers all running their own local embedded queue, and it happens on one > > > here, one there. > > > > That said, somebody else already opened a JIRA ticket for what looks like > a > > similar/nearly identical issue: > > https://issues.apache.org/jira/browse/AMQ-3051 > > > > Dan > > > > > > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] < > > [hidden email] > > <http://user/SendEmail.jtp?type=node&node=3177622&i=1><[hidden > email] <http://user/SendEmail.jtp?type=node&node=3177622&i=2>> > >> wrote: > > > >> One thought, there may be an issue with the default setting of true > >> for concurrentStoreAndDispatchQueues that is causing the index to be > >> out of sync with the store. > >> > >> It is worth trying to disable that, use: > >> <persistenceAdapter> > >> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb" > >> concurrentStoreAndDispatchQueues="false"/> > >> </persistenceAdapter> > >> > >> How often is this occurring, to really get to the bottom of this a > >> test case will be needed. > >> Does that data file eventually get written, or are there any disk space > >> issues? > >> It may help to configure an IO exception handler just in case. > >> http://activemq.apache.org/configurable-ioexception-handling.html > >> > >> Would be a good idea to create a jira issue to track this and attach > >> what ever log and application information you have to it. > >> > >> On 6 January 2011 12:39, dcheckoway <[hidden email]< > http://user/SendEmail.jtp?type=node&node=3177500&i=0>> > >> wrote: > >> > >> > > >> > I'm using an embedded broker (version 5.4.2) with persistence enabled. > > >> We're > >> > pumping hundreds of millions of messages per day through this thing. > >> Every > >> > once in a while, all of a sudden the KahaDB directory starts growing > >> > uncontrollably, and these errors start spewing out in the log over and > > >> > over... > >> > > >> > -------------------------- > >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; > >> > AbstractStoreCursor - Failed to fill batch > >> > java.lang.RuntimeException: java.io.IOException: Could not locate data > > >> file > >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) > > >> > >> > at > >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) > >> > at > >> > > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) > >> > at > >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) > >> > at > >> > > >> > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > > >> > >> > at > >> > > >> > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > >> > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > >> > >> > at java.lang.Thread.run(Thread.java:619) > >> > Caused by: java.io.IOException: Could not locate data file > >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > >> > at > org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) > >> > >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592) > >> > at > >> > > >> > org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) > > >> > >> > at > >> > > >> > org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) > > >> > >> > at > >> > > >> > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) > > >> > >> > at > >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728) > >> > at > >> > > >> > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) > > >> > >> > at > >> > > >> > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) > > >> > >> > ... 10 more > >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; > Queue > >> - > >> > Failed to page in more queue messages > >> > java.lang.RuntimeException: java.lang.RuntimeException: > >> java.io.IOException: > >> > Could not locate data file > >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) > > >> > >> > at > >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) > >> > at > >> > > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) > >> > at > >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) > >> > at > >> > > >> > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > > >> > >> > at > >> > > >> > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > >> > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > >> > >> > at java.lang.Thread.run(Thread.java:619) > >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not > >> locate > >> > data file > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) > > >> > >> > ... 9 more > >> > Caused by: java.io.IOException: Could not locate data file > >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > >> > at > org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) > >> > >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592) > >> > at > >> > > >> > org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) > > >> > >> > at > >> > > >> > org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) > > >> > >> > at > >> > > >> > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) > > >> > >> > at > >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728) > >> > at > >> > > >> > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) > > >> > >> > at > >> > > >> > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > > >> > >> > at > >> > > >> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) > > >> > >> > ... 10 more > >> > -------------------------- > >> > > >> > It doesn't happen consistently, but it has happened numerous times. > The > >> > only way to "fix" it is to kill the app, blow away the KahaDB data > >> > directory, and restart (ugh). > >> > > >> > Has anybody else seen this issue? Any idea why it happens? > >> > > >> > Here's the embedded broker config, fwiw: > >> > -------------------------- > >> > <broker xmlns="http://activemq.apache.org/schema/core"; > >> > id="embeddedActivemqBroker" > >> > useJmx="false" > >> > persistent="true" > >> > dataDirectory="/usr/local/embedded/activemq-data" > >> > schedulerSupport="false" > >> > advisorySupport="false" > >> > enableStatistics="true"> > >> > <destinationPolicy> > >> > <policyMap> > >> > <policyEntries> > >> > <policyEntry queue=">" producerFlowControl="false" > >> > memoryLimit="20mb"/> > >> > </policyEntries> > >> > </policyMap> > >> > </destinationPolicy> > >> > </broker> > >> > -------------------------- > >> > > >> > Help?! > >> > > >> > ----- > >> > Dan Checkoway > >> > dcheckoway gmail com > >> > -- > >> > View this message in context: > >> > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t> > < > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t&by-user=t>> > > >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > > >> > >> > >> > >> -- > >> http://blog.garytully.com > >> http://fusesource.com > >> > >> > >> ------------------------------ > >> View message @ > >> > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html?by-user=t> > >> > >> > >> > > > > > > ----- > > Dan Checkoway > > dcheckoway gmail com > > -- > > View this message in context: > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html?by-user=t> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > -- > http://blog.garytully.com > http://fusesource.com > > > ------------------------------ > View message @ > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177622.html > > To unsubscribe from KahaDB error: "Could not locate data file", click > here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>. > > ----- Dan Checkoway dcheckoway gmail com -- View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177745.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
