To get rid of the INFO messages, set org.apache.activemq.store.kahadb.LOG_SLOW_ACCESS_TIME to something like 8000
What message size are you using in your tests? Joe http://www.ttmsolutions.com Daniel Kluesing-2 wrote: > > I tried the suggestion of going with the default cursor, but I still get > OOM errors. I've included my full config file below, I think I'm running > fairly vanilla/default. > > After about 350k persistent messages, the logs start to look like: > > INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took > 3118 ms > INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took > 5118 ms > INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took > 2736 ms > INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took > 2945 ms > INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took > 2654 ms > INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took > 3174 ms > INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took > 5891 ms > INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took > 2906 ms > INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took > 7619 ms > Exception in thread "InactivityMonitor WriteCheck" > java.lang.OutOfMemoryError: Java heap space > at java.util.jar.Attributes.read(Attributes.java:377) > at java.util.jar.Manifest.read(Manifest.java:182) > at java.util.jar.Manifest.<init>(Manifest.java:52) > at > java.util.jar.JarFile.getManifestFromReference(JarFile.java:165) > at java.util.jar.JarFile.getManifest(JarFile.java:146) > at > sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:221) > at java.net.URLClassLoader.access$000(URLClassLoader.java:56) > at java.net.URLClassLoader$1.run(URLClassLoader.java:195) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > at > org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132) > at > org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106) > at > org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) > at java.util.TimerThread.mainLoop(Timer.java:512) > at java.util.TimerThread.run(Timer.java:462) > > Config file: > > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd";> > > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > > <value>file:${activemq.base}/conf/credentials.properties</value> > </property> > </bean> > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="sub01chi" dataDirectory="${activemq.base}/data"> > > <managementContext> > <managementContext createConnector="true"/> > </managementContext> > > <persistenceAdapter> > <kahaDB directory="${activemq.base}/data/kahadb"/> > </persistenceAdapter> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue="P>" > producerFlowControl="true" > memoryLimit="10mb"></policyEntry> > </policyEntries> > </policyMap> > </destinationPolicy> > <systemUsage> > <systemUsage sendFailIfNoSpace="true"> > </systemUsage> > </systemUsage> > <transportConnectors> > <transportConnector name="openwire" > uri="tcp://0.0.0.0:61616"/> > </transportConnectors> > </broker> > <import resource="jetty.xml"/> > </beans> > > -----Original Message----- > From: Rob Davies [mailto:rajdav...@gmail.com] > Sent: Monday, January 18, 2010 10:42 PM > To: users@activemq.apache.org > Subject: Re: OOM with high KahaDB index time > > > On 18 Jan 2010, at 22:14, Daniel Kluesing wrote: > >> Hi, >> >> I'm running the 5.3 release as a standalone broker. In one case, a >> producer is running without a consumer, producing small, persistent >> messages, with the FileCursor pendingQueuePolicy (per >> https://issues.apache.org/activemq/browse/AMQ-2512) >> option and flow control memoryLimit set to 100mb for the queue in >> question. (Through a policy entry) >> >> As the queue grows above 300k messages, KahaDB indexing starts >> climbing above 1 second. At around 350k messages, the indexing is >> taking over 8 seconds. At this point, I start getting java out of >> heap space errors in essentially random parts of the code. After a >> while, the producers timeout with a channel inactive for too long >> error, and the entire broker basically wedges itself. At this point, >> consumers are generally unable to bind to the broker quitting with >> timeout errors. When they can connect, consuming a single message >> triggers an index re-build, which takes 2-8seconds. Turning on >> verbose garbage collection, the jvm is collecting like mad but >> reclaiming no space. >> >> If I restart the broker, it comes back up, I can consume the old >> messages, and can handle another 350k messages until it wedges. >> >> I can reproduce under both default gc and incremental gc. >> >> Two questions: >> - It seems like someone is holding onto a handle to the messages >> after they have been persisted to disk - is this a known issue? >> Should I open a JIRA for it? (Or is there another explanation?) >> >> - Is there any documentation about the internals of KahaDB - the >> kind of indices etc? I'd like to get a better understanding of the >> index performance and in general how KahaDB compares to something >> like BerkeleyDB. >> >> Thanks >> >> >> > > > There's is some confusion over naming of our persistence options that > doesn't help. There is Kaha - which uses multiple log files and a Hash > based index - this is currently used by the FileCursor - whilst KahaDB > is a newer implementation, which is more robust and typically uses a > BTreeIndex. There is currently a new implementation of the Filecursor > btw - but that's a different matter. You can't currently configure the > HashIndex via the FileCursor - but it looks like this is the problem > you are encountering - as it looks like you need to increase the max > hash buckets. > > > So I would recommend the following > 1. Use the default pendingQueuePolicy (which only uses a FileCursor > for non-persistent messages - and uses the underlying database for > persistent messages > 2. Try KahaDB - which - with the BTreeIndex - will not hit the > problems you are seeing with the Filecursor > > or - increase the maximum number of hash buckets for the FileCursor > index - by setting a Java system property - maximumCapacity to 65536 > (the default is 16384) > > cheers, > > Rob > > http://twitter.com/rajdavies > I work here: http://fusesource.com > My Blog: http://rajdavies.blogspot.com/ > I'm writing this: http://www.manning.com/snyder/ > > > > > > > -- View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27263696.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
