On 14-12-08 08:57 PM, Tim Bain wrote:
The log line tells you how you're doing against two different limits: the
per-destination limit and the entire-broker limit. The per-destination
limit is first, and the log line says you're using 0% of what you've
allocated per destination, so you're right that you're fine there. The
full-broker limit is second, and the log line says you're using 104% of the
available memory, which is why you're getting flow controlled: although the
individual destination is empty, the broker as a whole is full, and this
one message just happens to be the straw that breaks the camel's back. So
figure out what other destination has lots of messages (or large messages)
to be using 70GB(!) of memory...
Thanks for the response. I've spent some time digging into it further
and your remarks helped to clarify some things. The following links
helped a bunch, also:
http://www.christianposta.com/blog/?p=273
http://open-source-adventures.blogspot.ca/2009/01/limiting-disk-store-usage-with-active.html
(contains some deprecated info)
https://access.redhat.com/documentation/en-US/Fuse_Message_Broker/5.4/html/Using_Persistent_Messages/files/FuseMBPersistIntro.html
I've since adjusted my config in the following way:
<persistenceAdapter>
<kahaDB
directory="${activemq.data}/kahadb"
journalMaxFileLength="100mb"
enableJournalDiskSyncs="true"
checksumJournalFiles="true"
checkForCorruptJournalFiles="true"
indexCacheSize="10000"
cleanupInterval="30000"
checkpointInterval="5000"
/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="50 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
I'd noticed that the log messages mentioned the delay time in increasing
intervals of 30 seconds. Could this be related to cleanupInterval?
I should point out that these message numbers are an infrequent event
with this application. The norm is to pass significantly fewer messages
at a time. With the above changes, i've run this maximal sync several
times without further trouble, save for the last run, in which a single
message refused to leave the queue, and without any suggestion in the
log that AMQ was flow controlling the consumer. I'll keep playing with it.
And did you mean ~700MB? That's much closer to what i'm seeing of the
actual data being sent over the wire.
