Sorry for taking a while to get back to you. Based on your originally message and after analyzing some index files I have actually discovered a bug with KahaDB that causes the index file to grow too large when things like out of memory errors occur or any unclean shutdown. On recovery the free pages in the index file are not being properly tracked which is leading to excessive index file usage. So this is probably the issue that you are seeing. I will be fixing this in version 5.14.4. Upon upgrade you will need to delete the old index and have it be rebuilt and then going forward it should work as intended. You can track the status of the bug here: https://issues.apache.org/jira/browse/AMQ-6590
In terms of the in memory cache, only a fixed number of pages are cached. The default is 10 thousand pages and each page is 4KB. So roughly 40 MB of memory will be used at most. If the number of pages in use grows beyond that then the cache will swap pages based on the defined eviction strategy. You can increase the number of pages in memory (which would help performance at the cost of eating up memory) by tweaking the indexCacheSize value of KahaDB. See http://activemq.apache.org/kahadb.html for more info. On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <n...@xiangjiabao.com> wrote: > Christopher: > > > Thank you for your explanation! > > Hope there would be other proper ways to shrink the index file in future > ActiveMQ releases. > > Once we monitored that our broker is killed by system due to "out of > memory". > > I know that meta cache in memory will be synced into index file at > checking points. _*Does this mean there is as much memory as the > db.data file being used?*_ > > Even though our broker is idle, we still find that it uses up much > memory. Could they be relevant associated? Thanks > > > Kind regards, > > Scorpio > > On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote: > > The index grows based off the amount of data it tracks in the journal, > > but doesn't shrink. This means if you had a lot of data in your > > journal at some point (several gigabytes, etc) then the index would > > have grown in size with enough page files to keep track of all of the > > data. This is usually not an issue because the index size generally > > stabilizes at some point relative to the work load of the broker. If > > you really want to shrink it the only way right now is to stop the > > broker, delete the index and restart so it does a replay and rebuilds > > the index but the size would just grow again as messages flow. > > > > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email] > > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote: > > > > > Hi, All, > > > > > > In my activemq instance, I find that the kahadb index file is larger > > than > > > the journal file. > > > > > > As far as I know, /db.data/ is the meta store, index for the journal > > file, > > > which should means it is much smaller than the journal file. > > However, it > > > does not look like that. The index file seems to keep growing while the > > > journal file is rotated. Do I miss something? Thanks. > > > > > > Log attached: > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh > > > 16M db-16.log > > > 1.2G db.data > > > 3.2M db.redo > > > 4.0K lock > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# > > > > > > Kind regards, > > > Scorpio > > > > > > > > > > > > -- > > > View this message in context: > > http://activemq.2283324.n4.nabble.com/kahadb-index-file- > larger-than-journal-file-tp4721284.html > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > ------------------------------------------------------------------------ > > If you reply to this email, your message will be added to the > > discussion below: > > http://activemq.2283324.n4.nabble.com/kahadb-index-file- > larger-than-journal-file-tp4721284p4721300.html > > > > To start a new topic under ActiveMQ - User, email > > ml-node+s2283324n2341805...@n4.nabble.com > > To unsubscribe from kahadb index file larger than journal file, click > > here > > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro= > unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj > EyODR8MTkzNzMzOTQ5NQ==>. > > NAML > > <http://activemq.2283324.n4.nabble.com/template/ > NamlServlet.jtp?macro=macro_viewer&id=instant_html% > 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.view.web.template.NodeNamespace&breadcrumbs= > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/kahadb-index-file-larger-than-journal-file- > tp4721284p4721315.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
