Chris, What will be the behavior for anyone who doesn't delete their index after upgrading? Will the broker continue to function as-is in that scenario, or is this a required procedure during an upgrade beyond 5.14.3?
Tim On Feb 2, 2017 5:02 AM, "Christopher Shannon" < christopher.l.shan...@gmail.com> wrote: > Sorry for taking a while to get back to you. > > Based on your originally message and after analyzing some index files I > have actually discovered a bug with KahaDB that causes the index file to > grow too large when things like out of memory errors occur or any unclean > shutdown. On recovery the free pages in the index file are not being > properly tracked which is leading to excessive index file usage. So this > is probably the issue that you are seeing. I will be fixing this in > version 5.14.4. Upon upgrade you will need to delete the old index and > have it be rebuilt and then going forward it should work as intended. You > can track the status of the bug here: > https://issues.apache.org/jira/browse/AMQ-6590 > > In terms of the in memory cache, only a fixed number of pages are cached. > The default is 10 thousand pages and each page is 4KB. So roughly 40 MB of > memory will be used at most. If the number of pages in use grows beyond > that then the cache will swap pages based on the defined eviction > strategy. You can increase the number of pages in memory (which would help > performance at the cost of eating up memory) by tweaking the indexCacheSize > value of KahaDB. See http://activemq.apache.org/kahadb.html for more > info. > > On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <n...@xiangjiabao.com> wrote: > > > Christopher: > > > > > > Thank you for your explanation! > > > > Hope there would be other proper ways to shrink the index file in future > > ActiveMQ releases. > > > > Once we monitored that our broker is killed by system due to "out of > > memory". > > > > I know that meta cache in memory will be synced into index file at > > checking points. _*Does this mean there is as much memory as the > > db.data file being used?*_ > > > > Even though our broker is idle, we still find that it uses up much > > memory. Could they be relevant associated? Thanks > > > > > > Kind regards, > > > > Scorpio > > > > On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote: > > > The index grows based off the amount of data it tracks in the journal, > > > but doesn't shrink. This means if you had a lot of data in your > > > journal at some point (several gigabytes, etc) then the index would > > > have grown in size with enough page files to keep track of all of the > > > data. This is usually not an issue because the index size generally > > > stabilizes at some point relative to the work load of the broker. If > > > you really want to shrink it the only way right now is to stop the > > > broker, delete the index and restart so it does a replay and rebuilds > > > the index but the size would just grow again as messages flow. > > > > > > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email] > > > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote: > > > > > > > Hi, All, > > > > > > > > In my activemq instance, I find that the kahadb index file is larger > > > than > > > > the journal file. > > > > > > > > As far as I know, /db.data/ is the meta store, index for the journal > > > file, > > > > which should means it is much smaller than the journal file. > > > However, it > > > > does not look like that. The index file seems to keep growing while > the > > > > journal file is rotated. Do I miss something? Thanks. > > > > > > > > Log attached: > > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh > > > > 16M db-16.log > > > > 1.2G db.data > > > > 3.2M db.redo > > > > 4.0K lock > > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# > > > > > > > > Kind regards, > > > > Scorpio > > > > > > > > > > > > > > > > -- > > > > View this message in context: > > > http://activemq.2283324.n4.nabble.com/kahadb-index-file- > > larger-than-journal-file-tp4721284.html > > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > > > > ------------------------------------------------------------ > ------------ > > > If you reply to this email, your message will be added to the > > > discussion below: > > > http://activemq.2283324.n4.nabble.com/kahadb-index-file- > > larger-than-journal-file-tp4721284p4721300.html > > > > > > To start a new topic under ActiveMQ - User, email > > > ml-node+s2283324n2341805...@n4.nabble.com > > > To unsubscribe from kahadb index file larger than journal file, click > > > here > > > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro= > > unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj > > EyODR8MTkzNzMzOTQ5NQ==>. > > > NAML > > > <http://activemq.2283324.n4.nabble.com/template/ > > NamlServlet.jtp?macro=macro_viewer&id=instant_html% > > 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > > BasicNamespace-nabble.view.web.template.NabbleNamespace- > > nabble.view.web.template.NodeNamespace&breadcrumbs= > > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > > > > > > > > > -- > > View this message in context: http://activemq.2283324.n4. > > nabble.com/kahadb-index-file-larger-than-journal-file- > > tp4721284p4721315.html > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > >
