The most likely place for you to have old messages laying around is in the DLQ. The next most likely place is on a topic for an offline (and never coming back) durable subscriber.
Tim On Sep 20, 2016 2:02 PM, "Christopher Shannon" < christopher.l.shan...@gmail.com> wrote: Do you have old messages laying around? KahaDB can't clean up old journal files if there are messages scattered throughout the logs. Even a single unacked message means that a journal file (and the future files) can't be cleaned up. This situation tends to happen either when there are old messages not being acked quick enough or there are different destinations with different speeds. Ie if one queue has a very slow message rate and another queue has a high message rate. Assuming there isn't some sort of bug with the GC process then one way to help lower the usage would be to split up destinations into different stores using multiKahaDB. Take a look at Gary's blog post on this: http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html On Tue, Sep 20, 2016 at 1:04 PM, Hill, Steve <sh...@armada.net> wrote: > Hi fellow ActiveMQ users! > > I have searched all over the internet and can not find a solution to a > problem we are experiencing where old log files are not removed from > KAHADB. Needless to say we uncovered this problem when we exceeded the > size of our persistent store in production :-) > > We are using client side acknowledgement for all reads from the Queue and > have restarted all clients and also ActiveMQ numerous times, yet the > messages do not get removed. We are also using the network connector to > move messages to the queue that has an active consumer. We were using > 5.13.0 but moved to 5.14.0 on the hope it might address this problem. > > Follows are some more detailed messages; does anyone have an idea how we > can resolve this problem? > > KAHADB logging is enabled and we see the following messages in the log > file. > > > 2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1278 as contained ack(s) refer to > referenced file: [1278, 1277] > > 2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1279 as contained ack(s) refer to > referenced file: [674] > > 2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1280 as contained ack(s) refer to > referenced file: [691] > > 2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1281 as contained ack(s) refer to > referenced file: [699] > > 2016-09-20 08:59:10,209 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1282 as contained ack(s) refer to > referenced file: [699] > > 2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1283 as contained ack(s) refer to > referenced file: [705] > > 2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1284 as contained ack(s) refer to > referenced file: [711] > > 2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1288 as contained ack(s) refer to > referenced file: [674] > > 2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1289 as contained ack(s) refer to > referenced file: [790] > > 2016-09-20 08:59:10,212 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1290 as contained ack(s) refer to > referenced file: [1290, 1278] > > 2016-09-20 08:59:10,213 [eckpoint Worker] TRACE MessageDatabase > - not removing data file: 1291 as contained ack(s) refer to > referenced file: [790] > > The files at this point are months old. > > > -rw-r--r--. 1 activemq activemq 33554432 Jul 14 21:23 db-674.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 16 01:57 db-691.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 17 02:10 db-699.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 18 03:55 db-705.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 19 01:32 db-711.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 25 15:15 db-790.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 26 10:32 db-858.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 27 00:19 db-865.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 27 03:03 db-866.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 28 01:45 db-877.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 29 01:18 db-888.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 30 02:10 db-899.log > > -rw-r--r--. 1 activemq activemq 33554432 Jul 31 01:11 db-904.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 1 00:19 db-909.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 1 04:42 db-910.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 2 01:07 db-918.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 3 01:38 db-930.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 4 00:19 db-939.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 4 03:16 db-940.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 5 00:23 db-949.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 6 02:42 db-960.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 7 02:18 db-968.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 8 00:44 db-975.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 8 14:00 db-980.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 8 15:41 db-981.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 9 01:43 db-1051.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 11 00:20 db-1066.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 11 03:01 db-1067.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 12 00:19 db-1074.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 12 03:05 db-1075.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 13 00:19 db-1082.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 13 03:04 db-1083.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 14 01:06 db-1093.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 15 01:18 db-1111.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 16 02:38 db-1126.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 17 00:19 db-1137.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 17 02:32 db-1138.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 18 00:21 db-1155.log > > -rw-r--r--. 1 activemq activemq 33554432 Aug 19 00:51 db-1167.log > > -rw-r--r--. 1 activemq activemq 33554432 Sep 19 13:42 db-1277.log > > -rw-r--r--. 1 activemq activemq 597139 Sep 19 14:59 db-1279.log > > Thanks! > > Steve Hill > > ARMADA > > > > > The information contained in this e-mail and any attachments is > confidential and > intended only for the recipient. If you are not the intended recipient, the > information contained in this message may not be used, copied, or > forwarded to > third parties or otherwise distributed for any other purpose. Please > notify the > sender if you received this e-mail in error and delete the e-mail and its > attachments promptly. Nothing in this e-mail may be used or deemed to > form the > basis of a contractual or any other legally binding obligation unless > separately > confirmed in writing by an authorized representative of ARMADA. >
