https://issues.apache.org/jira/browse/AMQ-6440
Tim Bain wrote > I was going to write a JIRA enhancement request for that when I had some > free time, but if you have time to write it before I get to it, that would > be fine. Please put the link here if you do. > > On Sep 22, 2016 9:00 AM, "lichtin" < > lichtin@ > > wrote: > >> Changing trace levels and using the debugger is not always an option in >> production. >> I'm wondering why we could not revive the "KahaDBJournalReader" tool >> from https://issues.jboss.org/browse/MB-756 and ship it with ActiveMQ. >> >> >> Tim Bain wrote >> > Alternatively, you all could run the KahaDB file analysis tool ( >> > http://activemq.apache.org/why-do-kahadb-log-files- >> remain-after-cleanup.html) >> > with a debugger attached to try to see what destination has the >> message(s) >> > that are keeping 980 and 1438 alive. (You say that 1328 shouldn't be >> > there, but the log shows that it's there because it contains acks for >> > messages in 980, so it should be there, at least until 980 is deleted.) >> > >> > Tim >> > >> > On Sep 22, 2016 6:39 AM, "Steve Hill" < >> >> > shill@.co >> >> > > wrote: >> > >> > After the comment about the TOPIC I decided to take a quick look at >> that >> > in >> > the web interface (just to make sure nothing had gone awry) In deed >> one >> > of >> > our "queues" had been created as a topic, not sure why as we use the >> same >> > code to connect to the queues however once removed most of the data >> files >> > vanished. >> > >> > Under the topics there are several activemq.advisory.xyz topics. Could >> > these be the source of the remaining kahadb log files that still >> remain? >> > >> > I am now down to three data files that should not be there. (db-980, >> > db-1329, and db-1438) >> > >> > >> > -rw-r--r--. 1 activemq activemq 33554432 Aug 8 14:00 db-980.log >> > -rw-r--r--. 1 activemq activemq 527645 Sep 19 19:11 db-1329.log >> > drwxr-xr-x. 5 activemq activemq 4096 Sep 20 14:57 .. >> > -rw-r--r--. 1 activemq activemq 8 Sep 21 13:17 lock >> > -rw-r--r--. 1 activemq activemq 33553518 Sep 21 21:52 db-1438.log >> > drwxr-xr-x. 2 activemq activemq 12288 Sep 22 08:20 . >> > -rw-r--r--. 1 activemq activemq 3287296 Sep 22 08:34 db.redo >> > -rw-r--r--. 1 activemq activemq 12357632 Sep 22 08:34 db.data >> > -rw-r--r--. 1 activemq activemq 33554432 Sep 22 08:34 db-1456.log >> > >> > From kahadb logs >> > >> > 2016-09-22 08:31:07,503 [eckpoint Worker] TRACE MessageDatabase >> > - gc candidates: [1329] >> > 2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase >> > - ackMessageFileMap: {1329=[980], 1438=[], 1456=[]} >> > 2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase >> > - not removing data file: 1329 as contained ack(s) refer to >> referenced >> > file: [980] >> > 2016-09-22 08:31:07,505 [eckpoint Worker] TRACE MessageDatabase >> > - Not yet time to check for compaction: 7 of 10 cycles >> > 2016-09-22 08:31:07,505 [eckpoint Worker] DEBUG MessageDatabase >> > - Checkpoint done. >> > >> > Thanks! >> > Steve. >> > >> > >> > On 9/22/16 8:14 AM, Steve Hill wrote: >> > >> >> Chris/Tim: >> >> >> >> Thanks for the ideas. Unfortunately neither are the case in our >> >> situation. The web client shows 0 messages in any of our queues and >> we >> >> are >> >> not using topics. We did set up the remote debug port and downloaded >> the >> >> source code if there is anything useful to further troubleshoot that >> >> could >> >> be gleaned from there? >> >> >> >> Thanks! >> >> Steve. >> >> >> >> >> >> The most likely place for you to have old messages laying around is in >> >> the >> >> DLQ. The next most likely place is on a topic for an offline (and >> never >> >> coming back) durable subscriber. >> >> >> >> Tim >> >> >> >> On Sep 20, 2016 2:02 PM, "Christopher Shannon" < >> >> >> >> > christopher.l.shannon@ >> >> >> wrote: >> >> >> >> Do you have old messages laying around? KahaDB can't clean up old >> >> journal >> >> files if there are messages scattered throughout the logs. Even a >> single >> >> unacked message means that a journal file (and the future files) can't >> be >> >> cleaned up. This situation tends to happen either when there are old >> >> messages not being acked quick enough or there are different >> destinations >> >> with different speeds. Ie if one queue has a very slow message rate >> and >> >> another queue has a high message rate. >> >> >> >> Assuming there isn't some sort of bug with the GC process then one way >> to >> >> help lower the usage would be to split up destinations into different >> >> stores using multiKahaDB. Take a look at Gary's blog post on this: >> >> http://blog.garytully.com/2011/11/activemq-multiple- >> kahadb-instances.html >> >> >> >> >> >> On Tue, Sep 20, 2016 at 1:04 PM, Hill, Steve < >> >> > SHill@ >> >> > > wrote: >> >> >> >> Hi fellow ActiveMQ users! >> >>> >> >>> I have searched all over the internet and can not find a solution to >> a >> >>> problem we are experiencing where old log files are not removed from >> >>> KAHADB. Needless to say we uncovered this problem when we exceeded >> the >> >>> size of our persistent store in production :-) >> >>> >> >>> We are using client side acknowledgement for all reads from the Queue >> >>> and >> >>> have restarted all clients and also ActiveMQ numerous times, yet the >> >>> messages do not get removed. We are also using the network connector >> to >> >>> move messages to the queue that has an active consumer. We were >> using >> >>> 5.13.0 but moved to 5.14.0 on the hope it might address this problem. >> >>> >> >>> Follows are some more detailed messages; does anyone have an idea how >> we >> >>> can resolve this problem? >> >>> >> >>> KAHADB logging is enabled and we see the following messages in the >> log >> >>> file. >> >>> >> >>> >> >>> 2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1278 as contained ack(s) refer to >> >>> referenced file: [1278, 1277] >> >>> >> >>> 2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1279 as contained ack(s) refer to >> >>> referenced file: [674] >> >>> >> >>> 2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1280 as contained ack(s) refer to >> >>> referenced file: [691] >> >>> >> >>> 2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1281 as contained ack(s) refer to >> >>> referenced file: [699] >> >>> >> >>> 2016-09-20 08:59:10,209 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1282 as contained ack(s) refer to >> >>> referenced file: [699] >> >>> >> >>> 2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1283 as contained ack(s) refer to >> >>> referenced file: [705] >> >>> >> >>> 2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1284 as contained ack(s) refer to >> >>> referenced file: [711] >> >>> >> >>> 2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1288 as contained ack(s) refer to >> >>> referenced file: [674] >> >>> >> >>> 2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1289 as contained ack(s) refer to >> >>> referenced file: [790] >> >>> >> >>> 2016-09-20 08:59:10,212 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1290 as contained ack(s) refer to >> >>> referenced file: [1290, 1278] >> >>> >> >>> 2016-09-20 08:59:10,213 [eckpoint Worker] TRACE MessageDatabase >> >>> - not removing data file: 1291 as contained ack(s) refer to >> >>> referenced file: [790] >> >>> >> >>> The files at this point are months old. >> >>> >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 14 21:23 db-674.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 16 01:57 db-691.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 17 02:10 db-699.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 18 03:55 db-705.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 19 01:32 db-711.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 25 15:15 db-790.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 26 10:32 db-858.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 27 00:19 db-865.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 27 03:03 db-866.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 28 01:45 db-877.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 29 01:18 db-888.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 30 02:10 db-899.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 31 01:11 db-904.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 1 00:19 db-909.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 1 04:42 db-910.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 2 01:07 db-918.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 3 01:38 db-930.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 4 00:19 db-939.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 4 03:16 db-940.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 5 00:23 db-949.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 6 02:42 db-960.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 7 02:18 db-968.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 8 00:44 db-975.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 8 14:00 db-980.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 8 15:41 db-981.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 9 01:43 db-1051.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 11 00:20 db-1066.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 11 03:01 db-1067.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 12 00:19 db-1074.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 12 03:05 db-1075.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 13 00:19 db-1082.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 13 03:04 db-1083.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 14 01:06 db-1093.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 15 01:18 db-1111.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 16 02:38 db-1126.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 17 00:19 db-1137.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 17 02:32 db-1138.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 18 00:21 db-1155.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 19 00:51 db-1167.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 33554432 Sep 19 13:42 db-1277.log >> >>> >> >>> -rw-r--r--. 1 activemq activemq 597139 Sep 19 14:59 db-1279.log >> >>> >> >>> Thanks! >> >>> >> >>> Steve Hill >> >>> >> >>> ARMADA >> >>> >> >>> >> >>> >> >>> >> >>> The information contained in this e-mail and any attachments is >> >>> confidential and >> >>> intended only for the recipient. If you are not the intended >> recipient, >> >>> >> >> the >> >> >> >>> information contained in this message may not be used, copied, or >> >>> forwarded to >> >>> third parties or otherwise distributed for any other purpose. Please >> >>> notify the >> >>> sender if you received this e-mail in error and delete the e-mail and >> >>> its >> >>> attachments promptly. Nothing in this e-mail may be used or deemed >> to >> >>> form the >> >>> basis of a contractual or any other legally binding obligation unless >> >>> separately >> >>> confirmed in writing by an authorized representative of ARMADA. >> >>> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> View this message in context: http://activemq.2283324.n4. >> nabble.com/KAHADB-clean-up-old-log-files-tp4716661p4716804.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> -- View this message in context: http://activemq.2283324.n4.nabble.com/KAHADB-clean-up-old-log-files-tp4716661p4716824.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
