KahaDB log files not getting cleaned up -- Why?

[Andy Boothe [WCG]]

Hello,

I have an ActiveMQ instance that has been running nicely for a couple of 
months, but doesn't seem to clean up its KahaDB logs like I expect it to. (More 
specifically, I expect to have like 1 log file, but in reality I have ~1500 log 
files with 49G of data.)

ActiveMQ has processed about 12M messages over the past couple of months, and 
has roughly 106K messages currently pending (in DLQs):

[cid:8627D172-A47C-4687-BDE1-89CFA12B5F38]

Given the number of messages currently pending, I would not expect the KahaDB 
logs to be very big. In reality, they're about 49G:

$ cd apache-activemq-5.8.0/data/kahadb/
$ ls -lht | head
total 49G
-rw-rw-r-- 1 ubuntu ubuntu  32M Dec 13 16:00 db-1562.log
-rw-rw-r-- 1 ubuntu ubuntu  36M Dec 13 16:00 db.data
-rw-rw-r-- 1 ubuntu ubuntu 173K Dec 13 16:00 db.redo
-rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 14:47 db-1561.log
-rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 13:28 db-1560.log
-rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 12:09 db-1559.log
-rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 10:55 db-1558.log
-rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 09:48 db-1557.log
-rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 08:40 db-1556.log

Based on the maximum log ID, almost none of the log files appear to have been 
cleaned up:

$ ls db-*.log | wc
   1540    1540   17421

I'm using ActiveMQ 5.8.0. The only change I've made to the vanilla 
configuration is Java heap size, which shouldn't affect whether or not DB log 
files get cleaned up (AFAIK). I'm "connecting" to ActiveMQ with Camel. 
Obviously from the number of queues, I have several different workflows in 
flight, but here's a representative example of how I'm using ActiveMQ in my 
Java code:


                from("activemq:statuslinks")

                    .bean(new LinkExtract())

                    .split(body(String.class).tokenize("\n"))

                    .filter(new LinkHandled(jdbcConnectionString))

                    .to("activemq:linkunwind");

The only problems I'm seeing in my activemq.log file appear to be "normal." I 
get rashes of transport exceptions every few weeks, which I have associated 
(rightly or wrongly) with hitting the web admin interface. Given the 
infrequency and nature of the errors, I don't think they explain why the logs 
wouldn't be deleted. In case it's useful, though, here are a couple of 
representative lines from the log file:

2013-12-13 09:46:52,269 | WARN  | Transport Connection to: 
tcp://127.0.0.1:39604 failed: 
org.apache.activemq.transport.InactivityIOException: Channel was inactive for 
too (>30000) long: tcp://127.0.0.1:39604 | 
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
InactivityMonitor Worker
2013-12-13 14:43:41,215 | WARN  | Transport Connection to: 
tcp://127.0.0.1:35711 failed: java.io.EOFException | 
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: 
tcp:///127.0.0.1:35711@61616

Given the use case — very simple message processing using Camel — and 
ActiveMQ's vanilla configuration, I would expect the log files to be tiny. 
Clearly, though they're not. Does anyone have any idea why those files aren't 
getting deleted? I'm sure I'm doing (or not doing) something very silly, but 
after pouring over the logs and documentation, I just can't figure out what I'm 
doing wrong. Does anyone have an idea why my logs wouldn't be getting cleaned 
up?

Thanks everyone in advance for your attention and help! And thank you ActiveMQ 
devs for such a wonderful product! :)


Andy Boothe