Ok, so I hooked up jconsole and looked at the queues, all of the queues
were showing zero. Right underneath the queues there was a section that
said "RecoveredXaTransaction". It had one entry
XID_[1096044365_globalId&3132362e392e312e3133392e746..... There
really is not very much more information available in the attributes. I
ended up hitting the heuristicRollback operation and shortly afterwards
the old KahaDB files were removed. Great news!
I checked the queues afterwards and rolling back the XA transaction did
not cause the message to go anywhere. Does anyone know what this might
have been and how to avoid it from occurring in the future?
Thanks!
Steve.
On 9/22/16 9:09 AM, Tim Bain wrote:
Apparently you can scratch the "but not impossible" bit from my reply.
Thanks Tim, I didn't know it wasn't allowed.
On Sep 22, 2016 7:07 AM, "Tim Bain" <tb...@alumni.duke.edu> wrote:
Topics will only hold onto messages if 1) a consumer is currently
connected but hasn't consumed the messages, or 2) a consumer subscribed
durably but is now disconnected. #1 indicates a slow (or intentionally
misbehaving) consumer, and isn't likely to be a problem over the span of
time you're describing. It also would result in the messages being
discarded when the broker is restarted, so if you've done a restart and the
files weren't cleaned up, this isn't the problem. #2 would be unusual for
advisory topic subscriptions, but not impossible (especially since you
apparently had this situation with your mis-created non-advisory topic).
I'd use JConsole (because it gives more detail than the web console) to go
through each queue looking for a non-zero number of messages, and each
topic looking for offline durable subscribers that have unconsumed messages.
Alternatively, you all could run the KahaDB file analysis tool (
http://activemq.apache.org/why-do-kahadb-log-files-
remain-after-cleanup.html) with a debugger attached to try to see what
destination has the message(s) that are keeping 980 and 1438 alive. (You
say that 1328 shouldn't be there, but the log shows that it's there because
it contains acks for messages in 980, so it should be there, at least until
980 is deleted.)
Tim
On Sep 22, 2016 6:39 AM, "Steve Hill" <sh...@futurelogix.co.uk> wrote:
After the comment about the TOPIC I decided to take a quick look at that
in the web interface (just to make sure nothing had gone awry) In deed one
of our "queues" had been created as a topic, not sure why as we use the
same code to connect to the queues however once removed most of the data
files vanished.
Under the topics there are several activemq.advisory.xyz topics. Could
these be the source of the remaining kahadb log files that still remain?
I am now down to three data files that should not be there. (db-980,
db-1329, and db-1438)
-rw-r--r--. 1 activemq activemq 33554432 Aug 8 14:00 db-980.log
-rw-r--r--. 1 activemq activemq 527645 Sep 19 19:11 db-1329.log
drwxr-xr-x. 5 activemq activemq 4096 Sep 20 14:57 ..
-rw-r--r--. 1 activemq activemq 8 Sep 21 13:17 lock
-rw-r--r--. 1 activemq activemq 33553518 Sep 21 21:52 db-1438.log
drwxr-xr-x. 2 activemq activemq 12288 Sep 22 08:20 .
-rw-r--r--. 1 activemq activemq 3287296 Sep 22 08:34 db.redo
-rw-r--r--. 1 activemq activemq 12357632 Sep 22 08:34 db.data
-rw-r--r--. 1 activemq activemq 33554432 Sep 22 08:34 db-1456.log
From kahadb logs
2016-09-22 08:31:07,503 [eckpoint Worker] TRACE MessageDatabase
- gc candidates: [1329]
2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase
- ackMessageFileMap: {1329=[980], 1438=[], 1456=[]}
2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1329 as contained ack(s) refer to
referenced file: [980]
2016-09-22 08:31:07,505 [eckpoint Worker] TRACE MessageDatabase
- Not yet time to check for compaction: 7 of 10 cycles
2016-09-22 08:31:07,505 [eckpoint Worker] DEBUG MessageDatabase
- Checkpoint done.
Thanks!
Steve.
On 9/22/16 8:14 AM, Steve Hill wrote:
Chris/Tim:
Thanks for the ideas. Unfortunately neither are the case in our
situation. The web client shows 0 messages in any of our queues and we are
not using topics. We did set up the remote debug port and downloaded the
source code if there is anything useful to further troubleshoot that could
be gleaned from there?
Thanks!
Steve.
The most likely place for you to have old messages laying around is in the
DLQ. The next most likely place is on a topic for an offline (and never
coming back) durable subscriber.
Tim
On Sep 20, 2016 2:02 PM, "Christopher Shannon" <
christopher.l.shan...@gmail.com> wrote:
Do you have old messages laying around? KahaDB can't clean up old journal
files if there are messages scattered throughout the logs. Even a single
unacked message means that a journal file (and the future files) can't be
cleaned up. This situation tends to happen either when there are old
messages not being acked quick enough or there are different destinations
with different speeds. Ie if one queue has a very slow message rate and
another queue has a high message rate.
Assuming there isn't some sort of bug with the GC process then one way to
help lower the usage would be to split up destinations into different
stores using multiKahaDB. Take a look at Gary's blog post on this:
http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
On Tue, Sep 20, 2016 at 1:04 PM, Hill, Steve <sh...@armada.net> wrote:
Hi fellow ActiveMQ users!
I have searched all over the internet and can not find a solution to a
problem we are experiencing where old log files are not removed from
KAHADB. Needless to say we uncovered this problem when we exceeded the
size of our persistent store in production :-)
We are using client side acknowledgement for all reads from the Queue and
have restarted all clients and also ActiveMQ numerous times, yet the
messages do not get removed. We are also using the network connector to
move messages to the queue that has an active consumer. We were using
5.13.0 but moved to 5.14.0 on the hope it might address this problem.
Follows are some more detailed messages; does anyone have an idea how we
can resolve this problem?
KAHADB logging is enabled and we see the following messages in the log
file.
2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1278 as contained ack(s) refer to
referenced file: [1278, 1277]
2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1279 as contained ack(s) refer to
referenced file: [674]
2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1280 as contained ack(s) refer to
referenced file: [691]
2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1281 as contained ack(s) refer to
referenced file: [699]
2016-09-20 08:59:10,209 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1282 as contained ack(s) refer to
referenced file: [699]
2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1283 as contained ack(s) refer to
referenced file: [705]
2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1284 as contained ack(s) refer to
referenced file: [711]
2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1288 as contained ack(s) refer to
referenced file: [674]
2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1289 as contained ack(s) refer to
referenced file: [790]
2016-09-20 08:59:10,212 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1290 as contained ack(s) refer to
referenced file: [1290, 1278]
2016-09-20 08:59:10,213 [eckpoint Worker] TRACE MessageDatabase
- not removing data file: 1291 as contained ack(s) refer to
referenced file: [790]
The files at this point are months old.
-rw-r--r--. 1 activemq activemq 33554432 Jul 14 21:23 db-674.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 16 01:57 db-691.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 17 02:10 db-699.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 18 03:55 db-705.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 19 01:32 db-711.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 25 15:15 db-790.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 26 10:32 db-858.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 27 00:19 db-865.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 27 03:03 db-866.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 28 01:45 db-877.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 29 01:18 db-888.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 30 02:10 db-899.log
-rw-r--r--. 1 activemq activemq 33554432 Jul 31 01:11 db-904.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 1 00:19 db-909.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 1 04:42 db-910.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 2 01:07 db-918.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 3 01:38 db-930.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 4 00:19 db-939.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 4 03:16 db-940.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 5 00:23 db-949.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 6 02:42 db-960.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 7 02:18 db-968.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 8 00:44 db-975.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 8 14:00 db-980.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 8 15:41 db-981.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 9 01:43 db-1051.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 11 00:20 db-1066.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 11 03:01 db-1067.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 12 00:19 db-1074.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 12 03:05 db-1075.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 13 00:19 db-1082.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 13 03:04 db-1083.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 14 01:06 db-1093.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 15 01:18 db-1111.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 16 02:38 db-1126.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 17 00:19 db-1137.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 17 02:32 db-1138.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 18 00:21 db-1155.log
-rw-r--r--. 1 activemq activemq 33554432 Aug 19 00:51 db-1167.log
-rw-r--r--. 1 activemq activemq 33554432 Sep 19 13:42 db-1277.log
-rw-r--r--. 1 activemq activemq 597139 Sep 19 14:59 db-1279.log
Thanks!
Steve Hill
ARMADA
The information contained in this e-mail and any attachments is
confidential and
intended only for the recipient. If you are not the intended recipient,
the
information contained in this message may not be used, copied, or
forwarded to
third parties or otherwise distributed for any other purpose. Please
notify the
sender if you received this e-mail in error and delete the e-mail and its
attachments promptly. Nothing in this e-mail may be used or deemed to
form the
basis of a contractual or any other legally binding obligation unless
separately
confirmed in writing by an authorized representative of ARMADA.
