As suggested by Mark Johnson I'm evaluating a multi-kahadb setup:
<persistenceAdapter>
<mKahaDB directory="${activemq.base}/data/kahadb">
<filteredPersistenceAdapters>
<!-- kahaDB per destinations -->
<filteredKahaDB perDestination="true">
<persistenceAdapter>
<kahaDB journalMaxFileLength="1mb"/>
</persistenceAdapter>
</filteredKahaDB>
</filteredPersistenceAdapters>
</mKahaDB>
</persistenceAdapter>
But, after that, I'm facing the following warning:
2022-10-04 18:05:23,555 | WARN | Destination name is longer than
'MaximumFileNameLength' system property, potential problem with recovery
can result from name truncation. |
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter | ActiveMQ
Transport: tcp:///192.168.100.123:34926@1883
Where can I change the MaximumFileNameLength to be greater than 64
characters?
I saw this ticket related to the underlying problem, but I can't see where
to change that configuration: https://issues.apache.org/jira/browse/AMQ-4183
Att,
*Rafael Pimenta*
Em seg., 3 de out. de 2022 às 10:34, Rafael Pimenta <rafa.spime...@gmail.com>
escreveu:
> Hello Marco,
> First of All, Thanks for your reply and detailed explanation.
>
> To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
> expired messages will be discarded automatically:
>
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false" />
> </deadLetterStrategy>
>
> Because of that configuration, I assume that the problems fall on durable
> topics.
> As we know the number of durable topics must be created every time our
> system start, I'll study virtual topics and check if it is suitable for our
> purposes. I'll let you ASAP.
>
> Thanks!
> *Rafael Pimenta*
>
>
>
>
> Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <jamiro...@gmail.com>
> escreveu:
>
>> Hi Rafael,
>> I’m just an ActiveMQ classic user.
>>
>> I will report the behavior of ActiveMQ I experienced.
>>
>> As far as I know the ActiveMQ classic journal is composed of append only
>> files.
>> Each file should be by default 32MB in size (this can be changed, I think
>> is controlled by journalLogFileSize)
>> A journal file gets released only when ALL the messages within are
>> consumed.
>> Furthermore, messages form different queues are appended to and mixed
>> within the same file until the maximum size gets reached.
>>
>> As consequence if you have unprocessed messages (DLQ or offline consumers)
>> the used storage will increase, usually in the same order of magnitude as
>> the total traffic served by the broker: a small percentage of unconsumed
>> messages will prevent journal file cleanup.
>>
>> For example an unprocessed 1KByte message will retain 32MB on disk.
>>
>> As far as I know there is no building functionality to perform journal
>> compaction.
>>
>> In our case the developed internally a small application to offload
>> unprocessed messages to an external database/ requeue DLQ messages to the
>> same DLQ in order to free storage.
>>
>> In my experience persistent Topics are quite hard to monitor, if the list
>> of subscribers is not dynamic (known when application starts) you could
>> think to use VirtualTopics so that you will get back to a normal queue
>> uses
>> case for each consumer.
>>
>> Regards,
>> Marc
>>
>> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
>> rafa.spime...@gmail.com> ha scritto:
>>
>> > Hi Dears,
>> > I'm using ActiveMQ 5.15.9 and after about one month our customers
>> started
>> > to complain about the Kahadb folder increase until reaching its limit
>> and
>> > crash the system. Then they delete the folder to recover the system,
>> but we
>> > are losing data in this process.
>> > We activate the kahadb log and seems that some old messages are getting
>> > stuck for some reason and do not allow the ActiveMQ to delete the
>> db-x.log
>> > files:
>> >
>> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
>> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
>> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
>> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
>> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259],
>> 260=[259,
>> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262,
>> 254,
>> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
>> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
>> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274,
>> 271],
>> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277,
>> 278],
>> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
>> 283=[159],
>> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
>> 287],
>> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
>> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297],
>> 298=[174,
>> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
>> 303=[302,
>> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
>> 306,
>> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
>> 313=[312,
>> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
>> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
>> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
>> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
>> 330],
>> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251,
>> 333,
>> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337,
>> 339],
>> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
>> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
>> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
>> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
>> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
>> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
>> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
>> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
>> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
>> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
>> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376,
>> 377],
>> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
>> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
>> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
>> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
>> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
>> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
>> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
>> 274,
>> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
>> 399=[380,
>> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
>> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
>> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
>> 410=[409,
>> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
>> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
>> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
>> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
>> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
>> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
>> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
>> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
>> 436=[49,
>> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
>> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
>> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
>> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
>> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
>> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
>> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49,
>> 53,
>> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
>> 463=[58,
>> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466,
>> 467],
>> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470,
>> 471],
>> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474,
>> 475],
>> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
>> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483,
>> 484],
>> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
>> 66],
>> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
>> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
>> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
>> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
>> 459,
>> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
>> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511],
>> 512=[512,
>> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
>> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
>> 438,
>> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
>> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
>> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
>> 526],
>> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
>> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
>> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
>> 540=[538,
>> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
>> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
>> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
>> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
>> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64,
>> 68,
>> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
>> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
>> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568,
>> 569],
>> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
>> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
>> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
>> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
>> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
>> 589=[211],
>> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
>> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596,
>> 597],
>> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
>> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
>> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
>> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
>> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
>> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
>> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
>> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621,
>> 622],
>> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
>> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
>> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
>> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
>> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
>> 639=[243]} |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
>> > contained ack(s) refer to referenced file: [244, 243] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
>> > contained ack(s) refer to referenced file: [123] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
>> > contained ack(s) refer to referenced file: [244, 246] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
>> > contained ack(s) refer to referenced file: [246, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,564 | WARN | *Stealing link for clientId
>> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
>> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
>> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>> > contained ack(s) refer to referenced file: [123] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>> > contained ack(s) refer to referenced file: [249, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>> 237] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>> > contained ack(s) refer to referenced file: [123] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>> > contained ack(s) refer to referenced file: [249, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>> 237] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
>> > contained ack(s) refer to referenced file: [140, 108, 97] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
>> > contained ack(s) refer to referenced file: [253, 254] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>> > contained ack(s) refer to referenced file: [140, 141] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker not removing data file: 252 as contained ack(s) refer
>> to
>> > referenced file: [140, 108, 97] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
>> > contained ack(s) refer to referenced file: [253, 254] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>> > contained ack(s) refer to referenced file: [140, 141] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> >
>> > We also use durable topics as the devices that subscribe to these topics
>> > and send MQTT messages to others work in a very bad network environment,
>> > this is the way we found to avoid data loss while offline.
>> > I think the problem is related to durable topics with offline
>> > subscriptions, but how can I confirm this assumption, eg which topics?
>> What
>> > signatures?
>> >
>> > I much appreciate any thoughts!
>> > Thanks in advance
>> >
>>
>
