After a while of using the ActiveMQ with mKahadb it crashes with the
following errors:
2022-10-05 14:14:06,735 | WARN | Error subscribing to
/DvxSrv/12/Mine/Sampling |
org.apache.activemq.transport.mqtt.strategy.AbstractMQTTSubscriptionStrategy
| ActiveMQ Transport: tcp:///10.26.163.157:40380@1883
*java.lang.OutOfMemoryError: Java heap space*
at
org.apache.activemq.store.kahadb.disk.journal.Journal.recoveryCheck(Journal.java:499)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.disk.journal.Journal.start(Journal.java:334)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:458)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:502)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:306)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:221)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.startAdapter(MultiKahaDBPersistenceAdapter.java:212)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.getMatchingPersistenceAdapter(MultiKahaDBPersistenceAdapter.java:205)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.createTopicMessageStore(MultiKahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:99)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:626)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.jmx.ManagedTopicRegion.createDestination(ManagedTopicRegion.java:56)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:155)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:339)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:242)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:558)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:345)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:188)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:418)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:104)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:131)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:104)[activemq-broker-5.15.9.jar:5.15.9]
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:104)[activemq-broker-5.15.9.jar:5.15.9]
2022-10-05 14:14:12,329 | WARN | Exception encountered during context
initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'secHandlerCollection' defined in class path resource
[jetty.xml]: Cannot create inner bean
'org.eclipse.jetty.webapp.WebAppContext#9b6d9f7' of type
[org.eclipse.jetty.webapp.WebAppContext] while setting bean property
'handlers' with key [1]; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'org.eclipse.jetty.webapp.WebAppContext#9b6d9f7' defined in
class path resource [jetty.xml]: Initialization of bean failed; nested
exception is java.lang.OutOfMemoryError: *GC overhead limit exceeded* |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2022-10-05 14:14:14,454 | INFO | Stopping BrokerService[localhost] due to
exception, java.io.IOException: GC overhead limit exceeded |
org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Journal
Checkpoint Worker
java.io.IOException: *GC overhead limit exceeded*
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)[activemq-client-5.15.9.jar:5.15.9]
at
org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:451)[activemq-kahadb-store-5.15.9.jar:5.15.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)[:1.8.0_301]
at java.util.concurrent.FutureTask.runAndReset(Unknown
Source)[:1.8.0_301]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)[:1.8.0_301]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)[:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)[:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)[:1.8.0_301]
at java.lang.Thread.run(Unknown Source)[:1.8.0_301]
It seems to me that the mKahadb is using a lot more memory ram to handle
the queues and topics than the default kahadb.
Despite that, I realized that maybe the Java is not configured to use the
max possible Memory Ram that the customer has available for the ActiveMQ.
So, I set up the *JAVA_OPTS *system env in our customer with the values
-Xms512m -Xmx8196m. After that, I restarted the ActiveMQ but I still
receive the "*GC overhead limit exceeded" *exception*. *I don't know if I
need to do more config changes.
Questions:
1. Is the memory ram used by the Java GC a known problem when we set up the
ActiveMQ to use the mKahadb?
2. How do I know how much memory ram the ActiveMQ is currently using for
the GC purpose?
Note: The devices that send messages to our ActiveMQ broker run in a poor
network, so they can become offline for hours.
Thanks
*Rafael Pimenta*
Em qua., 5 de out. de 2022 às 10:18, Rafael Pimenta <rafa.spime...@gmail.com>
escreveu:
> As suggested by Mark Johnson I'm evaluating a multi-kahadb setup:
> <persistenceAdapter>
> <mKahaDB directory="${activemq.base}/data/kahadb">
> <filteredPersistenceAdapters>
> <!-- kahaDB per destinations -->
> <filteredKahaDB perDestination="true">
> <persistenceAdapter>
> <kahaDB journalMaxFileLength="1mb"/>
> </persistenceAdapter>
> </filteredKahaDB>
> </filteredPersistenceAdapters>
> </mKahaDB>
> </persistenceAdapter>
>
> But, after that, I'm facing the following warning:
> 2022-10-04 18:05:23,555 | WARN | Destination name is longer than
> 'MaximumFileNameLength' system property, potential problem with recovery
> can result from name truncation. |
> org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter | ActiveMQ
> Transport: tcp:///192.168.100.123:34926@1883
>
> Where can I change the MaximumFileNameLength to be greater than 64
> characters?
> I saw this ticket related to the underlying problem, but I can't see where
> to change that configuration:
> https://issues.apache.org/jira/browse/AMQ-4183
>
> Att,
> *Rafael Pimenta*
>
> Em seg., 3 de out. de 2022 às 10:34, Rafael Pimenta <
> rafa.spime...@gmail.com> escreveu:
>
>> Hello Marco,
>> First of All, Thanks for your reply and detailed explanation.
>>
>> To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
>> expired messages will be discarded automatically:
>>
>> <deadLetterStrategy>
>> <sharedDeadLetterStrategy processExpired="false" />
>> </deadLetterStrategy>
>>
>> Because of that configuration, I assume that the problems fall on durable
>> topics.
>> As we know the number of durable topics must be created every time our
>> system start, I'll study virtual topics and check if it is suitable for our
>> purposes. I'll let you ASAP.
>>
>> Thanks!
>> *Rafael Pimenta*
>>
>>
>>
>>
>> Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <jamiro...@gmail.com>
>> escreveu:
>>
>>> Hi Rafael,
>>> I’m just an ActiveMQ classic user.
>>>
>>> I will report the behavior of ActiveMQ I experienced.
>>>
>>> As far as I know the ActiveMQ classic journal is composed of append only
>>> files.
>>> Each file should be by default 32MB in size (this can be changed, I think
>>> is controlled by journalLogFileSize)
>>> A journal file gets released only when ALL the messages within are
>>> consumed.
>>> Furthermore, messages form different queues are appended to and mixed
>>> within the same file until the maximum size gets reached.
>>>
>>> As consequence if you have unprocessed messages (DLQ or offline
>>> consumers)
>>> the used storage will increase, usually in the same order of magnitude as
>>> the total traffic served by the broker: a small percentage of unconsumed
>>> messages will prevent journal file cleanup.
>>>
>>> For example an unprocessed 1KByte message will retain 32MB on disk.
>>>
>>> As far as I know there is no building functionality to perform journal
>>> compaction.
>>>
>>> In our case the developed internally a small application to offload
>>> unprocessed messages to an external database/ requeue DLQ messages to the
>>> same DLQ in order to free storage.
>>>
>>> In my experience persistent Topics are quite hard to monitor, if the list
>>> of subscribers is not dynamic (known when application starts) you could
>>> think to use VirtualTopics so that you will get back to a normal queue
>>> uses
>>> case for each consumer.
>>>
>>> Regards,
>>> Marc
>>>
>>> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
>>> rafa.spime...@gmail.com> ha scritto:
>>>
>>> > Hi Dears,
>>> > I'm using ActiveMQ 5.15.9 and after about one month our customers
>>> started
>>> > to complain about the Kahadb folder increase until reaching its limit
>>> and
>>> > crash the system. Then they delete the folder to recover the system,
>>> but we
>>> > are losing data in this process.
>>> > We activate the kahadb log and seems that some old messages are getting
>>> > stuck for some reason and do not allow the ActiveMQ to delete the
>>> db-x.log
>>> > files:
>>> >
>>> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
>>> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
>>> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
>>> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141],
>>> 256=[256,
>>> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259],
>>> 260=[259,
>>> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262,
>>> 254,
>>> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
>>> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
>>> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274,
>>> 271],
>>> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277,
>>> 278],
>>> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
>>> 283=[159],
>>> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
>>> 287],
>>> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
>>> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297],
>>> 298=[174,
>>> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
>>> 303=[302,
>>> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
>>> 306,
>>> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
>>> 313=[312,
>>> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
>>> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
>>> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326,
>>> 278,
>>> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
>>> 330],
>>> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251,
>>> 333,
>>> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337,
>>> 339],
>>> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
>>> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347],
>>> 349=[348,
>>> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
>>> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21],
>>> 355=[353,
>>> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356,
>>> 358],
>>> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
>>> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361,
>>> 362],
>>> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
>>> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
>>> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372,
>>> 373],
>>> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376,
>>> 377],
>>> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
>>> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383],
>>> 385=[384,
>>> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
>>> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
>>> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
>>> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
>>> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
>>> 274,
>>> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
>>> 399=[380,
>>> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
>>> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
>>> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
>>> 410=[409,
>>> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
>>> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
>>> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421],
>>> 422=[385,
>>> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
>>> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
>>> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400,
>>> 424,
>>> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
>>> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
>>> 436=[49,
>>> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52,
>>> 40],
>>> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
>>> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446],
>>> 447=[438,
>>> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
>>> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
>>> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
>>> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459],
>>> 460=[49, 53,
>>> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
>>> 463=[58,
>>> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466,
>>> 467],
>>> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470,
>>> 471],
>>> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474,
>>> 475],
>>> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
>>> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483,
>>> 484],
>>> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
>>> 66],
>>> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
>>> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
>>> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
>>> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
>>> 459,
>>> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
>>> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511],
>>> 512=[512,
>>> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76],
>>> 517=[515,
>>> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
>>> 438,
>>> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
>>> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
>>> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
>>> 526],
>>> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
>>> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517,
>>> 534,
>>> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
>>> 540=[538,
>>> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544,
>>> 543],
>>> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
>>> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
>>> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
>>> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558],
>>> 559=[64, 68,
>>> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
>>> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
>>> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568,
>>> 569],
>>> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
>>> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577,
>>> 561,
>>> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
>>> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
>>> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
>>> 589=[211],
>>> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
>>> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596,
>>> 597],
>>> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597,
>>> 599,
>>> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602,
>>> 586,
>>> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
>>> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609,
>>> 611,
>>> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599,
>>> 600],
>>> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
>>> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618],
>>> 619=[618,
>>> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621,
>>> 622],
>>> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
>>> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615,
>>> 619,
>>> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
>>> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
>>> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
>>> 639=[243]} |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
>>> > contained ack(s) refer to referenced file: [244, 243] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
>>> > contained ack(s) refer to referenced file: [123] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
>>> > contained ack(s) refer to referenced file: [244, 246] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
>>> > contained ack(s) refer to referenced file: [246, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,564 | WARN | *Stealing link for clientId
>>> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
>>> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
>>> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>>> > contained ack(s) refer to referenced file: [123] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>>> > contained ack(s) refer to referenced file: [249, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>>> 237] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>>> > contained ack(s) refer to referenced file: [123] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>>> > contained ack(s) refer to referenced file: [249, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>>> 237] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
>>> > contained ack(s) refer to referenced file: [140, 108, 97] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
>>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
>>> > contained ack(s) refer to referenced file: [253, 254] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>>> > contained ack(s) refer to referenced file: [140, 141] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker not removing data file: 252 as contained ack(s)
>>> refer to
>>> > referenced file: [140, 108, 97] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
>>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
>>> > contained ack(s) refer to referenced file: [253, 254] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>>> > contained ack(s) refer to referenced file: [140, 141] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> >
>>> > We also use durable topics as the devices that subscribe to these
>>> topics
>>> > and send MQTT messages to others work in a very bad network
>>> environment,
>>> > this is the way we found to avoid data loss while offline.
>>> > I think the problem is related to durable topics with offline
>>> > subscriptions, but how can I confirm this assumption, eg which topics?
>>> What
>>> > signatures?
>>> >
>>> > I much appreciate any thoughts!
>>> > Thanks in advance
>>> >
>>>
>>
