Hello ActiveMQ Classic community, We recently saw a broker running 5.15.16 getting stuck during recovery. When ActiveMQ was started, we saw the following log messages and then nothing else:
2024-04-18 17:21:52,670 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@25bbe1b6: startup date [Thu Apr 18 17:21:52 UTC 2024]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main 2024-04-18 17:21:53,886 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/data/kahadb] | org.apache.activemq.broker.BrokerService | main 2024-04-18 17:22:04,246 | INFO | Page File: /data/kahadb/db.data. Recovering pageFile free list due to prior unclean shutdown.. | org.apache.activemq.store.kahadb.disk.page.PageFile | KahaDB Index Free Page Recovery 2024-04-18 17:22:04,250 | INFO | KahaDB is version 6 | org.apache.activemq.store.kahadb.MessageDatabase | main 2024-04-18 17:22:09,123 | INFO | Recovering from the journal @190198:17434278 | org.apache.activemq.store.kahadb.MessageDatabase | main 2024-04-18 17:53:10,507 | INFO | Page File: /data/kahadb/db.data. Recovered pageFile free list of size: 377183 | org.apache.activemq.store.kahadb.disk.page.PageFile | KahaDB Index Free Page Recovery During this time, we saw our KahaDB directory starting to grow in size. Over the course of around a day, we saw our KahaDB directory grow by around 1 TB. Taking a look in the directory, the space was primarily consumed by *.tmp files. We were able to fix our broker by deleting db.redo and the large *.tmp files and restarting ActiveMQ. We are still in the process of digging deeper into what exactly happened here, but has the ActiveMQ Classic community seen anything like this before? Thank you in advance. Regards, Derek Zhang
