The WARN messages about ActiveMQShutdownException are likely a symptom and not a cause of the broker shutdown. Those exceptions are being thrown because a few threads are still trying to access the journal after it's been shut down (as part of the overall broker shutdown process). You'd need to provide more of the log for additional investigation about the root cause of the broker shutdown.
The WARN message about AMQ222269 is just indicating that you're using an unfixed value for journal-pool-files in broker.xml (i.e. "-1"). It's safer to use a fixed value so that the pool of journal files can grow and especially shrink as necessary. You can read more about journal-pool-files in the documentation [1]. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/persistence.html#configuring-the-message-journal On Wed, Mar 15, 2023 at 10:54 AM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: > We recently encountered a fatal error in which Artemis stopped. Can > anyone help explain how we might have got here? Does this indicate issues > with journal storage? > > > > I don’t expect we’ll find a smoking gun, but perhaps you can suggest other > log entries to look for or other diagnostics to capture? > > > > WARN [org.apache.activemq.artemis.core.server] AMQ222153: Cannot locate > record for message id = 530583 on Journal > > org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal > must be in state=LOADED, was [STOPPED] > > at > org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendDeleteRecord(JournalImpl.java:1171) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteMessage(AbstractJournalStorageManager.java:454) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.QueueImpl.postAcknowledge(QueueImpl.java:4054) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.RefsOperation.afterCommit(RefsOperation.java:188) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:560) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:303) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:284) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) > ~[artemis-commons-2.27.1.jar:?] > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) > ~[artemis-commons-2.27.1.jar:?] > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) > ~[artemis-commons-2.27.1.jar:?] > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > ~[?:?] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > ~[?:?] > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > ~[artemis-commons-2.27.1.jar:?] > > WARN [org.apache.activemq.artemis.core.server] AMQ222154: Error checking > DLQ > > org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal > must be in state=LOADED, was [STOPPED] > > at > org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendUpdateRecord(JournalImpl.java:1053) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.journal.Journal.tryAppendUpdateRecord(Journal.java:152) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.updateDeliveryCount(AbstractJournalStorageManager.java:709) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(QueueImpl.java:3401) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.RefsOperation.rollbackRedelivery(RefsOperation.java:169) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(RefsOperation.java:119) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(TransactionImpl.java:570) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done(TransactionImpl.java:421) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:284) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) > ~[artemis-commons-2.27.1.jar:?] > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) > ~[artemis-commons-2.27.1.jar:?] > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) > ~[artemis-commons-2.27.1.jar:?] > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > ~[?:?] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > ~[?:?] > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > ~[artemis-commons-2.27.1.jar:?]----- a bunch more of these and then this > final one ----- > > WARN [org.apache.activemq.artemis.core.server] AMQ222154: Error checking > DLQ > > org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal > must be in state=LOADED, was [STOPPED] > > at > org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendUpdateRecord(JournalImpl.java:1053) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.journal.Journal.tryAppendUpdateRecord(Journal.java:152) > ~[artemis-journal-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.updateDeliveryCount(AbstractJournalStorageManager.java:709) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(QueueImpl.java:3401) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.RefsOperation.rollbackRedelivery(RefsOperation.java:169) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(RefsOperation.java:119) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(TransactionImpl.java:570) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done(TransactionImpl.java:421) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:284) > ~[artemis-server-2.27.1.jar:2.27.1] > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) > ~[artemis-commons-2.27.1.jar:?] > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) > ~[artemis-commons-2.27.1.jar:?] > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) > ~[artemis-commons-2.27.1.jar:?] > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > ~[?:?] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > ~[?:?] > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > ~[artemis-commons-2.27.1.jar:?] > > INFO [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server > > INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio > authentication filter > > INFO [io.hawt.HawtioContextListener] Destroying hawtio services > > INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed > artemis-plugin plugin > > INFO [org.apache.activemq.hawtio.branding.PluginContextListener] > Destroyed activemq-branding plugin > > INFO [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server > > INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ > Artemis Message Broker version 2.27.1 > [f22970c9-bdd1-11ed-bdc3-22fe978b5405] stopped, uptime 2 hours 13 minutes > > > > Upon startup we are seeing this WARN msg, not sure what it means but it > may have something to do with this issue. Maybe we have something > miss-configured and that is why we are seeing this issues? > > WARN [org.apache.activemq.artemis.core.server] AMQ222269: Please use a > fixed value for "journal-pool-files". Default changed per > https://issues.apache.org/jira/browse/ARTEMIS-1628 > > > > > > Thanks > > john > > [image: rg] <https://www.redpointglobal.com/> > > John Lilley > > Data Management Chief Architect, Redpoint Global Inc. > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
