We recently encountered a fatal error in which Artemis stopped. Can anyone help explain how we might have got here? Does this indicate issues with journal storage?
I don’t expect we’ll find a smoking gun, but perhaps you can suggest other log entries to look for or other diagnostics to capture? WARN [org.apache.activemq.artemis.core.server] AMQ222153: Cannot locate record for message id = 530583 on Journal org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal must be in state=LOADED, was [STOPPED] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendDeleteRecord(JournalImpl.java:1171) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteMessage(AbstractJournalStorageManager.java:454) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.postAcknowledge(QueueImpl.java:4054) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterCommit(RefsOperation.java:188) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:560) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:303) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:284) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.1.jar:?] WARN [org.apache.activemq.artemis.core.server] AMQ222154: Error checking DLQ org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal must be in state=LOADED, was [STOPPED] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendUpdateRecord(JournalImpl.java:1053) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.Journal.tryAppendUpdateRecord(Journal.java:152) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.updateDeliveryCount(AbstractJournalStorageManager.java:709) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(QueueImpl.java:3401) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.rollbackRedelivery(RefsOperation.java:169) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(RefsOperation.java:119) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(TransactionImpl.java:570) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done(TransactionImpl.java:421) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:284) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.1.jar:?]----- a bunch more of these and then this final one ----- WARN [org.apache.activemq.artemis.core.server] AMQ222154: Error checking DLQ org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal must be in state=LOADED, was [STOPPED] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendUpdateRecord(JournalImpl.java:1053) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.Journal.tryAppendUpdateRecord(Journal.java:152) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.updateDeliveryCount(AbstractJournalStorageManager.java:709) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(QueueImpl.java:3401) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.rollbackRedelivery(RefsOperation.java:169) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(RefsOperation.java:119) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(TransactionImpl.java:570) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done(TransactionImpl.java:421) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:284) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.1.jar:?] INFO [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter INFO [io.hawt.HawtioContextListener] Destroying hawtio services INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin INFO [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.27.1 [f22970c9-bdd1-11ed-bdc3-22fe978b5405] stopped, uptime 2 hours 13 minutes Upon startup we are seeing this WARN msg, not sure what it means but it may have something to do with this issue. Maybe we have something miss-configured and that is why we are seeing this issues? WARN [org.apache.activemq.artemis.core.server] AMQ222269: Please use a fixed value for "journal-pool-files". Default changed per https://issues.apache.org/jira/browse/ARTEMIS-1628 Thanks john [rg] <https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
