> Would it be possible to just add more files to the journal from the colocated backup to recover missing messages ?
If you had a backup broker then it's certainly possible you could replace the offending journal with the journal from the backup. However, I wouldn't copy/replace individual journal files. You'd want to do the whole thing. > Alternatively is there a way to know in which journal file that offending prepared transaction is, accept the fact that journal file is gone and have it removed from journal ? You might be able to comb through the output of the 'data print' command to find which file it's in, but I'm not certain even that would be effective. In my opinion patching the broker is by far the simplest option (assuming you know Java). It's really just a couple of lines. > While I understand we are far behind sadly that is not something we can do quickly. I've worked in large organizations where upgrading things was difficult so I can empathize. That said, it's important to stay up-to-date if you're relying on community (i.e. free) support. If you had commercial support that would, of course, be a different story. You were commenting on ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading. Justin [1] https://issues.apache.org/jira/browse/ARTEMIS-2460 On Thu, May 12, 2022 at 6:05 PM Roy Cohen <[email protected]> wrote: > Alternatively is there a way to know in which journal file that offending > prepared transaction is, accept the fact that journal file is gone and have > it removed from journal ? > > On 12 May 2022, at 23:08, Roy Cohen <[email protected]> wrote: > > Thank you Justin. > > While I understand we are far behind sadly that is not something we can do > quickly. > > I did manage to restore the broker using its colocated backup binding from > the other broker and it is back up. > > Question: Would it be possible to just add more files to the journal from > the colocated backup to recover missing messages ? > > > > > On 12 May 2022, at 21:02, Justin Bertram <[email protected]> wrote: > > > > As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix > is > > pretty straightforward as it just ignores the prepared transaction > causing > > the problem. > > > > At the moment I see two ways for you to proceed. > > > > 1) Download the latest version of ActiveMQ Artemis [2], create a new > > instance, copy your journal to the instance's, start that instance, and > > then move messages from that instance back to your 1.3.0 broker (e.g. via > > Camel, JMS bridge, etc.). > > 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the > > artemis-server module (cd artemis-server; mvn clean install -DskipTests), > > deploy the fixed artemis-server-1.3.0.jar to your broker > > > > Aside from that, I strongly recommend you upgrade your broker to the > latest > > version. Version 1.3.0 was released in June 2016, almost 6 years ago now. > > There's been 36 releases since then. > > > > > > Justin > > > > [1] https://issues.apache.org/jira/browse/ARTEMIS-1764 > > [2] https://activemq.apache.org/components/artemis/download/ > > [3] > > > https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz > > [4] > > > https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5 > > > > > > On Wed, May 11, 2022 at 5:35 PM Roy Cohen <[email protected]> wrote: > > > >> Hi All > >> > >> We have an urgent situation with our Artemis brokers pair in production > >> which somehow has left 01 without any DLQs and 02 in a state where it’s > not > >> starting up. > >> > >> The one which isn’t starting up is throwing this Exception, > unfortunately > >> I haven’t been able to find much online. Any help would be greatly > >> appreciated > >> > >> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server] > >> (main) AMQ224000: Failure in initialisation: > >> java.lang.IllegalStateException: Cannot find queue with id 80 > >> at > >> > org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427) > >> [artemis-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394) > >> [artemis-jms-server-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65) > >> [artemis-cli-1.3.0.jar:1.3.0] > >> at > >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83) > >> [artemis-cli-1.3.0.jar:1.3.0] > >> at > >> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110) > >> [artemis-cli-1.3.0.jar:1.3.0] > >> at > >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72) > >> [artemis-cli-1.3.0.jar:1.3.0] > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> [rt.jar:1.8.0_191] > >> at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> [rt.jar:1.8.0_191] > >> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> [rt.jar:1.8.0_191] > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> [rt.jar:1.8.0_191] > >> at > >> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122) > >> [artemis-boot.jar:1.3.0] > >> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45) > >> [artemis-boot.jar:1.3.0] > >> > >> > >
