Thanks again Justin. Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@)
So, I was able to find the offending amq file, when I remove it from the journal the broker does start up just fine however all queues are showing zero count, whereas with a colocated backup restore we at least got a “snapshot” of the day of the backup. So we were thinking d that and append files from the live journal on top ? Another question: I keep seeing journal and journal1 directories whereas broker.xml has journal configured, any idea what is journal1 for ? > On 13 May 2022, at 14:48, Justin Bertram <[email protected]> wrote: > >> Would it be possible to just add more files to the journal from the > colocated backup to recover missing messages ? > > If you had a backup broker then it's certainly possible you could replace > the offending journal with the journal from the backup. However, I wouldn't > copy/replace individual journal files. You'd want to do the whole thing. > >> Alternatively is there a way to know in which journal file that offending > prepared transaction is, accept the fact that journal file is gone and have > it removed from journal ? > > You might be able to comb through the output of the 'data print' command to > find which file it's in, but I'm not certain even that would be effective. > > In my opinion patching the broker is by far the simplest option (assuming > you know Java). It's really just a couple of lines. > >> While I understand we are far behind sadly that is not something we can > do quickly. > > I've worked in large organizations where upgrading things was difficult so > I can empathize. That said, it's important to stay up-to-date if you're > relying on community (i.e. free) support. If you had commercial support > that would, of course, be a different story. You were commenting on > ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading. > > > Justin > > [1] https://issues.apache.org/jira/browse/ARTEMIS-2460 > > On Thu, May 12, 2022 at 6:05 PM Roy Cohen <[email protected]> wrote: > >> Alternatively is there a way to know in which journal file that offending >> prepared transaction is, accept the fact that journal file is gone and have >> it removed from journal ? >> >> On 12 May 2022, at 23:08, Roy Cohen <[email protected]> wrote: >> >> Thank you Justin. >> >> While I understand we are far behind sadly that is not something we can do >> quickly. >> >> I did manage to restore the broker using its colocated backup binding from >> the other broker and it is back up. >> >> Question: Would it be possible to just add more files to the journal from >> the colocated backup to recover missing messages ? >> >> >> >>> On 12 May 2022, at 21:02, Justin Bertram <[email protected]> wrote: >>> >>> As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix >> is >>> pretty straightforward as it just ignores the prepared transaction >> causing >>> the problem. >>> >>> At the moment I see two ways for you to proceed. >>> >>> 1) Download the latest version of ActiveMQ Artemis [2], create a new >>> instance, copy your journal to the instance's, start that instance, and >>> then move messages from that instance back to your 1.3.0 broker (e.g. via >>> Camel, JMS bridge, etc.). >>> 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the >>> artemis-server module (cd artemis-server; mvn clean install -DskipTests), >>> deploy the fixed artemis-server-1.3.0.jar to your broker >>> >>> Aside from that, I strongly recommend you upgrade your broker to the >> latest >>> version. Version 1.3.0 was released in June 2016, almost 6 years ago now. >>> There's been 36 releases since then. >>> >>> >>> Justin >>> >>> [1] https://issues.apache.org/jira/browse/ARTEMIS-1764 >>> [2] https://activemq.apache.org/components/artemis/download/ >>> [3] >>> >> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz >>> [4] >>> >> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5 >>> >>> >>> On Wed, May 11, 2022 at 5:35 PM Roy Cohen <[email protected]> wrote: >>> >>>> Hi All >>>> >>>> We have an urgent situation with our Artemis brokers pair in production >>>> which somehow has left 01 without any DLQs and 02 in a state where it’s >> not >>>> starting up. >>>> >>>> The one which isn’t starting up is throwing this Exception, >> unfortunately >>>> I haven’t been able to find much online. Any help would be greatly >>>> appreciated >>>> >>>> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server] >>>> (main) AMQ224000: Failure in initialisation: >>>> java.lang.IllegalStateException: Cannot find queue with id 80 >>>> at >>>> >> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427) >>>> [artemis-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394) >>>> [artemis-jms-server-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65) >>>> [artemis-cli-1.3.0.jar:1.3.0] >>>> at >>>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83) >>>> [artemis-cli-1.3.0.jar:1.3.0] >>>> at >>>> >> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110) >>>> [artemis-cli-1.3.0.jar:1.3.0] >>>> at >>>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72) >>>> [artemis-cli-1.3.0.jar:1.3.0] >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> [rt.jar:1.8.0_191] >>>> at >>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> [rt.jar:1.8.0_191] >>>> at >>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> [rt.jar:1.8.0_191] >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> [rt.jar:1.8.0_191] >>>> at >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122) >>>> [artemis-boot.jar:1.3.0] >>>> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45) >>>> [artemis-boot.jar:1.3.0] >>>> >>>> >> >>
