> So we were thinking d that and append files from the live journal on top ?
I don't understand what you're asking here. > I keep seeing journal and journal1 directories whereas broker.xml has journal configured, any idea what is journal1 for ? I'm not sure. If you're using colocation with replication then that might be a replica backup caused by failing over and then stopping the backup before it was able to fail-back. In that situation when the backup restarts it will move its existing journal to a backup directory and then re-sync with the live broker. Out of curiosity, did you rule out patching the broker? If so, why? Justin On Fri, May 13, 2022 at 8:59 AM Roy Cohen <[email protected]> wrote: > Thanks again Justin. > > Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@) > > So, I was able to find the offending amq file, when I remove it from the > journal the broker does start up just fine however all queues are showing > zero count, whereas with a colocated backup restore we at least got a > “snapshot” of the day of the backup. So we were thinking d that and append > files from the live journal on top ? > > Another question: I keep seeing journal and journal1 directories whereas > broker.xml has journal configured, any idea what is journal1 for ? > > > > > On 13 May 2022, at 14:48, Justin Bertram <[email protected]> wrote: > > > >> Would it be possible to just add more files to the journal from the > > colocated backup to recover missing messages ? > > > > If you had a backup broker then it's certainly possible you could replace > > the offending journal with the journal from the backup. However, I > wouldn't > > copy/replace individual journal files. You'd want to do the whole thing. > > > >> Alternatively is there a way to know in which journal file that > offending > > prepared transaction is, accept the fact that journal file is gone and > have > > it removed from journal ? > > > > You might be able to comb through the output of the 'data print' command > to > > find which file it's in, but I'm not certain even that would be > effective. > > > > In my opinion patching the broker is by far the simplest option (assuming > > you know Java). It's really just a couple of lines. > > > >> While I understand we are far behind sadly that is not something we can > > do quickly. > > > > I've worked in large organizations where upgrading things was difficult > so > > I can empathize. That said, it's important to stay up-to-date if you're > > relying on community (i.e. free) support. If you had commercial support > > that would, of course, be a different story. You were commenting on > > ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading. > > > > > > Justin > > > > [1] https://issues.apache.org/jira/browse/ARTEMIS-2460 > > > > On Thu, May 12, 2022 at 6:05 PM Roy Cohen <[email protected]> wrote: > > > >> Alternatively is there a way to know in which journal file that > offending > >> prepared transaction is, accept the fact that journal file is gone and > have > >> it removed from journal ? > >> > >> On 12 May 2022, at 23:08, Roy Cohen <[email protected]> wrote: > >> > >> Thank you Justin. > >> > >> While I understand we are far behind sadly that is not something we can > do > >> quickly. > >> > >> I did manage to restore the broker using its colocated backup binding > from > >> the other broker and it is back up. > >> > >> Question: Would it be possible to just add more files to the journal > from > >> the colocated backup to recover missing messages ? > >> > >> > >> > >>> On 12 May 2022, at 21:02, Justin Bertram <[email protected]> wrote: > >>> > >>> As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The > fix > >> is > >>> pretty straightforward as it just ignores the prepared transaction > >> causing > >>> the problem. > >>> > >>> At the moment I see two ways for you to proceed. > >>> > >>> 1) Download the latest version of ActiveMQ Artemis [2], create a new > >>> instance, copy your journal to the instance's, start that instance, and > >>> then move messages from that instance back to your 1.3.0 broker (e.g. > via > >>> Camel, JMS bridge, etc.). > >>> 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the > >>> artemis-server module (cd artemis-server; mvn clean install > -DskipTests), > >>> deploy the fixed artemis-server-1.3.0.jar to your broker > >>> > >>> Aside from that, I strongly recommend you upgrade your broker to the > >> latest > >>> version. Version 1.3.0 was released in June 2016, almost 6 years ago > now. > >>> There's been 36 releases since then. > >>> > >>> > >>> Justin > >>> > >>> [1] https://issues.apache.org/jira/browse/ARTEMIS-1764 > >>> [2] https://activemq.apache.org/components/artemis/download/ > >>> [3] > >>> > >> > https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz > >>> [4] > >>> > >> > https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5 > >>> > >>> > >>> On Wed, May 11, 2022 at 5:35 PM Roy Cohen <[email protected]> > wrote: > >>> > >>>> Hi All > >>>> > >>>> We have an urgent situation with our Artemis brokers pair in > production > >>>> which somehow has left 01 without any DLQs and 02 in a state where > it’s > >> not > >>>> starting up. > >>>> > >>>> The one which isn’t starting up is throwing this Exception, > >> unfortunately > >>>> I haven’t been able to find much online. Any help would be greatly > >>>> appreciated > >>>> > >>>> 2022-05-11 20:35:38,673 ERROR > [org.apache.activemq.artemis.core.server] > >>>> (main) AMQ224000: Failure in initialisation: > >>>> java.lang.IllegalStateException: Cannot find queue with id 80 > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427) > >>>> [artemis-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394) > >>>> [artemis-jms-server-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65) > >>>> [artemis-cli-1.3.0.jar:1.3.0] > >>>> at > >>>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83) > >>>> [artemis-cli-1.3.0.jar:1.3.0] > >>>> at > >>>> > >> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110) > >>>> [artemis-cli-1.3.0.jar:1.3.0] > >>>> at > >>>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72) > >>>> [artemis-cli-1.3.0.jar:1.3.0] > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>>> [rt.jar:1.8.0_191] > >>>> at > >>>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>>> [rt.jar:1.8.0_191] > >>>> at > >>>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>> [rt.jar:1.8.0_191] > >>>> at java.lang.reflect.Method.invoke(Method.java:498) > >>>> [rt.jar:1.8.0_191] > >>>> at > >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122) > >>>> [artemis-boot.jar:1.3.0] > >>>> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45) > >>>> [artemis-boot.jar:1.3.0] > >>>> > >>>> > >> > >> > >
