On 13 May 2022, at 20:47, Justin Bertram <[email protected]> wrote:
So we were thinking d that and append files from the live journal on top ? I don't understand what you're asking here. If my understanding is correct and the journal is always appending then I’m theory getting its co located backup from the other broker plus its current journal minus the file which causes it to crash should get us almost to where we are ? The reason why I’m asking this is because removing the offending file alone from the journal leaving me with zero messages, specially the DLQs which we need. However by taking the approach above judging by the number of messages in the queue plus DLQ it appears that we are almost close to where we were before the crash has happened (which by the way was caused by an unexpected data centre hardware fault). I keep seeing journal and journal1 directories whereas broker.xml has journal configured, any idea what is journal1 for ? I'm not sure. If you're using colocation with replication then that might be a replica backup caused by failing over and then stopping the backup before it was able to fail-back. In that situation when the backup restarts it will move its existing journal to a backup directory and then re-sync with the live broker. Yes we are using collocation with replication. So in theory if the above is correct then that directory is sort of a temp directory which should be left alone ? Out of curiosity, did you rule out patching the broker? If so, why? Because despite the fact it is a two lines change the efforts of patching, building, publishing, testing and getting it through the many environments is almost just as big as an upgrade. Even though technically speaking we can just patch prod where it failed and replace the binary that’s not how it should be done. I can only hope that with the current failure we’ve had a proper upgrade will be deemed as a higher priority… Justin On Fri, May 13, 2022 at 8:59 AM Roy Cohen <[email protected]> wrote: Thanks again Justin. Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@) So, I was able to find the offending amq file, when I remove it from the journal the broker does start up just fine however all queues are showing zero count, whereas with a colocated backup restore we at least got a “snapshot” of the day of the backup. So we were thinking d that and append files from the live journal on top ? Another question: I keep seeing journal and journal1 directories whereas broker.xml has journal configured, any idea what is journal1 for ? On 13 May 2022, at 14:48, Justin Bertram <[email protected]> wrote: Would it be possible to just add more files to the journal from the colocated backup to recover missing messages ? If you had a backup broker then it's certainly possible you could replace the offending journal with the journal from the backup. However, I wouldn't copy/replace individual journal files. You'd want to do the whole thing. Alternatively is there a way to know in which journal file that offending prepared transaction is, accept the fact that journal file is gone and have it removed from journal ? You might be able to comb through the output of the 'data print' command to find which file it's in, but I'm not certain even that would be effective. In my opinion patching the broker is by far the simplest option (assuming you know Java). It's really just a couple of lines. While I understand we are far behind sadly that is not something we can do quickly. I've worked in large organizations where upgrading things was difficult so I can empathize. That said, it's important to stay up-to-date if you're relying on community (i.e. free) support. If you had commercial support that would, of course, be a different story. You were commenting on ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading. Justin [1] https://issues.apache.org/jira/browse/ARTEMIS-2460 On Thu, May 12, 2022 at 6:05 PM Roy Cohen <[email protected]> wrote: Alternatively is there a way to know in which journal file that offending prepared transaction is, accept the fact that journal file is gone and have it removed from journal ? On 12 May 2022, at 23:08, Roy Cohen <[email protected]> wrote: Thank you Justin. While I understand we are far behind sadly that is not something we can do quickly. I did manage to restore the broker using its colocated backup binding from the other broker and it is back up. Question: Would it be possible to just add more files to the journal from the colocated backup to recover missing messages ? On 12 May 2022, at 21:02, Justin Bertram <[email protected]> wrote: As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix is pretty straightforward as it just ignores the prepared transaction causing the problem. At the moment I see two ways for you to proceed. 1) Download the latest version of ActiveMQ Artemis [2], create a new instance, copy your journal to the instance's, start that instance, and then move messages from that instance back to your 1.3.0 broker (e.g. via Camel, JMS bridge, etc.). 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the artemis-server module (cd artemis-server; mvn clean install -DskipTests), deploy the fixed artemis-server-1.3.0.jar to your broker Aside from that, I strongly recommend you upgrade your broker to the latest version. Version 1.3.0 was released in June 2016, almost 6 years ago now. There's been 36 releases since then. Justin [1] https://issues.apache.org/jira/browse/ARTEMIS-1764 [2] https://activemq.apache.org/components/artemis/download/ [3] https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz [4] https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5 On Wed, May 11, 2022 at 5:35 PM Roy Cohen <[email protected]> wrote: Hi All We have an urgent situation with our Artemis brokers pair in production which somehow has left 01 without any DLQs and 02 in a state where it’s not starting up. The one which isn’t starting up is throwing this Exception, unfortunately I haven’t been able to find much online. Any help would be greatly appreciated 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server] (main) AMQ224000: Failure in initialisation: java.lang.IllegalStateException: Cannot find queue with id 80 at org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427) [artemis-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394) [artemis-jms-server-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65) [artemis-cli-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83) [artemis-cli-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110) [artemis-cli-1.3.0.jar:1.3.0] at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72) [artemis-cli-1.3.0.jar:1.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_191] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191] at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122) [artemis-boot.jar:1.3.0] at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45) [artemis-boot.jar:1.3.0]
