This was caused by ARTEMIS-4582 [1]. I've opened ARTEMIS-4742 [2] and sent a PR [3] to address the problem.
To be clear, this has nothing to do with replication or load/throughput. It's just a buffer formatting issue caused by a configuration change. Justin [1] https://issues.apache.org/jira/browse/ARTEMIS-4582 [2] https://issues.apache.org/jira/browse/ARTEMIS-4742 [3] https://github.com/apache/activemq-artemis/pull/4904 On Thu, Apr 18, 2024 at 7:15 AM MILOVIDOV Aleksandr <aleksandr.milovi...@raiffeisen.ru.invalid> wrote: > Hi All, > > We have faced problem when upgrading one of ActiveMQ Artemis servers from > 2.32.0 to 2.33.0. > Artemis cannot start and writes an error to the log: > > 2024-04-18 13:31:44,975 ERROR [org.apache.activemq.artemis.core.server] > AMQ224000: Failure in initialisation > java.lang.IndexOutOfBoundsException: readerIndex(82) + length(1) exceeds > writerIndex(82): UnpooledHeapByteBuf(ridx: 82, widx: 82, cap: 82/82) > at > io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) > ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final] > at > io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730) > ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final] > at > io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529) > ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final] > at > org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:150) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:79) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.persistence.config.PersistedSecuritySetting.decode(PersistedSecuritySetting.java:255) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newSecurityRecord(AbstractJournalStorageManager.java:2127) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.lambda$loadBindingJournal$5(AbstractJournalStorageManager.java:1640) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList$SparseArray.clear(SparseArrayLinkedList.java:114) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList.clearSparseArrayList(SparseArrayLinkedList.java:173) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList.clear(SparseArrayLinkedList.java:227) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1613) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:3856) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:3489) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.server.impl.ReplicationPrimaryActivation.run(ReplicationPrimaryActivation.java:145) > [artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:738) > [artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:628) > [artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66) > [artemis-cli-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) > [artemis-cli-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221) > [artemis-cli-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) > [artemis-cli-2.33.0.jar:2.33.0] > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) > ~[?:?] > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ~[?:?] > at > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:157) > [artemis-boot.jar:2.33.0] > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) > [artemis-boot.jar:2.33.0] > > and warning: > > 2024-04-18 13:31:45,977 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator > is closed (maybe it was garbage collected) > java.lang.IllegalStateException: Server locator is closed (maybe it was > garbage collected) > at > org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1906) > ~[artemis-core-client-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:678) > ~[artemis-core-client-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:579) > ~[artemis-core-client-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:558) > ~[artemis-core-client-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:501) > ~[artemis-server-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > ~[artemis-commons-2.33.0.jar:2.33.0] > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > [?:?] > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > [?:?] > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > [artemis-commons-2.33.0.jar:2.33.0] > > It can start with new version only if we remove all existing data from > Artemis instance data directory. > Also we have exported all data from this non-working instance with version > 2.33.0 using artemis data exp utility, and imported to another instance > with version 2.33.0, and it has been started successfully. > Artemis is configured using primary/backup replication with pluggable > quorum. > We have also successfully upgraded several other instances of Artemis > which has smaller number of producers/consumers and lower throughput. > > -- > Best regards, > Aleksandr > > > > ----------------------------------- > > This message and any attachment are confidential and may be privileged or > otherwise protected from disclosure. If you are not the intended recipient > any use, distribution, copying or disclosure is strictly prohibited. If you > have received this message in error, please notify the sender immediately > either by telephone or by e-mail and delete this message and any attachment > from your system. Correspondence via e-mail is for information purposes > only. AO Raiffeisenbank neither makes nor accepts legally binding > statements by e-mail unless otherwise agreed. > > ----------------------------------- >
