Because bin/artemis includes references to the jboss logmanager causing artemis
to fail on startup
Diffing my two instances I see:
# Set Defaults Properties
ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
# finding the Log Manager
LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
if [ -z "$LOG_MANAGER" ] ; then
# this is the one found when the server was created
LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
fi
WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
if [ -z "$WILDFLY_COMMON" ] ; then
# this is the one found when the server was created
WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
fi
-Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
and
-Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
-Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
all have to go.
I also see a change in the schema for bootstrap.xml (binding child of web), and
the browse permission added to management.xml
Along with the new log4j.properties you mentioned
Some of those changes might be earlier if they’re backwards compatible, I’ve
carried forward this configuration for awhile, but in particular the shell
script changes appear to be manditory.
From: Clebert Suconic <clebert.suco...@gmail.com>
Date: Thursday, October 13, 2022 at 12:49 PM
To: users@activemq.apache.org <users@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
I think the record would pile up unacked at the source mirror.
and @Stephen baker: sorry about my mistake on this fix...
Why would the upgrade be difficult on 2.27? it's just adding a
log4j2.properties.. everything else should be the same.
You should probably bring a patched version yourself until we can make
a release? I'm thinking we should make a release next week.
On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
<stephen.ba...@rmssoftwareinc.com> wrote:
>
> Your patch does resolve the error. Artemis 2.27 looks like it will be a
> difficult upgrade, I ended up making a new instance and merging config over.
>
> I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
>
> What is the impact of this issue? I’m trying to decide whether to advise our
> IT team to continue with the planned upgrade or hold off until 2.27. We will
> definitely encounter this condition in production. From a surface reading,
> possibly a resource leak?
>
>
>
>
> From: Clebert Suconic <clebert.suco...@gmail.com>
> Date: Wednesday, October 12, 2022 at 9:54 PM
> To: users@activemq.apache.org <users@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> Notice that main is now using SLF4j / log4j... (in case you manually
> upgrade to a snapshot)
>
>
> We are still working the details for an upgrade.
>
>
> if you need to patch your 2.25.0 it's a straight change to make there
>
> On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> <clebert.suco...@gmail.com> wrote:
> >
> > I don't know how I would test it yet. It's fairly late in the night
> > for me.. I will think about it tomorrow.
> >
> >
> > but here is a tentative fix:
> >
> > https://github.com/apache/activemq-artemis/pull/4256
> >
> > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > <stephen.ba...@rmssoftwareinc.com> wrote:
> > >
> > > That’s the full output with regular logging levels. I can reproduce at
> > > will so I have enabled trace level logging and pasted the result in
> > > https://issues.apache.org/jira/browse/ARTEMIS-4045
> > >
> > > Let’s take further discussion there?
> > >
> > > From: Clebert Suconic <clebert.suco...@gmail.com>
> > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > To: users@activemq.apache.org <users@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > is this the actual trace? or you cut some to post here?
> > >
> > >
> > > Just puzzled by skipDelivery calling performAck..
> > >
> > > artemis-test-artemis-1-m-1 | at
> > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at
> > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > [artemis-server-2.25.0.jar:2.25.0]
> > >
> > >
> > > can you post the full stack if this is not it?
> > >
> > >
> > > it definitely needs fixing... I'm investigating it.
> > >
> > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > <stephen.ba...@rmssoftwareinc.com> wrote:
> > > >
> > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is
> > > > it a concern that warrants further investigation?
> > > >
> > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR
> > > > [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver:
> > > > java.lang.IllegalStateException: this method requires to be called
> > > > within the handler, use the executor
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
> > > > [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> > > > [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
> > > > [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at
> > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > [java.base:]
> > > > artemis-test-artemis-1-m-1 | at
> > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > [java.base:]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > > > [artemis-commons-2.25.0.jar:]
> > > >
> > > >
> > > > From: Stephen Baker <stephen.ba...@rmssoftwareinc.com>
> > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > To: users@activemq.apache.org <users@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > I set up some docker images in this configuration as a preliminary
> > > > step. One oddity:
> > > >
> > > > Configure 2.25 side not to run the reaper
> > > > Send message to 2.25 side
> > > > Observe that after expiry the message shows up in the expiry queue on
> > > > the 2.20 side, but not on the 2.25 side, the message is removed from
> > > > the original queue on both sides.
> > > >
> > > > If the message is originally sent to the 2.20 side it shows up in both
> > > > queues as expected.
> > > >
> > > > There’s probably a reason for it, but I didn’t expect this change. I
> > > > thought that we would continue to see the old bugs until both sides
> > > > were updated.
> > > >
> > > > From: Clebert Suconic <clebert.suco...@gmail.com>
> > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > To: users@activemq.apache.org <users@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > Yeah.. something like that... not necessarily in there though. but a
> > > > similar test.
> > > >
> > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > <stephen.ba...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > Ok, I agree based on a cursory reading of that patch. The extra
> > > > > ackReason defaults to normal in one direction and isn’t read in the
> > > > > other direction. Killed, replaced, and expired being interpreted as
> > > > > normal just means that the 2.20 bugs will persist until both sides
> > > > > are updated.
> > > > >
> > > > > I’ll test it out with different version docker containers. I suppose
> > > > > as far as writing tests you mean something like the
> > > > > MultiVersionReplicaTest.
> > > > >
> > > > > Stephen E. Baker
> > > > >
> > > > >
> > > > > From: Clebert Suconic <clebert.suco...@gmail.com>
> > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > To: users@activemq.apache.org <users@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > In theory it should work.
> > > > >
> > > > >
> > > > > Only change that might break compatibility is
> > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > >
> > > > >
> > > > > I tried to not break compatibility, but I just realized we should add
> > > > > a test to validate compatibility between mirrors.
> > > > >
> > > > >
> > > > >
> > > > > so, I will say it should be compatible, but I would test it before
> > > > > doing it in the real system.
> > > > >
> > > > >
> > > > >
> > > > > if you are willing to contribute to a compatibility test :)
> > > > >
> > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > <stephen.ba...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > We are planning our production upgrade from 2.20 to 2.25. These
> > > > > > upgrades involve a loss of service in the window between stopping
> > > > > > the live and when the backup instance becomes ready to process
> > > > > > messages.
> > > > > >
> > > > > > I was wondering if the mirror protocol is expected to be compatible
> > > > > > between those versions. If so we could upgrade our cold site, and
> > > > > > then wait for a planned failover to avoid any additional down time.
> > > > > > I know that quite a bit of work was done by Clebert in 2.24 so I
> > > > > > was hoping he could weigh in.
> > > > > >
> > > > > > Stephen E Baker
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > > > > Do not click links or open attachments unless you recognize the
> > > > > sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > > > Do not click links or open attachments unless you recognize the sender
> > > > and know the content is safe.
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > > > Do not click links or open attachments unless you recognize the sender
> > > > and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> > > not click links or open attachments unless you recognize the sender and
> > > know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not
> click links or open attachments unless you recognize the sender and know the
> content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not
click links or open attachments unless you recognize the sender and know the
content is safe.
