Hi Justin,
Thanks for the response. I will try that example. Yes, I am suspecting that
it is an environmental issue too. I am running the broker instances in
Kubernetes pods on two different servers. I have an operator pod that creates
a statefullSet consisting of the master and slave brokers. The pods start up in
master first (ss-0) and then the slave is started (ss-1). I delete the master
pod and the slave goes active. The failover to the slave works as expected. My
operator pod sees that the master broker pod has gone away and then starts it
up. I think this is where the issue is. The master establishes the cluster
connection to the slave (which is now the active broker) but it does not seem
to interact with the slave instance to tell it to no longer be active. The
master and slave instances reestablish the cluster connection and setup the
queues associated with the cluster connections. I also see some core messages
go to the slave, but I not sure what they represent. The master instance just
completes its startup and goes active. Thus creating two active broker
instances, in essence a split-brain condition. I have turned on logging as such
<broker-plugins>
<broker-plugin
class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
I have also edited the log4j2.properties file inside the Kubernetes pod and set
the logger.artemis_server.level=INFO property to TRACE and have copious amount
of log data. What methods would be involved in the startup of the master
server that would stage the fail-back? That is what is the expected path
through the code during startup to have a successful fail-back. With that info
I may be able to trace what is derailing the fail-back. Would the trace output
be any use to you to determine what is going on?
Would the broker.xml be able to give you enough to attempt to recreate the
issue? I will attach the master one. The slave only differs by the ha-policy
config I posted earlier.
I plan to run the example you pointed out on two VM servers without Kubernetes
being involved to see if I can isolate the issue to a Kubernetes pod setup
issue?
Thanks Paul
kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- cat
/home/jboss/amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xmlns:xi=http://www.w3.org/2001/XInclude
xsi:schemaLocation="urn:activemq
/schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following
configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak
it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for
performance considerations.
<journal-retention-directory period="7" unit="DAYS"
storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on
the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the
<network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>>
-->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is
meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially
visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful
ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d
%s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1
%2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and
notifications -->
<connector
name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the
disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the
connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This
is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for
the global-max-size.
You may specify a different value here if you need to customize it to
your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address
mode.
if global-max-size is specified the full address mode will be
specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system
(Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at
amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection,
set this to false
as duplicate detection requires
applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered
large, so we start using files to hold their data.
default: 102400, -1 would mean to
disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or
Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to
the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for
more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor
name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor
name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<jgroups-file>jgroups-ping.xml</jgroups-file>
<jgroups-channel>activemq_broadcast_channel</jgroups-channel>
<connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<jgroups-file>jgroups-ping.xml</jgroups-file>
<jgroups-channel>activemq_broadcast_channel</jgroups-channel>
<refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>artemis</connector-ref>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>2</retry-interval-multiplier>
<max-retry-interval>32000</max-retry-interval>
<initial-connect-attempts>20</initial-connect-attempts>
<reconnect-attempts>-1</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops> <discovery-group-ref
discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be
auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the
system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in
memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue.
-->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin
class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin
class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/>
</metrics>
</core>
</configuration>
From: Justin Bertram <[email protected]>
Date: Wednesday, December 13, 2023 at 3:23 PM
To: [email protected] <[email protected]>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
ActiveMQ Artemis ships with an example in the
examples/features/ha/replicated-failback directory that demonstrates how
this works. I ran this example and it worked fine. I also started up the
brokers manually and triggered failover and failback. Again, everything
worked fine. My only guess is that there's something configured wrong or
perhaps there's an environmental issue that's preventing failback from
working properly for you. If you could provide clear steps on how to
reproduce this issue I could investigate further.
Justin
On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <[email protected]>
wrote:
> Hi,
>
> I am having an issue with failback to the master broker. I start the
> master broker then the slave broker. I see the replication from the master
> to the slave happen. I stop the master/active broker and a failover happens
> to the slave as expected. When the master is started a failback to it is
> expected but does not happen. The following is my ha-policy config. Did I
> miss something in the ha-policy configuration? We are using version 2.28.0
> and java17.
>
> Thanks,
> Paul
>
>
> Master broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
> Slave broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> The following are the logs from the master and slave brokers
> HA “master” broker log after restart
> 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> Starting embedded web server
> 2023-12-08 20:42:57,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 15, result: NO_BINDINGS
> 2023-12-08 20:42:57,627 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3
> 2023-12-08 20:42:57,657 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>,
> local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> activemq-branding plugin
> 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> artemis-plugin plugin
> 2023-12-08 20:42:57,766 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 18, result: NO_BINDINGS
> 2023-12-08 20:42:57,767 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 20, result: NO_BINDINGS
> 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising
> hawtio services
> 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration
> will be discovered via system properties
> 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> Hawtio 2.15.0
> 2023-12-08 20:42:57,975 INFO
> [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> role principal classes:
> "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> servlet is disabled
> 2023-12-08 20:42:57,989 INFO
> [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> property: [key=policyLocation,
> value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> HTTP Server started at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> Artemis Jolokia REST API available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> >
> 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console
> available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> 2023-12-08 20:42:59,524 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> 2023-12-08 20:42:59,535 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 22, result: NO_BINDINGS
> 2023-12-08 20:42:59,536 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,549 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 24, result: NO_BINDINGS
> 2023-12-08 20:42:59,550 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,583 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: OK
> 2023-12-08 20:42:59,584 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@5f46053b
> 2023-12-08 20:42:59,593 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
> 2023-12-08 20:42:59,594 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 28, result: OK
> 2023-12-08 20:42:59,616 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 29, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:59,615 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> with transaction: null
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 31, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 32, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 35, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 39, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:43:08,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> 2023-12-08 20:43:08,464 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
>
>
> HA “slave” broker log.
>
> 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
>
> 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> AMQ221031: backup announced
> 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221084: Requested 0 quorum votes
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221083: ignoring quorum vote as max cluster size is 1.
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221071: Failing over based on quorum vote results.
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
> 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:51,946 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:56,977 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2023-12-08 20:42:57,445 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: NO_BINDINGS
> 2023-12-08 20:42:57,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6
> 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
> 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is unable to connect to destination.
> Retrying
> 2023-12-08 20:42:57,629 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> 2023-12-08 20:42:57,630 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> 2023-12-08 20:42:57,648 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 29, result: NO_BINDINGS
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 31, result: NO_BINDINGS
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,708 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 33, result: OK
> 2023-12-08 20:42:57,709 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@3645037f
> 2023-12-08 20:42:57,719 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 35, result: OK
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 36, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:57,753 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> with transaction: null
> 2023-12-08 20:42:57,760 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 38, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> with transaction: null
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 41, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 43, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 44, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 46, result: OK
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 48, result: OK
> 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
>
>
>
