Hi Justin,
I tried the example and it works. I turned up logging to get a success path
through the code to maybe see where things get de-railed. Is it possible to
use jgroups instead of UDP for the cluster connection in the example? That
would give me a better success signature.
My standby instance has this in the broker.xml which is the same as in the
example:
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
I am seeing this series of log messages that suggests that failback is not
enabled. Is the attemptFailback=false the default at this point in startup and
it is changed later or it there something gone wrong at this point?
2024-01-11 18:24:14,066 TRACE
[org.apache.activemq.artemis.core.client.impl.Topology]
Topology@7e0a913c[owner=ServerLocatorImpl
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
QuorumManager(server=null) about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434
connector = Pair[a=TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
b=null]
2024-01-11 18:24:14,067 TRACE
[org.apache.activemq.artemis.core.client.impl.Topology]
Topology@7e0a913c[owner=ServerLocatorImpl
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<mailto:org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048>
about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
Pair[a=TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
b=null]
2024-01-11 18:24:14,067 DEBUG
[org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
ClientSessionFactoryImpl received backup update for live/backup pair =
TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
/ null but it didn't belong to TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
2024-01-11 18:24:14,067 TRACE
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
Cluster Connected
2024-01-11 18:24:14,068 DEBUG
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
Starting backup manager
2024-01-11 18:24:14,069 DEBUG
[org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy backup
config ClusterConnectionConfiguration{name='my-cluster', address='',
connectorName='artemis', clientFailureCheckPeriod=30000, connectionTTL=60000,
retryInterval=1000, retryIntervalMultiplier=2.0, maxRetryInterval=32000,
initialConnectAttempts=20, reconnectAttempts=-1, callTimeout=30000,
callFailoverTimeout=-1, duplicateDetection=true,
messageLoadBalancingType=ON_DEMAND, compositeMembers=null, staticConnectors=[],
discoveryGroupName='my-discovery-group', maxHops=1,
confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
minLargeMessageSize=102400, clusterNotificationInterval=1000,
clusterNotificationAttempts=2}
2024-01-11 18:24:14,081 DEBUG
[org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
BackupManager connecting to DiscoveryBackupConnector
[group=DiscoveryGroupConfiguration{name='my-discovery-group',
refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
2024-01-11 18:24:14,082 DEBUG
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
Starting Backup Server
2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] started,
waiting live to fail before it gets active
2024-01-11 18:24:14,083 TRACE
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
Setting server state as started
2024-01-11 18:24:14,083 TRACE
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
looking up the node through nodeLocator.locateNode()
2024-01-11 18:24:14,083 DEBUG
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
Connecting towards a possible live, connection
information=Pair[a=TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
2024-01-11 18:24:14,083 DEBUG
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
2024-01-11 18:24:14,083 TRACE
[org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
Calling
clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
2024-01-11 18:24:14,083 TRACE
[org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
java.lang.Exception: trace
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:846)
~[artemis-core-client-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:252)
~[artemis-core-client-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:610)
~[artemis-core-client-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:628)
~[artemis-core-client-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(ClusterController.java:313)
~[artemis-server-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(SharedNothingBackupActivation.java:348)
~[artemis-server-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:199)
~[artemis-server-2.28.0.jar:2.28.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4473)
~[artemis-server-2.28.0.jar:2.28.0]
2024-01-11 18:24:14,084 DEBUG
[org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] Trying
reconnection attempt 0/0
2024-01-11 18:24:14,084 DEBUG
[org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] Trying
to connect with
connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<mailto:connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18>
and currentConnectorConfig: TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
2024-01-11 18:24:14,084 DEBUG
[org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] Connector
NettyConnector
[host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using
native epoll
2024-01-11 18:24:14,084 DEBUG
[org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] Started
EPOLL Netty Connector version 4.1.86.Final to
cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
2024-01-11 18:24:14,084 DEBUG
[org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] Remote
destination:
cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/10.44.0.25:61616
Thanks,
Paul
From: Justin Bertram <jbert...@apache.org>
Date: Wednesday, December 13, 2023 at 3:23 PM
To: users@activemq.apache.org <users@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
ActiveMQ Artemis ships with an example in the
examples/features/ha/replicated-failback directory that demonstrates how
this works. I ran this example and it worked fine. I also started up the
brokers manually and triggered failover and failback. Again, everything
worked fine. My only guess is that there's something configured wrong or
perhaps there's an environmental issue that's preventing failback from
working properly for you. If you could provide clear steps on how to
reproduce this issue I could investigate further.
Justin
On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <paul.shie...@hpe.com>
wrote:
> Hi,
>
> I am having an issue with failback to the master broker. I start the
> master broker then the slave broker. I see the replication from the master
> to the slave happen. I stop the master/active broker and a failover happens
> to the slave as expected. When the master is started a failback to it is
> expected but does not happen. The following is my ha-policy config. Did I
> miss something in the ha-policy configuration? We are using version 2.28.0
> and java17.
>
> Thanks,
> Paul
>
>
> Master broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
> Slave broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> The following are the logs from the master and slave brokers
> HA “master” broker log after restart
> 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> Starting embedded web server
> 2023-12-08 20:42:57,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 15, result: NO_BINDINGS
> 2023-12-08 20:42:57,627 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3
> 2023-12-08 20:42:57,657 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>,
> local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> activemq-branding plugin
> 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> artemis-plugin plugin
> 2023-12-08 20:42:57,766 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 18, result: NO_BINDINGS
> 2023-12-08 20:42:57,767 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 20, result: NO_BINDINGS
> 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising
> hawtio services
> 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration
> will be discovered via system properties
> 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> Hawtio 2.15.0
> 2023-12-08 20:42:57,975 INFO
> [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> role principal classes:
> "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> servlet is disabled
> 2023-12-08 20:42:57,989 INFO
> [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> property: [key=policyLocation,
> value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> HTTP Server started at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> Artemis Jolokia REST API available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> >
> 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console
> available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> 2023-12-08 20:42:59,524 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> 2023-12-08 20:42:59,535 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 22, result: NO_BINDINGS
> 2023-12-08 20:42:59,536 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,549 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 24, result: NO_BINDINGS
> 2023-12-08 20:42:59,550 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,583 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: OK
> 2023-12-08 20:42:59,584 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@5f46053b
> 2023-12-08 20:42:59,593 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
> 2023-12-08 20:42:59,594 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 28, result: OK
> 2023-12-08 20:42:59,616 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 29, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:59,615 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> with transaction: null
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 31, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 32, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 35, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 39, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:43:08,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> 2023-12-08 20:43:08,464 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
>
>
> HA “slave” broker log.
>
> 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
>
> 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> AMQ221031: backup announced
> 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221084: Requested 0 quorum votes
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221083: ignoring quorum vote as max cluster size is 1.
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221071: Failing over based on quorum vote results.
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
> 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:51,946 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:56,977 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2023-12-08 20:42:57,445 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: NO_BINDINGS
> 2023-12-08 20:42:57,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6
> 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
> 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is unable to connect to destination.
> Retrying
> 2023-12-08 20:42:57,629 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> 2023-12-08 20:42:57,630 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> 2023-12-08 20:42:57,648 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 29, result: NO_BINDINGS
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 31, result: NO_BINDINGS
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,708 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 33, result: OK
> 2023-12-08 20:42:57,709 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@3645037f
> 2023-12-08 20:42:57,719 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 35, result: OK
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 36, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:57,753 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> with transaction: null
> 2023-12-08 20:42:57,760 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 38, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> with transaction: null
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 41, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 43, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 44, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 46, result: OK
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 48, result: OK
> 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
>
>
>
