> Is it possible to use jgroups instead of UDP for the cluster connection
in the example?
Yes. Check out examples/features/clustered/clustered-jgroups for an example
configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
initial discovery. Once brokers are discovered then normal TCP connections
are created.
Before I continue looking through your logs, etc. can you confirm that you
resolved the journal persistence issue I described in my previous email? If
that's not resolved there's no chance for failback to occur.
Justin
On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <paul.shie...@hpe.com>
wrote:
> Hi Justin,
>
> I tried the example and it works. I turned up logging to get a success
> path through the code to maybe see where things get de-railed. Is it
> possible to use jgroups instead of UDP for the cluster connection in the
> example? That would give me a better success signature.
>
> My standby instance has this in the broker.xml which is the same as in the
> example:
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> I am seeing this series of log messages that suggests that failback is not
> enabled. Is the attemptFailback=false the default at this point in startup
> and it is changed later or it there something gone wrong at this point?
>
> 2024-01-11 18:24:14,066 TRACE
> [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c[owner=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> QuorumManager(server=null) about node up =
> 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null]
>
> 2024-01-11 18:24:14,067 TRACE
> [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c[owner=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048>
> about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null]
>
> 2024-01-11 18:24:14,067 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> ClientSessionFactoryImpl received backup update for live/backup pair =
> TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> / null but it didn't belong to TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
>
> 2024-01-11 18:24:14,067 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Cluster Connected
>
> 2024-01-11 18:24:14,068 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting backup manager
>
> 2024-01-11 18:24:14,069 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> backup config ClusterConnectionConfiguration{name='my-cluster', address='',
> connectorName='artemis', clientFailureCheckPeriod=30000,
> connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> minLargeMessageSize=102400, clusterNotificationInterval=1000,
> clusterNotificationAttempts=2}
>
> 2024-01-11 18:24:14,081 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
>
> 2024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
>
> java.lang.Exception: trace
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:846)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:252)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:610)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:628)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(ClusterController.java:313)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(SharedNothingBackupActivation.java:348)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:199)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4473)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> Trying reconnection attempt 0/0
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> Trying to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:connectorFactory
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18>
> and currentConnectorConfig: TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Connector NettyConnector
> [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using native epoll
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Started EPOLL Netty Connector version 4.1.86.Final to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Remote destination:
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.25:61616
>
>
> Thanks,
> Paul
>
> From: Justin Bertram <jbert...@apache.org>
> Date: Wednesday, December 13, 2023 at 3:23 PM
> To: users@activemq.apache.org <users@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> ActiveMQ Artemis ships with an example in the
> examples/features/ha/replicated-failback directory that demonstrates how
> this works. I ran this example and it worked fine. I also started up the
> brokers manually and triggered failover and failback. Again, everything
> worked fine. My only guess is that there's something configured wrong or
> perhaps there's an environmental issue that's preventing failback from
> working properly for you. If you could provide clear steps on how to
> reproduce this issue I could investigate further.
>
>
> Justin
>
> On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <paul.shie...@hpe.com
> >
> wrote:
>
> > Hi,
> >
> > I am having an issue with failback to the master broker. I start the
> > master broker then the slave broker. I see the replication from the
> master
> > to the slave happen. I stop the master/active broker and a failover
> happens
> > to the slave as expected. When the master is started a failback to it is
> > expected but does not happen. The following is my ha-policy config.
> Did I
> > miss something in the ha-policy configuration? We are using version
> 2.28.0
> > and java17.
> >
> > Thanks,
> > Paul
> >
> >
> > Master broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <master>
> >
> > <check-for-live-server>true</check-for-live-server>
> >
> > </master>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > Slave broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > The following are the logs from the master and slave brokers
> > HA “master” broker log after restart
> > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > Starting embedded web server
> > 2023-12-08 20:42:57,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 15, result: NO_BINDINGS
> > 2023-12-08 20:42:57,627 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3
> > 2023-12-08 20:42:57,657 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f>,
> > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > activemq-branding plugin
> > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > artemis-plugin plugin
> > 2023-12-08 20:42:57,766 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 18, result: NO_BINDINGS
> > 2023-12-08 20:42:57,767 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 20, result: NO_BINDINGS
> > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> Initialising
> > hawtio services
> > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> Configuration
> > will be discovered via system properties
> > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > Hawtio 2.15.0
> > 2023-12-08 20:42:57,975 INFO
> > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> > role principal classes:
> > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > servlet is disabled
> > 2023-12-08 20:42:57,989 INFO
> > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > property: [key=policyLocation,
> > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > HTTP Server started at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > Artemis Jolokia REST API available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> >
> > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> Console
> > available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > 2023-12-08 20:42:59,524 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > 2023-12-08 20:42:59,535 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 22, result: NO_BINDINGS
> > 2023-12-08 20:42:59,536 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,549 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 24, result: NO_BINDINGS
> > 2023-12-08 20:42:59,550 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,583 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: OK
> > 2023-12-08 20:42:59,584 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@5f46053b
> > 2023-12-08 20:42:59,593 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > 2023-12-08 20:42:59,594 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 28, result: OK
> > 2023-12-08 20:42:59,616 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:59,615 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> > with transaction: null
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 31, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 32, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 35, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 39, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:43:08,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > 2023-12-08 20:43:08,464 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> >
> >
> > HA “slave” broker log.
> >
> > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > synchronized with live server,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> >
> > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221031: backup announced
> > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221084: Requested 0 quorum votes
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221071: Failing over based on quorum vote results.
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:51,946 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:56,977 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> destroyed
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212004: Failed to connect to server.
> > 2023-12-08 20:42:57,445 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: NO_BINDINGS
> > 2023-12-08 20:42:57,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6
> > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > Retrying
> > 2023-12-08 20:42:57,629 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > 2023-12-08 20:42:57,630 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > 2023-12-08 20:42:57,648 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 29, result: NO_BINDINGS
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 31, result: NO_BINDINGS
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,708 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 33, result: OK
> > 2023-12-08 20:42:57,709 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@3645037f
> > 2023-12-08 20:42:57,719 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 35, result: OK
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:57,753 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> > with transaction: null
> > 2023-12-08 20:42:57,760 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 38, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > with transaction: null
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 41, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 43, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 44, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 46, result: OK
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 48, result: OK
> > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> >
> >
> >
>
