Hi Justin, A follow-on question about persistent data and failover. The documentation makes this statement. “When using replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network. Therefore all (persistent) data received by the live server will be duplicated to the backup.” [1]. We are using the MQTT protocol. We have published messages that have the retained flag set. We also use “last will” messages and set the lastWillRetain flag to true. When the last will message is sent by the broker the retained message for the publisher is updated with the last will message. Now for the question. Are last will messages considered persistent data and are duplicated to the backup or is the retained message updated with the last will message duplicated to the backup broker? We seem to be losing last will messages on a broker failover (ie the retained message does not contain the last will message).
Thanks, Paul [1]: https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies From: Justin Bertram <jbert...@apache.org> Date: Friday, January 19, 2024 at 11:47 AM To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Trouble with Replication HA Master/Slave config performing failback > So even though the persistent data will be stored pod-local... If you disable persistence on the broker then no data will be persisted on the local pod. > ...enabling persistence has the knock-on effect of also making it available > to the backup server in addition to whatever the “replication” model does? Only persistent data is replicated to the backup. This is noted in the documentation [1]. To be clear, this is not "in addition" to what replication is doing. The replication manager "listens" for persisted data and that's the information that it sends to the backup. If you disable persistence then no data is replicated. Generally speaking, client applications make a basic decision about the messages they send - persistent or non-persistent. Sending persistent messages comes with certain reliability guarantees but worse performance. Sending non-persistent messages comes with better performance but less reliability. The ability to survive a failure is one of the reliability benefits enjoyed by persistent messages. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies> On Fri, Jan 19, 2024 at 10:50 AM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: Hi Justin, This does seem counter-intuitive. We’re using replication, not shared storage. So even though the persistent data will be stored pod-local, enabling persistence has the knock-on effect of also making it available to the backup server in addition to whatever the “replication” model does? john [rg]<https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> Sent: Friday, January 19, 2024 9:42 AM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Trouble with Replication HA Master/Slave config performing failback *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** > ...do we currently set persistenceEnable=false? I would expect there’s no > reason for setting it true, since the pod-local storage will evaporate with > the pod. You definitely want to enable persistence because only persistent data will be available on the backup. Justin On Fri, Jan 19, 2024 at 10:21 AM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: Hi Paul, Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us? Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod. Thanks John Error! Filename not specified.<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Shields, Paul Michael <paul.shie...@hpe.com<mailto:paul.shie...@hpe.com>> Sent: Friday, January 19, 2024 8:51 AM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Trouble with Replication HA Master/Slave config performing failback *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** Hi John, The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files. the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use. Paul From: John Lilley <john.lil...@redpointglobal.com.INVALID<mailto:john.lil...@redpointglobal.com.INVALID>> Date: Thursday, January 18, 2024 at 5:49 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> <users@activemq.apache.org<mailto:users@activemq.apache.org>> Subject: RE: Trouble with Replication HA Master/Slave config performing failback Hi Justin, Can you elaborate on the lock file in the <node-manager-lock-directory>? Does it need to be specified in the broker.xml for both the master and the slave? Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election? Thanks john Error! Filename not specified.<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> Sent: Monday, January 15, 2024 9:56 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Trouble with Replication HA Master/Slave config performing failback *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV. Justin On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <lino.pere...@redpointglobal.com.invalid<mailto:lino.pere...@redpointglobal.com.invalid>> wrote: Hi Justin, Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case? Thanks, Lino Error! Filename not specified.<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1> Lino Pereira C++ Developer, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 lino.pere...@redpointglobal.com<mailto:lino.pere...@redpointglobal.com> -----Original Message----- From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> Sent: Friday, January 12, 2024 4:53 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Trouble with Replication HA Master/Slave config performing failback *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** I sent that email on December 15. You can find it on the web-based mailing list interface [1]. Justin [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <paul.shie...@hpe.com<mailto:paul.shie...@hpe.com>> wrote: > Hi Justin, > > Forgive me but I don’t remember a journal persistence issue discussion. I > did an email search too and did not see that discussion either. Would you > please refresh me on the discussion? I may have inadvertently deleted that? > > I do see data being replicated to my production slave/standby instance on > initial startup of the two broker instances. Failover works as expected. > But when the primary instance starts backup I do not see the active slave > replicate to the starting primary and I do not see the active slave restart > and go to the backup server state. > > What is the communication sequence when the primary starts up after it > quit with the active backup broker? In other words, how is the active > backup broker notified to start replication to the newly started primary > broker? > > I ran the replicated-failback example again with TRACE level logging > turned on and I do see the attemptFailback=false message in the > target/server1/log/artemis.log file at startup, so it must be changed later > or am I not understanding the purpose of this flag? > > 2024-01-12 14:40:18,341 DEBUG > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ****** > BackupManager connecting to DiscoveryBackupConnector > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000, > discoveryInitialWaitTimeout=10000}] > > 2024-01-12 14:40:18,341 DEBUG > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Starting Backup Server > > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server] > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] > started, waiting live to fail before it gets active > > 2024-01-12 14:40:18,341 TRACE > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Setting server state as started > > 2024-01-12 14:40:18,341 TRACE > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > looking up the node through nodeLocator.locateNode() > > 2024-01-12 14:40:18,341 DEBUG > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Connecting towards a possible live, connection > information=Pair[a=TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=localhost, b=null], > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04 > > 2024-01-12 14:40:18,341 DEBUG > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04 > > 2024-01-12 14:40:18,341 TRACE > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Calling > clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=localhost) > > From my production slave broker log > > > 024-01-11 18:24:14,082 DEBUG > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Starting Backup Server > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server] > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] > started, waiting live to fail before it gets active > > 2024-01-11 18:24:14,083 TRACE > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Setting server state as started > > 2024-01-11 18:24:14,083 TRACE > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > looking up the node through nodeLocator.locateNode() > > 2024-01-11 18:24:14,083 DEBUG > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Connecting towards a possible live, connection > information=Pair[a=TransportConfiguration(name=artemis, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434 > > 2024-01-11 18:24:14,083 DEBUG > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434 > > 2024-01-11 18:24:14,083 TRACE > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > Calling > clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local) > > Thanks, > Paul > From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> > Date: Friday, January 12, 2024 at 3:38 PM > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > <users@activemq.apache.org<mailto:users@activemq.apache.org>> > Subject: Re: Trouble with Replication HA Master/Slave config performing > failback > > Is it possible to use jgroups instead of UDP for the cluster connection > in the example? > > Yes. Check out examples/features/clustered/clustered-jgroups for an example > configuration. Keep in mind that UDP, JGroups, etc. is only used for > initial discovery. Once brokers are discovered then normal TCP connections > are created. > > Before I continue looking through your logs, etc. can you confirm that you > resolved the journal persistence issue I described in my previous email? If > that's not resolved there's no chance for failback to occur. > > > Justin > > > > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael < > paul.shie...@hpe.com<mailto:paul.shie...@hpe.com>> > wrote: > > > Hi Justin, > > > > I tried the example and it works. I turned up logging to get a success > > path through the code to maybe see where things get de-railed. Is it > > possible to use jgroups instead of UDP for the cluster connection in the > > example? That would give me a better success signature. > > > > My standby instance has this in the broker.xml which is the same as in > the > > example: > > > > <ha-policy> > > > > <replication> > > > > <slave> > > > > <allow-failback>true</allow-failback> > > > > </slave> > > > > </replication> > > > > </ha-policy> > > > > I am seeing this series of log messages that suggests that failback is > not > > enabled. Is the attemptFailback=false the default at this point in > startup > > and it is changed later or it there something gone wrong at this point? > > > > 2024-01-11 18:24:14,066 TRACE > > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c > [owner=ServerLocatorImpl > > [initialConnectors=[TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group', > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing > > QuorumManager(server=null) about node up = > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector = > > Pair[a=TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > > b=null] > > > > 2024-01-11 18:24:14,067 TRACE > > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c > [owner=ServerLocatorImpl > > [initialConnectors=[TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group', > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing > > > org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<mailto:org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048> > > <mailto: <mailto:%0b>> > > org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<mailto:org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048> > > > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector = > > Pair[a=TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > > b=null] > > > > 2024-01-11 18:24:14,067 DEBUG > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] > > ClientSessionFactoryImpl received backup update for live/backup pair = > > TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local > > / null but it didn't belong to TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local > > > > 2024-01-11 18:24:14,067 TRACE > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > Cluster Connected > > > > 2024-01-11 18:24:14,068 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > Starting backup manager > > > > 2024-01-11 18:24:14,069 DEBUG > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy > > backup config ClusterConnectionConfiguration{name='my-cluster', > address='', > > connectorName='artemis', clientFailureCheckPeriod=30000, > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0, > > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1, > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true, > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null, > > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1, > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false, > > minLargeMessageSize=102400, clusterNotificationInterval=1000, > > clusterNotificationAttempts=2} > > > > 2024-01-11 18:24:14,081 DEBUG > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ****** > > BackupManager connecting to DiscoveryBackupConnector > > [group=DiscoveryGroupConfiguration{name='my-discovery-group', > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}] > > > > 2024-01-11 18:24:14,082 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > Starting Backup Server > > > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server] > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] > > started, waiting live to fail before it gets active > > > > 2024-01-11 18:24:14,083 TRACE > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > Setting server state as started > > > > 2024-01-11 18:24:14,083 TRACE > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > looking up the node through nodeLocator.locateNode() > > > > 2024-01-11 18:24:14,083 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > Connecting towards a possible live, connection > > information=Pair[a=TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434 > > > > 2024-01-11 18:24:14,083 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434 > > > > 2024-01-11 18:24:14,083 TRACE > > > [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] > > Calling > > > clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local) > > > > 2024-01-11 18:24:14,083 TRACE > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0 > > > > java.lang.Exception:trace<ftp://java.lang.Exception/trace> > > > > at > > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>] > > > > at > > > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>] > > > > at > > > org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>] > > > > at > > > org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>] > > > > at > > > org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>] > > > > at > > > org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>] > > > > at > > > https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 > > rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>] > > > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>] > > > > 2024-01-11 18:24:14,084 DEBUG > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] > > Trying reconnection attempt 0/0 > > > > 2024-01-11 18:24:14,084 DEBUG > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] > > Trying to connect with > > > connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<mailto:connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18> > > > <mailto:connectorFactory<mailto:connectorFactory> > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<mailto:=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18> > > > > and currentConnectorConfig: TransportConfiguration(name=artemis, > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local > > > > 2024-01-11 18:24:14,084 DEBUG > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] > > Connector NettyConnector > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local, > > port=61616, httpEnabled=false, httpUpgradeEnabled=false, > useServlet=false, > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > > using native epoll > > > > 2024-01-11 18:24:14,084 DEBUG > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] > > Started EPOLL Netty Connector version 4.1.86.Final to > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616> > > > > 2024-01-11 18:24:14,084 DEBUG > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] > > Remote destination: > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/ > > 10.44.0.25:61616<ftp://10.44.0.25/61616> > > > > > > Thanks, > > Paul > > > > From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> > > Date: Wednesday, December 13, 2023 at 3:23 PM > > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > > <users@activemq.apache.org<mailto:users@activemq.apache.org>> > > Subject: Re: Trouble with Replication HA Master/Slave config performing > > failback > > ActiveMQ Artemis ships with an example in the > > examples/features/ha/replicated-failback directory that demonstrates how > > this works. I ran this example and it worked fine. I also started up the > > brokers manually and triggered failover and failback. Again, everything > > worked fine. My only guess is that there's something configured wrong or > > perhaps there's an environmental issue that's preventing failback from > > working properly for you. If you could provide clear steps on how to > > reproduce this issue I could investigate further. > > > > > > Justin > > > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael < > paul.shie...@hpe.com<mailto:paul.shie...@hpe.com> > > > > > wrote: > > > > > Hi, > > > > > > I am having an issue with failback to the master broker. I start the > > > master broker then the slave broker. I see the replication from the > > master > > > to the slave happen. I stop the master/active broker and a failover > > happens > > > to the slave as expected. When the master is started a failback to it > is > > > expected but does not happen. The following is my ha-policy config. > > Did I > > > miss something in the ha-policy configuration? We are using version > > 2.28.0 > > > and java17. > > > > > > Thanks, > > > Paul > > > > > > > > > Master broker.xml config > > > > > > <ha-policy> > > > > > > <replication> > > > > > > <master> > > > > > > <check-for-live-server>true</check-for-live-server> > > > > > > </master> > > > > > > </replication> > > > > > > </ha-policy> > > > > > > Slave broker.xml config > > > > > > <ha-policy> > > > > > > <replication> > > > > > > <slave> > > > > > > <allow-failback>true</allow-failback> > > > > > > </slave> > > > > > > </replication> > > > > > > </ha-policy> > > > > > > The following are the logs from the master and slave brokers > > > HA “master” broker log after restart > > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221020: Started EPOLL Acceptor at > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883> > for > > > protocols [MQTT] > > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221020: Started EPOLL Acceptor at > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616> > for > > > protocols [CORE] > > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221007: Server is now live > > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0 > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f] > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003: > > > Starting embedded web server > > > 2023-12-08 20:42:57,625 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 15, result: NO_BINDINGS > > > 2023-12-08 20:42:57,627 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: > created > > > queue: > > > > > > QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@15525ed3 > > > 2023-12-08 20:42:57,657 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null, > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2> > > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<mailto:=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2> > > [ID=47edf16f>, > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, > > > remote=/10.44.0.18:34508<http://10.44.0.18:34508>]] > > > 2023-12-08 20:42:57,720 INFO > > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized > > > activemq-branding plugin > > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff > > > > > > [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl > > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff > > > > > > [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl > > > [initialConnectors=[TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883 > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f, > > > connector=TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > > > address=, server=ActiveMQServerImpl::name=amq-broker])) > > > [initialConnectors=[TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=null]] is connected > > > 2023-12-08 20:42:57,755 INFO > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized > > > artemis-plugin plugin > > > 2023-12-08 20:42:57,766 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 18, result: NO_BINDINGS > > > 2023-12-08 20:42:57,767 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 20, result: NO_BINDINGS > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] > > Initialising > > > hawtio services > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] > > Configuration > > > will be discovered via system properties > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to > > > Hawtio 2.15.0 > > > 2023-12-08 20:42:57,975 INFO > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio > > > authentication filter, JAAS realm: "activemq" authorized role(s): > "admin" > > > role principal classes: > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter] > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy > > > servlet is disabled > > > 2023-12-08 20:42:57,989 INFO > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden > > > property: [key=policyLocation, > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml] > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001: > > > HTTP Server started at > > > > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161> > > > > < > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161> > > > > > > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002: > > > Artemis Jolokia REST API available at > > > > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia> > > > > < > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia> > > > > > > > > 2023-12-08 > > > < > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > > > > > > > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis > > Console > > > available at > > > > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console> > > > > < > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console> > > > > > > > > 2023-12-08 > > > < > > > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08> > < > http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08> > > > > > > > > 20:42:58,455 INFO > [org.apache.activemq.artemis.core.server.plugin.impl] > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b, > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d> > > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<mailto:=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d> > > [ID=f85ec51b>, > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, > > > remote=/127.0.0.6:49865<http://127.0.0.6:49865>]] > > > 2023-12-08 20:42:59,524 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null, > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858> > > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<mailto:=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858> > > [ID=7cef23a2>, > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, > > > remote=/127.0.0.6:59691<http://127.0.0.6:59691>]] > > > 2023-12-08 20:42:59,535 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 22, result: NO_BINDINGS > > > 2023-12-08 20:42:59,536 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: > created > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session > connectionID: > > > 7cef23a2 > > > 2023-12-08 20:42:59,549 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 24, result: NO_BINDINGS > > > 2023-12-08 20:42:59,550 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: > created > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session > connectionID: > > > 7cef23a2 > > > 2023-12-08 20:42:59,583 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 26, result: OK > > > 2023-12-08 20:42:59,584 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: > created > > > queue: > > > > > > QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=true]@5f46053b > > > 2023-12-08 20:42:59,593 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: > created > > > consumer with ID: 0, with session name: > > 5fc2919b-960a-11ee-8014-12a7ed0622c7 > > > 2023-12-08 20:42:59,594 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 28, result: OK > > > 2023-12-08 20:42:59,616 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent > > > message with ID: 29, result: OK, transaction: UNAVAILABLE > > > 2023-12-08 20:42:59,615 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0, > > > timestamp=0,expiration=0, durable=false, > > > > > > address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > > with transaction: null > > > 2023-12-08 20:42:59,621 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 31, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0 > > > 2023-12-08 20:42:59,621 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0, > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false, > > > > > > address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > > with transaction: null > > > 2023-12-08 20:42:59,622 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 32, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0 > > > 2023-12-08 20:42:59,622 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0, > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false, > > > > > > address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > > with transaction: null > > > 2023-12-08 20:42:59,622 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 35, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0 > > > 2023-12-08 20:42:59,622 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0, > > > timestamp=0,expiration=0, durable=false, > > > > > > address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > > with transaction: null > > > 2023-12-08 20:42:59,622 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 39, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0 > > > 2023-12-08 20:43:08,447 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null, > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256> > > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<mailto:=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256> > > [ID=1d1d79ce>, > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, > > > remote=/127.0.0.6:60787<http://127.0.0.6:60787>]] > > > 2023-12-08 20:43:08,464 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null, > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7> > > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<mailto:=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7> > > [ID=8f88c6d5>, > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, > > > remote=/127.0.0.6:40065<http://127.0.0.6:40065>]] > > > > > > > > > HA “slave” broker log. > > > > > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] > > > started, waiting live to fail before it gets active > > > > > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is > > > synchronized with live server, > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364. > > > > > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221031: backup announced > > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote > > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221084: Requested 0 quorum votes > > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221083: ignoring quorum vote as max cluster size is 1. > > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221071: Failing over based on quorum vote results. > > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client] > > > AMQ212037: Connection failure to > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/ > > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: > > > The connection was > > > disconnected because of server shutdown [code=DISCONNECTED] > > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client] > > > AMQ212037: Connection failure to > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/ > > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: > > > The connection was > > > disconnected because of server shutdown [code=DISCONNECTED] > > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live' > > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221080: Deploying address DLQ supporting [ANYCAST] > > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ > > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST] > > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue > > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221007: Server is now live > > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221020: Started EPOLL Acceptor at > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883> > for > > > protocols [MQTT] > > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221020: Started EPOLL Acceptor at > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616> > for > > > protocols [CORE] > > > 2023-12-08 20:42:51,946 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null, > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > [ID=df357d0e>, > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, > > > remote=/127.0.0.6:56167<http://127.0.0.6:56167>]] > > > 2023-12-08 20:42:56,977 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: > > destroyed > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null, > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > [ID=df357d0e>, > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, > > > remote=/127.0.0.6:56167<http://127.0.0.6:56167>]] > > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client] > > > AMQ212004: Failed to connect to server. > > > 2023-12-08 20:42:57,445 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 26, result: NO_BINDINGS > > > 2023-12-08 20:42:57,447 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: > created > > > queue: > > > > > > QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@737e64a6 > > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection > > > failure detected. Unblocking a blocking call that will never get a > > response > > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server] > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52 > > > > > > [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl > > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52 > > > > > > [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl > > > [initialConnectors=[TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008 > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364, > > > connector=TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > > > address=, server=ActiveMQServerImpl::name=amq-broker])) > > > [initialConnectors=[TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=null]] is unable to connect to destination. > > > Retrying > > > 2023-12-08 20:42:57,629 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null, > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > [ID=483170f6>, > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, > > > remote=/127.0.0.6:55911<http://127.0.0.6:55911>]] > > > 2023-12-08 20:42:57,630 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null, > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > [ID=4514a2b3>, > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, > > > remote=/127.0.0.6:60199<http://127.0.0.6:60199>]] > > > 2023-12-08 20:42:57,648 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: > created > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null, > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364, > > > > > > transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection> > > > > > > =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > [ID=fcde4d78>, > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, > > > remote=/127.0.0.6:53221<http://127.0.0.6:53221>]] > > > 2023-12-08 20:42:57,663 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 29, result: NO_BINDINGS > > > 2023-12-08 20:42:57,663 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: > created > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session > connectionID: > > > fcde4d78 > > > 2023-12-08 20:42:57,679 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 31, result: NO_BINDINGS > > > 2023-12-08 20:42:57,679 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: > created > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session > connectionID: > > > fcde4d78 > > > 2023-12-08 20:42:57,708 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 33, result: OK > > > 2023-12-08 20:42:57,709 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: > created > > > queue: > > > > > > QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=true]@3645037f > > > 2023-12-08 20:42:57,719 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: > created > > > consumer with ID: 0, with session name: > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f > > > 2023-12-08 20:42:57,720 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 35, result: OK > > > 2023-12-08 20:42:57,755 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent > > > message with ID: 36, result: OK, transaction: UNAVAILABLE > > > 2023-12-08 20:42:57,753 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0, > > > timestamp=0,expiration=0, durable=false, > > > > > > address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > > with transaction: null > > > 2023-12-08 20:42:57,760 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 38, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0 > > > 2023-12-08 20:42:57,761 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0, > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false, > > > > > > address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > > with transaction: null > > > 2023-12-08 20:42:57,761 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 41, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0 > > > 2023-12-08 20:42:57,762 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0, > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false, > > > > > > address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > > with transaction: null > > > 2023-12-08 20:42:57,762 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 43, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0 > > > 2023-12-08 20:42:57,762 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: > > > acknowledged message: > > > > > > Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0, > > > timestamp=0,expiration=0, durable=false, > > > > > > address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > > with transaction: null > > > 2023-12-08 20:42:57,762 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: > > delivered > > > message with message ID: 44, to consumer on address: > > > activemq.notifications, queue: > > > > > > notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0 > > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52 > > > > > > [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl > > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52 > > > > > > [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > > > > queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, > > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl > > > [initialConnectors=[TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008 > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364, > > > connector=TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, > > > address=, server=ActiveMQServerImpl::name=amq-broker])) > > > [initialConnectors=[TransportConfiguration(name=artemis, > > > > > > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > > > > > > ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], > > > discoveryGroupConfiguration=null]] is connected > > > 2023-12-08 20:42:59,625 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 46, result: OK > > > 2023-12-08 20:42:59,625 INFO > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed > > > message with ID: 48, result: OK > > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection > > > failure detected. Unblocking a blocking call that will never get a > > response > > > > > > > > > > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
