Hi Justin, I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url: (tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100 Should we also be including the corresponding connector names? In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly. Thanks in advance for your help. Lino Thread-0 log traces: 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown] --- 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<mailto:connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<mailto:connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null --- 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<mailto:connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<mailto:connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management, needed = 65536, arriving = 0 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected] at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410) at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407) at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98) at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [rg] <https://www.redpointglobal.com/> Lino Pereira C++ Developer, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 lino.pere...@redpointglobal.com<mailto:lino.pere...@redpointglobal.com> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
