> Now, the backup process does start processing msgs right away... What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover. You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie. Lastly, please indicate what version of ActiveMQ Artemis you're using. Justin On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <lino.pere...@redpointglobal.com.invalid> wrote: > Hi Justin, > > I’m working with John Lilley on getting an ActiveMQ HA configuration > operational. Our current HA config is replication with master and slave > instances and with static connections, and it seems to be working, except > that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect > itself after a failover from the live to the backup instance. Now, the > backup process does start processing msgs right away, but the > ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, > before it stabilizes after the failover and is once again usable. I have > included the relevant log traces from the Thread-0 ActiveMQ-client showing > the Connection failure detection all the way through when the client is > usable again, over 1 minute later. I am hoping that, based on the trace, > you can find something that, perhaps, we should be doing differently in the > client side? > > > > Here is our Client connection url: > > (tcp://dm-activemq-live-svc: > 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100 > > > > Should we also be including the corresponding connector names? > > > > In an earlier email with John Lilley you recommended 2 brokers using > “shared storage” instead of “replication”, which I will try next, however, > since this issue seems to be on the client-side, I want to make sure our > client-side is setup correctly. > > > > Thanks in advance for your help. > > Lino > > > > Thread-0 log traces: > > 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] > client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection > failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected: > AMQ219015: The connection was disconnected because of server shutdown > [code=DISCONNECTED] > > 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure > captured on connectionID=a8a4a77c, performing failover or reconnection now > ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The > connection was disconnected because of server shutdown] > > --- > > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying > reconnection attempt 0/200 > > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying > to connect with > connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0 > and currentConnectorConfig: TransportConfiguration(name=null, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200 > > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:553 [] DEBUG - Connector NettyConnector > [host=dm-activemq-live-svc, port=61616, httpEnabled=false, > httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > using nio > > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version > 4.1.101.Final to dm-activemq-live-svc:61616 > > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.createConnection:877 [] DEBUG - Remote destination: > dm-activemq-live-svc/10.43.197.248:61616 > > > > 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector > towards NettyConnector [host=dm-activemq-live-svc, port=61616, > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > failed > > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying > to connect with > connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0 > and backupConnectorConfig: TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61617&host=dm-activemq-backup-svc > > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:553 [] DEBUG - Connector NettyConnector > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, > httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > using nio > > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version > 4.1.101.Final to dm-activemq-backup-svc:61617 > > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.createConnection:877 [] DEBUG - Remote destination: > dm-activemq-backup-svc/10.43.11.76:61617 > > > > 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector > towards NettyConnector [host=dm-activemq-backup-svc, port=61617, > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > failed > > 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no > connection been made, returning null > > --- > > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying > reconnection attempt 1/200 > > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying > to connect with > connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0 > and currentConnectorConfig: TransportConfiguration(name=null, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200 > > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:553 [] DEBUG - Connector NettyConnector > [host=dm-activemq-live-svc, port=61616, httpEnabled=false, > httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > using nio > > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version > 4.1.101.Final to dm-activemq-live-svc:61616 > > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.createConnection:877 [] DEBUG - Remote destination: > dm-activemq-live-svc/10.43.197.248:61616 > > 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] > NettyConnector.initChannel:762 [] DEBUG - Added > ActiveMQClientChannelHandler to Channel with id = ced14fbc > > > > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector > towards NettyConnector [host=dm-activemq-live-svc, port=61616, > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > failed > > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying > to connect with > connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0 > and backupConnectorConfig: TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61617&host=dm-activemq-backup-svc > > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:553 [] DEBUG - Connector NettyConnector > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, > httpUpgradeEnabled=false, useServlet=false, > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] > using nio > > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version > 4.1.101.Final to dm-activemq-backup-svc:61617 > > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] > NettyConnector.createConnection:877 [] DEBUG - Remote destination: > dm-activemq-backup-svc/10.43.11.76:61617 > > > > 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - > Connected with the > backupConnectorConfig=TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61617&host=dm-activemq-backup-svc > > > > 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection > successful > > 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't > reattach session 11, performing as a failover operation now and recreating > objects > > 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't > reattach session 10, performing as a failover operation now and recreating > objects > > > > 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] > AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address > activemq.management > > 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] > AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 > credits on address activemq.management, needed = 65536, arriving = 0 > > 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] > AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 > credits on address activemq.management > > 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure > captured on connectionID=a8a4a77c, performing failover or reconnection now > ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: > Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407) > > at > org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98) > > at > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > [image: rg] <https://www.redpointglobal.com/> > > Lino Pereira > > C++ Developer, Redpoint Global Inc. > > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 > > lino.pere...@redpointglobal.com > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
