Hello
I'm trying to figure out an issue we have been seeing in our AWS instances.
We have a 3 node Master/Slave/Slave cluster in AWS that keeps failing over
randomly.
It occurs about once or twice each day and does not seem to be a good
reason.
It looks like if the bridge stops it kills ActiveMQ I would have thought it
should just reconnect.
I'm hoping someone can help me out.
My config is as follows:
>Windows Server 2012 systems
>3 Node Master Slave
>persistenceAdapter kahaDB is shared on DFS
>lease-database-locker using SQL Server
>Networked to a USEast AWS cluster (Note no issues with failover)
>ActiveMQ Version 5.15.3
My Log segment:
INFO | jvm 1 | 2018/03/21 16:36:43 | INFO | Successfully connected to
ssl://XXXXXXXX01.corp.local:61617
INFO | jvm 1 | 2018/03/21 16:36:43 | INFO | Successfully connected to
ssl://XXXXXXXX01.corp.local:61617
INFO | jvm 1 | 2018/03/21 16:36:44 | INFO | Network connection between
vm://fmigrp01#8 and ssl://XXXXXXXX01.corp.local:61617 (fmiuseastp01) has
been established.
INFO | jvm 1 | 2018/03/21 16:36:44 | INFO | Network connection between
vm://fmigrp01#12 and ssl://XXXXXXXX01.corp.local:61617 (fmiuseastp01) has
been established.
INFO | jvm 1 | 2018/03/21 16:36:48 | INFO | Successfully refreshed JMS
Connection
INFO | jvm 1 | 2018/03/21 22:50:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/21 23:30:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/21 23:55:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/22 00:25:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/22 01:00:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/22 01:55:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/22 02:10:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/22 02:20:00 | INFO | VirtualTopic.heartbeat
Inactive for longer than 300000 ms - removing ...
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Apache ActiveMQ 5.15.3
(fmigrp01, ID:DDDDDDDD01-52527-1521555769961-1:2) is shutting down
INFO | jvm 1 | 2018/03/22 18:19:32 | WARN | Transport
(ssl://XXXXXXXX01.corp.local:61617) failed , not attempting to automatically
reconnect: {}
INFO | jvm 1 | 2018/03/22 18:19:32 | java.io.IOException: Bridge
stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.DemandForwardingBridgeSupport.stopFailoverTransport(DemandForwardingBridgeSupport.java:351)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.DemandForwardingBridgeSupport.stop(DemandForwardingBridgeSupport.java:327)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.DiscoveryNetworkConnector.handleStop(DiscoveryNetworkConnector.java:223)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.NetworkConnector$1.doStop(NetworkConnector.java:65)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)[activemq-client-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.NetworkConnector.stop(NetworkConnector.java:170)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)[activemq-client-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.broker.BrokerService.stopAllConnectors(BrokerService.java:2236)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.broker.BrokerService.stop(BrokerService.java:848)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:104)[activemq-console-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | WARN | Transport Connection to:
vm://fmigrp01#8 failed:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://fmigrp01#8) stopped.
INFO | jvm 1 | 2018/03/22 18:19:32 | WARN | Transport Connection to:
vm://fmigrp01#10 failed:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://fmigrp01#10) stopped.
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | fmigrp01 bridge to
fmiuseastp01 stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Network Connector
DiscoveryNetworkConnector:fmiuseast1:BrokerService[fmigrp01] stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | WARN | Transport
(ssl://XXXXXXXX01.corp.local:61617) failed , not attempting to automatically
reconnect: {}
INFO | jvm 1 | 2018/03/22 18:19:32 | java.io.IOException: Bridge
stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.DemandForwardingBridgeSupport.stopFailoverTransport(DemandForwardingBridgeSupport.java:351)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.DemandForwardingBridgeSupport.stop(DemandForwardingBridgeSupport.java:327)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.DiscoveryNetworkConnector.handleStop(DiscoveryNetworkConnector.java:223)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.NetworkConnector$1.doStop(NetworkConnector.java:65)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)[activemq-client-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.network.NetworkConnector.stop(NetworkConnector.java:170)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)[activemq-client-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.broker.BrokerService.stopAllConnectors(BrokerService.java:2236)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.broker.BrokerService.stop(BrokerService.java:848)[activemq-broker-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | at
org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:104)[activemq-console-5.15.3.jar:5.15.3]
INFO | jvm 1 | 2018/03/22 18:19:32 | WARN | Transport Connection to:
vm://fmigrp01#12 failed:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://fmigrp01#12) stopped.
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | fmigrp01 bridge to
fmiuseastp01 stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Network Connector
DiscoveryNetworkConnector:fmiuseast2:BrokerService[fmigrp01] stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Connector openwire stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Connector amqp stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Connector stomp stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Connector ssl stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Connector stomp+ssl
stopped
INFO | jvm 1 | 2018/03/22 18:19:32 | INFO | Connector amqp+ssl stopped
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Connector vm://fmigrp01
stopped
INFO | jvm 1 | 2018/03/22 18:19:33 | WARN | Setup of JMS message
listener invoker failed for destination 'esb.*' - trying to recover. Cause:
peer (vm://fmigrp01#15) stopped.
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | JobSchedulerStore:
\\corp.local\Production-GMBH\MISC\ActiveMQ\schedule\fmigrp01\scheduler
stopped.
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO |
PListStore:[\\corp.local\Production-GMBH\MISC\ActiveMQ\schedule\fmigrp01\tmp_storage]
stopped
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Stopping async queue tasks
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Stopping async topic tasks
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Stopped KahaDB
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | fmigrp01, released lease
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Apache ActiveMQ 5.15.3
(fmigrp01, ID:DDDDDDDD01-52527-1521555769961-1:2) uptime 1 day 9 hours
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Apache ActiveMQ 5.15.3
(fmigrp01, ID:DDDDDDDD01-52527-1521555769961-1:2) is shutdown
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@13b0427b: startup date [Wed
Mar 21 08:41:11 UTC 2018]; root of context hierarchy
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Apache Camel 2.19.3
(CamelContext: esbRoute1) is shutting down
INFO | jvm 1 | 2018/03/22 18:19:33 | INFO | Starting to graceful
shutdown 1 routes (timeout 300 seconds)
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
