I have deployed two statefulset replicas of ActiveMQ 5.15.13 as Network
Broker, but second POD activemq-1 is restarting after every 4m51s (I
removed readiness and liveness probes too), I enabled debug logs but could
not find any clue
**ActiveMQ-0:**
<broker xmlns="http://activemq.apache.org/schema/core";
brokerName="broker0" brokerId="A" persistent="false" useJmx="true">
<networkConnectors>
<networkConnector name="OpenBroker1"
uri="static:(tcp://activemq-1.broker-amq-tcp.production.svc.cluster.local:61616)"
networkTTL="3" />
</networkConnectors>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://
0.0.0.0:61616?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600
"/>
<transportConnector name="mqtt" uri="mqtt://
0.0.0.0:1883?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600
"/>
</transportConnectors>
**ActiveMQ-1:**
<broker xmlns="http://activemq.apache.org/schema/core";
brokerName="broker1" brokerId="B" persistent="false" useJmx="true">
<networkConnectors>
<networkConnector name="OpenBroker0"
uri="static:(tcp://activemq-0.broker-amq-tcp.production.svc.cluster.local:61616)"
networkTTL="3" />
</networkConnectors>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://
0.0.0.0:61616?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600
"/>
<transportConnector name="mqtt" uri="mqtt://
0.0.0.0:1883?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600
"/>
</transportConnectors>
Following are last logs in pod
DEBUG | WriteChecker: 10000ms elapsed since last write check.
DEBUG | Running WriteCheck[tcp://10.244.1.1:44940]
DEBUG | 30000ms elapsed since last read check.
DEBUG | WriteChecker: 10000ms elapsed since last write check.
DEBUG | Running WriteCheck[tcp://10.244.0.67:61616]
DEBUG | WriteChecker: 10000ms elapsed since last write check.
DEBUG | Running WriteCheck[tcp://10.244.1.1:44940]
DEBUG | Found 275 loggers
DEBUG | WriteChecker: 10000ms elapsed since last write check.
DEBUG | Running WriteCheck[tcp://10.244.0.67:61616]
INFO | Apache ActiveMQ 5.15.13 (broker1, B) is shutting down
DEBUG | Caught exception, must be shutting down. This exception is
ignored.
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)[:1.8.0_212]
at
java.lang.Runtime.removeShutdownHook(Runtime.java:239)[:1.8.0_212]
at
org.apache.activemq.broker.BrokerService.removeShutdownHook(BrokerService.java:2565)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService.stop(BrokerService.java:844)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:2588)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService$7.run(BrokerService.java:2555)[activemq-broker-5.15.13.jar:5.15.13]
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1,connector=networkConnectors,networkConnectorName=OpenBroker0
DEBUG | stopping broker1 bridge to broker0
DEBUG | Unregistering MBean
org.apache.activemq:brokerName=broker1,connector=networkConnectors,networkConnectorName=OpenBroker0,networkBridge=tcp_//10.244.0.67_61616,type=Broker
DEBUG | broker1 removeBroker: broker0 brokerInfo size: 0
DEBUG | Caught exception sending shutdown
org.apache.activemq.broker.BrokerStoppedException: Broker
BrokerService[broker1] is being stopped
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:205)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:165)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:157)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:134)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:313)[activemq-broker-5.15.13.jar:5.15.13]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_212]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_212]
DEBUG | Stopping transport
tcp://activemq-0.broker-amq-tcp.production.svc.cluster.local/10.244.0.67:61616
@45626
DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task] using
ExecutorService: java.util.concurrent.ThreadPoolExecutor@4d3d6604[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
DEBUG | Closed socket
Socket[addr=activemq-0.broker-amq-tcp.production.svc.cluster.local/
10.244.0.67,port=61616,localport=45626]
DEBUG | Forcing shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@4d3d6604[Running, pool size = 1,
active threads = 0, queued tasks = 0, completed tasks = 1]
DEBUG | Transport Connection to: vm://broker1#0 failed:
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://broker1#0) stopped.
org.apache.activemq.transport.TransportDisposedIOException: peer
(vm://broker1#0) stopped.
at
org.apache.activemq.transport.vm.VMTransport.stop(VMTransport.java:233)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.vm.VMTransportServer$1.stop(VMTransportServer.java:80)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:72)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:72)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.transport.ResponseCorrelator.stop(ResponseCorrelator.java:132)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.network.DemandForwardingBridgeSupport.stop(DemandForwardingBridgeSupport.java:331)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.network.DiscoveryNetworkConnector.handleStop(DiscoveryNetworkConnector.java:223)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.network.NetworkConnector$1.doStop(NetworkConnector.java:65)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.network.NetworkConnector.stop(NetworkConnector.java:170)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)[activemq-client-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService.stopAllConnectors(BrokerService.java:2250)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService.stop(BrokerService.java:854)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:2588)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerService$7.run(BrokerService.java:2555)[activemq-broker-5.15.13.jar:5.15.13]
DEBUG | Stopping connection: vm://broker1#0
DEBUG | Shutting down VM connectors for broker: broker1
DEBUG | Stopped transport: vm://broker1#0
INFO | Connector vm://broker1 stopped
DEBUG | Cleaning up connection resources: vm://broker1#0
INFO | broker1 bridge to broker0 stopped
DEBUG | remove connection id: broker1->broker0-39336-1606929012230-4:1
DEBUG | Shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@55c53a33[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown:
true and terminated: true took: 0.000 seconds.
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1,destinationType=Topic,destinationName=BPPM.status.uf_droop_limits,endpoint=Consumer,clientId=OpenBroker0_broker0_inbound_broker1,consumerId=broker1->broker0-39336-1606929012230-4_1_1_85
INFO | Network Connector
DiscoveryNetworkConnector:OpenBroker0:BrokerService[broker1] stopped
DEBUG | broker1 removing consumer:
broker1->broker0-39336-1606929012230-4:1:1:85 for destination:
topic://BPPM.status.uf_droop_limits
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1,connector=clientConnectors,connectorName=openwire
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Stopping connection: tcp://10.244.1.1:44940
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@458f973c[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 53] is shutdown:
true and terminated: true took: 0.001 seconds.
DEBUG | Stopping transport tcp:///10.244.1.1:44940@61616
DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task] using
ExecutorService: java.util.concurrent.ThreadPoolExecutor@60a2fea6[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Closed socket Socket[addr=/10.244.1.1
,port=44940,localport=61616]
DEBUG | Forcing shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@60a2fea6[Running, pool size = 1,
active threads = 0, queued tasks = 0, completed tasks = 1]
DEBUG | broker1 removing consumer:
broker1->broker0-39336-1606929012230-4:1:1:86 for destination:
topic://ALARM.control.request_filter
DEBUG | Stopped transport: tcp://10.244.1.1:44940
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Cleaning up connection resources: tcp://10.244.1.1:44940
DEBUG | remove connection id: broker0->broker1-40124-1606922636926-131:2
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
DEBUG | Connection Stopped: tcp://10.244.1.1:44940
INFO | Connector openwire stopped
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1,connector=clientConnectors,connectorName=mqtt
DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1
INFO | Connector mqtt stopped
WARN | Failed to remove consumer:
broker1->broker0-39336-1606929012230-4:1:1:93
org.apache.activemq.broker.BrokerStoppedException: Broker has been
stopped: org.apache.activemq.broker.BrokerService$6@74676c8d
at
org.apache.activemq.broker.ErrorBroker.addDestination(ErrorBroker.java:166)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:454)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:293)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:912)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:857)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:843)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:372)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:139)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:139)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:139)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:735)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:774)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:885)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1230)[activemq-broker-5.15.13.jar:5.15.13]
at
org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1160)[activemq-broker-5.15.13.jar:5.15.13]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_212]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_212]
....... Above error Failed to remove consumer repeate multiple times
here ................................................
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1,service=Health
DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=broker1,service=Log4JConfiguration
DEBUG | Stopping jmx connector
DEBUG | Unexported JMX RMI Registry
DEBUG | Shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@225129c[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 199] is shutdown:
true and terminated: false took: 0.000 seconds.
DEBUG | Forcing shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@7519ea70[Running, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
INFO | Apache ActiveMQ 5.15.13 (broker1, B) uptime 4 minutes
INFO | Apache ActiveMQ 5.15.13 (broker1, B) is shutdown
INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@27d415d9: startup date [Wed
Dec 02 17:10:11 GMT 2020]; root of context hierarchy
DEBUG | JspServlet.destroy()
DEBUG | JspServlet.destroy()
