I assume there aren't any failures earlier in broker1's logs that might be relevant?
>From those logs, it looks like Kubernetes is shutting down ActiveMQ rather than ActiveMQ shutting itself down, based largely on the fact that there is nothing that indicates that ActiveMQ initiated the shutdown. Might broker1's RAM usage be exceeding the limit you set for the pod, or maybe exceeding the remaining RAM on your underlying node? If there are no other nodes on which the pod could be rescheduled, that could result in this happening again and again. For that to be the cause, you'd need Xms < Xmx on the JVM and Xmx > remaining available RAM on the node when the pod is scheduled there. I think you'd also need memoryRequest < memoryLimit on the pod. Have you interrogated Kubernetes to determine why it believes that the pod exited, via a technique like https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/ ? Tim On Wed, Dec 2, 2020, 10:38 AM Imran Raza Khan <imranrazak...@gmail.com> wrote: > I have deployed two statefulset replicas of ActiveMQ 5.15.13 as Network > Broker, but second POD activemq-1 is restarting after every 4m51s (I > removed readiness and liveness probes too), I enabled debug logs but could > not find any clue > > **ActiveMQ-0:** > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="broker0" brokerId="A" persistent="false" useJmx="true"> > <networkConnectors> > <networkConnector name="OpenBroker1" > > uri="static:(tcp://activemq-1.broker-amq-tcp.production.svc.cluster.local:61616)" > networkTTL="3" /> > </networkConnectors> > > <transportConnectors> > <transportConnector name="openwire" uri="tcp:// > > 0.0.0.0:61616?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600 > "/> > <transportConnector name="mqtt" uri="mqtt:// > > 0.0.0.0:1883?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600 > "/> > </transportConnectors> > > **ActiveMQ-1:** > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="broker1" brokerId="B" persistent="false" useJmx="true"> > > <networkConnectors> > <networkConnector name="OpenBroker0" > > uri="static:(tcp://activemq-0.broker-amq-tcp.production.svc.cluster.local:61616)" > networkTTL="3" /> > </networkConnectors> > > <transportConnectors> > <transportConnector name="openwire" uri="tcp:// > > 0.0.0.0:61616?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600 > "/> > <transportConnector name="mqtt" uri="mqtt:// > > 0.0.0.0:1883?maximumConnections=1000&wireFormat.maxInactivityDuration=65000&wireFormat.maxFrameSize=104857600 > "/> > </transportConnectors> > > > Following are last logs in pod > > DEBUG | WriteChecker: 10000ms elapsed since last write check. > DEBUG | Running WriteCheck[tcp://10.244.1.1:44940] > DEBUG | 30000ms elapsed since last read check. > DEBUG | WriteChecker: 10000ms elapsed since last write check. > DEBUG | Running WriteCheck[tcp://10.244.0.67:61616] > DEBUG | WriteChecker: 10000ms elapsed since last write check. > DEBUG | Running WriteCheck[tcp://10.244.1.1:44940] > DEBUG | Found 275 loggers > DEBUG | WriteChecker: 10000ms elapsed since last write check. > DEBUG | Running WriteCheck[tcp://10.244.0.67:61616] > INFO | Apache ActiveMQ 5.15.13 (broker1, B) is shutting down > DEBUG | Caught exception, must be shutting down. This exception is > ignored. > java.lang.IllegalStateException: Shutdown in progress > at > > java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)[:1.8.0_212] > at > java.lang.Runtime.removeShutdownHook(Runtime.java:239)[:1.8.0_212] > at > > org.apache.activemq.broker.BrokerService.removeShutdownHook(BrokerService.java:2565)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService.stop(BrokerService.java:844)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:2588)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService$7.run(BrokerService.java:2555)[activemq-broker-5.15.13.jar:5.15.13] > DEBUG | Unregistering MBean > > org.apache.activemq:type=Broker,brokerName=broker1,connector=networkConnectors,networkConnectorName=OpenBroker0 > DEBUG | stopping broker1 bridge to broker0 > DEBUG | Unregistering MBean > > org.apache.activemq:brokerName=broker1,connector=networkConnectors,networkConnectorName=OpenBroker0,networkBridge=tcp_//10.244.0.67_61616,type=Broker > DEBUG | broker1 removeBroker: broker0 brokerInfo size: 0 > DEBUG | Caught exception sending shutdown > org.apache.activemq.broker.BrokerStoppedException: Broker > BrokerService[broker1] is being stopped > at > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:205)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:165)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:157)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:134)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:313)[activemq-broker-5.15.13.jar:5.15.13] > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_212] > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_212] > at java.lang.Thread.run(Thread.java:748)[:1.8.0_212] > DEBUG | Stopping transport > tcp://activemq-0.broker-amq-tcp.production.svc.cluster.local/ > 10.244.0.67:61616 > @45626 > DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task] using > ExecutorService: java.util.concurrent.ThreadPoolExecutor@4d3d6604[Running, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] > DEBUG | Closed socket > Socket[addr=activemq-0.broker-amq-tcp.production.svc.cluster.local/ > 10.244.0.67,port=61616,localport=45626] > DEBUG | Forcing shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@4d3d6604[Running, pool size = 1, > active threads = 0, queued tasks = 0, completed tasks = 1] > DEBUG | Transport Connection to: vm://broker1#0 failed: > org.apache.activemq.transport.TransportDisposedIOException: peer > (vm://broker1#0) stopped. > org.apache.activemq.transport.TransportDisposedIOException: peer > (vm://broker1#0) stopped. > at > > org.apache.activemq.transport.vm.VMTransport.stop(VMTransport.java:233)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.vm.VMTransportServer$1.stop(VMTransportServer.java:80)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:72)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:72)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.transport.ResponseCorrelator.stop(ResponseCorrelator.java:132)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.network.DemandForwardingBridgeSupport.stop(DemandForwardingBridgeSupport.java:331)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.network.DiscoveryNetworkConnector.handleStop(DiscoveryNetworkConnector.java:223)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.network.NetworkConnector$1.doStop(NetworkConnector.java:65)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.network.NetworkConnector.stop(NetworkConnector.java:170)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)[activemq-client-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService.stopAllConnectors(BrokerService.java:2250)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService.stop(BrokerService.java:854)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:2588)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerService$7.run(BrokerService.java:2555)[activemq-broker-5.15.13.jar:5.15.13] > DEBUG | Stopping connection: vm://broker1#0 > DEBUG | Shutting down VM connectors for broker: broker1 > DEBUG | Stopped transport: vm://broker1#0 > INFO | Connector vm://broker1 stopped > DEBUG | Cleaning up connection resources: vm://broker1#0 > INFO | broker1 bridge to broker0 stopped > DEBUG | remove connection id: broker1->broker0-39336-1606929012230-4:1 > DEBUG | Shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@55c53a33[Terminated, pool size = > 0, > active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown: > true and terminated: true took: 0.000 seconds. > DEBUG | Unregistering MBean > > org.apache.activemq:type=Broker,brokerName=broker1,destinationType=Topic,destinationName=BPPM.status.uf_droop_limits,endpoint=Consumer,clientId=OpenBroker0_broker0_inbound_broker1,consumerId=broker1->broker0-39336-1606929012230-4_1_1_85 > INFO | Network Connector > DiscoveryNetworkConnector:OpenBroker0:BrokerService[broker1] stopped > DEBUG | broker1 removing consumer: > broker1->broker0-39336-1606929012230-4:1:1:85 for destination: > topic://BPPM.status.uf_droop_limits > DEBUG | Unregistering MBean > > org.apache.activemq:type=Broker,brokerName=broker1,connector=clientConnectors,connectorName=openwire > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Stopping connection: tcp://10.244.1.1:44940 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@458f973c[Terminated, pool size = > 0, > active threads = 0, queued tasks = 0, completed tasks = 53] is shutdown: > true and terminated: true took: 0.001 seconds. > DEBUG | Stopping transport tcp:///10.244.1.1:44940@61616 > DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task] using > ExecutorService: java.util.concurrent.ThreadPoolExecutor@60a2fea6[Running, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Closed socket Socket[addr=/10.244.1.1 > ,port=44940,localport=61616] > DEBUG | Forcing shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@60a2fea6[Running, pool size = 1, > active threads = 0, queued tasks = 0, completed tasks = 1] > DEBUG | broker1 removing consumer: > broker1->broker0-39336-1606929012230-4:1:1:86 for destination: > topic://ALARM.control.request_filter > DEBUG | Stopped transport: tcp://10.244.1.1:44940 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Cleaning up connection resources: tcp://10.244.1.1:44940 > DEBUG | remove connection id: > broker0->broker1-40124-1606922636926-131:2 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > DEBUG | Connection Stopped: tcp://10.244.1.1:44940 > INFO | Connector openwire stopped > DEBUG | Unregistering MBean > > org.apache.activemq:type=Broker,brokerName=broker1,connector=clientConnectors,connectorName=mqtt > DEBUG | Publishing: vm://broker1 for broker transport URI: vm://broker1 > INFO | Connector mqtt stopped > WARN | Failed to remove consumer: > broker1->broker0-39336-1606929012230-4:1:1:93 > org.apache.activemq.broker.BrokerStoppedException: Broker has been > stopped: org.apache.activemq.broker.BrokerService$6@74676c8d > at > > org.apache.activemq.broker.ErrorBroker.addDestination(ErrorBroker.java:166)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:454)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:293)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:912)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:857)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:843)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:372)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:139)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:139)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:139)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:735)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:774)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:885)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1230)[activemq-broker-5.15.13.jar:5.15.13] > at > > org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1160)[activemq-broker-5.15.13.jar:5.15.13] > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_212] > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_212] > at java.lang.Thread.run(Thread.java:748)[:1.8.0_212] > > ....... Above error Failed to remove consumer repeate multiple times > here ................................................ > DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=broker1 > DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=broker1,service=Health > DEBUG | Unregistering MBean > > org.apache.activemq:type=Broker,brokerName=broker1,service=Log4JConfiguration > DEBUG | Stopping jmx connector > DEBUG | Unexported JMX RMI Registry > DEBUG | Shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@225129c[Terminated, pool size = 0, > active threads = 0, queued tasks = 0, completed tasks = 199] is shutdown: > true and terminated: false took: 0.000 seconds. > DEBUG | Forcing shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@7519ea70[Running, pool size = 0, > active threads = 0, queued tasks = 0, completed tasks = 0] > INFO | Apache ActiveMQ 5.15.13 (broker1, B) uptime 4 minutes > INFO | Apache ActiveMQ 5.15.13 (broker1, B) is shutdown > INFO | Closing > org.apache.activemq.xbean.XBeanBrokerFactory$1@27d415d9: startup date [Wed > Dec 02 17:10:11 GMT 2020]; root of context hierarchy > DEBUG | JspServlet.destroy() > DEBUG | JspServlet.destroy() >
