Hello All,
I am using an active-mq classic v 5.18.3 instance and we are running into an
issue where the active mq process shows steady increase in threads and memory
utilization and it eventually crashes.
We have many active connections(thousands) from different producers and
consumers going into this active-mq instance and we know that the
system(hardware) where this is run has enough CPU and memory to handle it.
The maximumConnections value for the NIO transport connector is set to 65000.
The heap memory for active mq is set to grow up to 12 GB. I got 2 thread dumps
from the active mq process and it shows the following 2 threads steadily
increasing:
"ActiveMQ BrokerService[TestBroker] Task-3906" #44297 daemon prio=5 os_prio=0
cpu=335.99ms elapsed=3286.97s tid=0x00007fb408017000 nid=0x120b9a runnable
[0x00007fb161a1e000]
java.lang.Thread.State: RUNNABLE
at
sun.nio.ch.EPoll.wait(java.base@11.0.12.0.101/Native<mailto:java.base@11.0.12.0.101/Native>
Method)
at
sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12.0.101/EPollSelectorImpl.java:120<mailto:java.base@11.0.12.0.101/EPollSelectorImpl.java:120>)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12.0.101/SelectorImpl.java:124<mailto:java.base@11.0.12.0.101/SelectorImpl.java:124>)
- locked <0x0000000530d66360> (a sun.nio.ch.Util$2)
- locked <0x0000000530d66308> (a sun.nio.ch.EPollSelectorImpl)
at
sun.nio.ch.SelectorImpl.select(java.base@11.0.12.0.101/SelectorImpl.java:136<mailto:java.base@11.0.12.0.101/SelectorImpl.java:136>)
at
org.apache.activemq.transport.nio.NIOSSLTransport.doHandshake(NIOSSLTransport.java:447)
at
org.apache.activemq.transport.nio.NIOSSLTransport.initializeStreams(NIOSSLTransport.java:165)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:543)
at
org.apache.activemq.transport.nio.NIOTransport.doStart(NIOTransport.java:181)
at
org.apache.activemq.transport.nio.NIOSSLTransport.doStart(NIOSSLTransport.java:479)
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:172)
at
org.apache.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:52)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
at
org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1077)
- locked <0x0000000530d5a3a0> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
at
org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:235)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128>)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12.0.101/ThreadPoolExecutor.java:628<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:628>)
at
java.lang.Thread.run(java.base@11.0.12.0.101/Thread.java:829<mailto:java.base@11.0.12.0.101/Thread.java:829>)
"ActiveMQ InactivityMonitor Worker 159" #44315 daemon prio=5 os_prio=0
cpu=125.60ms elapsed=3284.27s tid=0x00007fb8cc0d0800 nid=0x120bc0 waiting for
monitor entry [0x00007fb1ef572000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.stopAsync(ManagedTransportConnection.java:67)
- waiting to lock <0x0000000530d474b0> (a
org.apache.activemq.broker.jmx.ManagedTransportConnection)
at
org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:1140)
at
org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:249)
at
org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:215)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:346)
at
org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:95)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128>)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12.0.101/ThreadPoolExecutor.java:628<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:628>)
at
java.lang.Thread.run(java.base@11.0.12.0.101/Thread.java:829<mailto:java.base@11.0.12.0.101/Thread.java:829>)
Is there any know issue related to thread leaks/memory leak on version 5.18.3?
We have verified that the code used by the producers and consumers are closing
connections properly. We also verified that we are not seeing any major errors
in the network between the active mq broker and the clients. Could anyone
provide insight into why so many of these threads would get created and get
blocked(InactivityMonitor) and suggest possible steps to debug and find the
root cause?
Thanks in advance for the help!!
Karthik
CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains confidential
information belonging to the sender that is legally privileged and proprietary
and may be subject to protection under the law, including the Health Insurance
Portability and Accountability Act (HIPAA). If you are not the intended
recipient of this e-mail, you are prohibited from sharing, copying, or
otherwise using or disclosing its contents. If you have received this e-mail in
error, please notify the sender immediately by reply e-mail and permanently
delete this e-mail and any attachments without reading, forwarding or saving
them. Thank you.
