Hi Kevin- How many IPs are on that host? 65,000 connections is a risky limit, as there are only 65,535-per-IP allowed. This number also must account for OS TCP/IP tuning around how long to leave perviously closed and completed TCP/IP sockets in a CLOSE_WAIT and TIME_WAIT states before allowing re-use of the port. You may just be topping out the OS configured limit and need to scale horizontally or add more IPs for clients to access.
Please share the full thread dump. The partial provided thread dump is missing the other half of the blocked lock attempt. Thanks, Matt Pavlovich > On Dec 13, 2024, at 2:06 PM, Karthik Venkataraman > <karthik.venkatara...@eclinicalworks.com.INVALID> wrote: > > Hello All, > > I am using an active-mq classic v 5.18.3 instance and we are running into > an issue where the active mq process shows steady increase in threads and > memory utilization and it eventually crashes. > We have many active connections(thousands) from different producers and > consumers going into this active-mq instance and we know that the > system(hardware) where this is run has enough CPU and memory to handle it. > The maximumConnections value for the NIO transport connector is set to 65000. > The heap memory for active mq is set to grow up to 12 GB. I got 2 thread > dumps from the active mq process and it shows the following 2 threads > steadily increasing: > > "ActiveMQ BrokerService[TestBroker] Task-3906" #44297 daemon prio=5 os_prio=0 > cpu=335.99ms elapsed=3286.97s tid=0x00007fb408017000 nid=0x120b9a runnable > [0x00007fb161a1e000] > java.lang.Thread.State: RUNNABLE > at > sun.nio.ch.EPoll.wait(java.base@11.0.12.0.101/Native<mailto:java.base@11.0.12.0.101/Native> > Method) > at > sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12.0.101/EPollSelectorImpl.java:120<mailto:java.base@11.0.12.0.101/EPollSelectorImpl.java:120>) > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12.0.101/SelectorImpl.java:124<mailto:java.base@11.0.12.0.101/SelectorImpl.java:124>) > - locked <0x0000000530d66360> (a sun.nio.ch.Util$2) > - locked <0x0000000530d66308> (a sun.nio.ch.EPollSelectorImpl) > at > sun.nio.ch.SelectorImpl.select(java.base@11.0.12.0.101/SelectorImpl.java:136<mailto:java.base@11.0.12.0.101/SelectorImpl.java:136>) > at > org.apache.activemq.transport.nio.NIOSSLTransport.doHandshake(NIOSSLTransport.java:447) > at > org.apache.activemq.transport.nio.NIOSSLTransport.initializeStreams(NIOSSLTransport.java:165) > at > org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:543) > at > org.apache.activemq.transport.nio.NIOTransport.doStart(NIOTransport.java:181) > at > org.apache.activemq.transport.nio.NIOSSLTransport.doStart(NIOSSLTransport.java:479) > at > org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) > at > org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:172) > at > org.apache.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:52) > at > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) > at > org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72) > at > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) > at > org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1077) > - locked <0x0000000530d5a3a0> (a > org.apache.activemq.broker.jmx.ManagedTransportConnection) > at > org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:235) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128>) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12.0.101/ThreadPoolExecutor.java:628<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:628>) > at > java.lang.Thread.run(java.base@11.0.12.0.101/Thread.java:829<mailto:java.base@11.0.12.0.101/Thread.java:829>) > > "ActiveMQ InactivityMonitor Worker 159" #44315 daemon prio=5 os_prio=0 > cpu=125.60ms elapsed=3284.27s tid=0x00007fb8cc0d0800 nid=0x120bc0 waiting for > monitor entry [0x00007fb1ef572000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.jmx.ManagedTransportConnection.stopAsync(ManagedTransportConnection.java:67) > - waiting to lock <0x0000000530d474b0> (a > org.apache.activemq.broker.jmx.ManagedTransportConnection) > at > org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:1140) > at > org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:249) > at > org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:215) > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114) > at > org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114) > at > org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173) > at > org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:346) > at > org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:95) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:1128>) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12.0.101/ThreadPoolExecutor.java:628<mailto:java.base@11.0.12.0.101/ThreadPoolExecutor.java:628>) > at > java.lang.Thread.run(java.base@11.0.12.0.101/Thread.java:829<mailto:java.base@11.0.12.0.101/Thread.java:829>) > > Is there any know issue related to thread leaks/memory leak on version 5.18.3? > > We have verified that the code used by the producers and consumers are > closing connections properly. We also verified that we are not seeing any > major errors in the network between the active mq broker and the clients. > Could anyone provide insight into why so many of these threads would get > created and get blocked(InactivityMonitor) and suggest possible steps to > debug and find the root cause? > > Thanks in advance for the help!! > > Karthik > > CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains confidential > information belonging to the sender that is legally privileged and > proprietary and may be subject to protection under the law, including the > Health Insurance Portability and Accountability Act (HIPAA). If you are not > the intended recipient of this e-mail, you are prohibited from sharing, > copying, or otherwise using or disclosing its contents. If you have received > this e-mail in error, please notify the sender immediately by reply e-mail > and permanently delete this e-mail and any attachments without reading, > forwarding or saving them. Thank you.
