You would need to check the same thread dump you were inspecting previously (assuming you actually acquired a thread dump and weren't just looking at a dynamic list of threads). Threads are in constant flux so looking at "qtp132659602-341435" later almost certainly won't provide any meaningful information.
The fact that the number of blocked threads decreased indicates a resource contention issue and not a soft deadlock or other more serious problem. As one might expect, as the number of management users increases so will the load. It's possible that whatever is causing the contention can be modified or it may just be you need to limit management access. There's no way to know without more details on the source of the contention. Justin On Tue, Jan 28, 2025 at 3:20 PM Alexander Milovidov <milovid...@gmail.com> wrote: > Thank you Justin for the quick response. > > I checked the thread "qtp132659602-341435" - it looks exactly the same (and > also in the blocked state). > Or maybe it has been changed while I searched for it by name. At the moment > the load is not so high, and there are almost no blocked threads. I'll try > to analyze the thread dump later. > > This is a test instance of ActiveMQ Artemis where users can log on to the > management console and check their queues. When logged on, each user > performs a lot of operations to get information from management (mostly > list*, get* and is* operations). It is performed when the browser tab with > the management console is active. The production instance has a similar > workload. I don't know the exact number of users, but I can check it by > examining audit logs of the broker. > > > > вт, 28 янв. 2025 г. в 23:11, Justin Bertram <jbert...@apache.org>: > > > Based on the stack-trace you shared, there's a remote client making an > > HTTPS connection for a management operation of some kind via Jolokia > (i.e. > > the JMX-HTTP bridge). This thread is blocked waiting on another thread to > > release a lock that it needs. In order to diagnose the issue we first > need > > the stack-trace of whatever thread is holding that lock. It looks like > the > > lock is being held by a thread named "qtp132659602-341435". Can you > provide > > the appropriate stack-trace? > > > > Can you outline the specific use-case involved here and what a > "successful" > > outcome would be? > > > > Regarding your specific questions... > > > > > too many users logged in the management console... > > > > It's possible this is causing problems. > > > > > is there any metric to track it? > > > > I don't think so. > > > > > too many users working with the broker (number of connections, > sessions, > > consumers) > > > > The issue is related specifically to HTTP connections related to > > management. Messaging connections/users are not involved based on the > > information you've provided so far. > > > > > too many users that perform reconnection and re-authentication > frequently > > > > If those users are using HTTP for management operations then that might > be > > contributing. > > > > > too many roles (active directory domain groups) assigned to each user, > > most of which are not related to the management console access > > > > I don't see any evidence of that. > > > > > slow response of the domain controllers > > > > I don't see any evidence of that. > > > > > too many users trying to perform actions which are not permitted or > some > > other reason? > > > > It's possible this is contributing to the issue assuming the users are > > using HTTP for management. > > > > > Can it be solved by increasing the size of the jetty thread pool, or > some > > other thread pool limit? > > > > There's not enough information to answer that question at this point. If > > the problem is contention on a single resource then increasing the size > of > > a related thread pool could make the situation worse. > > > > > > Justin > > > > On Tue, Jan 28, 2025 at 1:45 PM Alexander Milovidov < > milovid...@gmail.com> > > wrote: > > > > > Hi All, > > > > > > Sometimes we have a lot of threads in the blocked state in Artemis (up > to > > > 400 on one of the test servers). I have checked several blocked > threads, > > > and they look similar. > > > When it happens, the management console becomes slow. When I checked > the > > > thread dump, I mentioned that there were many lines related to the > > > management access control and user authorization. > > > > > > What can be the possible reason for that? > > > - too many users logged in the management console (is there any metric > to > > > track it?) > > > - too many users working with the broker (number of connections, > > sessions, > > > consumers) > > > - too many users that perform reconnection and re-authentication > > frequently > > > - too many roles (active directory domain groups) assigned to each > user, > > > most of which are not related to the management console access > > > - slow response of the domain controllers > > > - too many users trying to perform actions which are not permitted > > > or some other reason? > > > > > > Can it be solved by increasing the size of the jetty thread pool, or > some > > > other thread pool limit? > > > > > > Sorry for the huge amount of text. The details: > > > > > > Example blocked thread: > > > ID 223653 > > > State Blocked > > > Name qtp132659602-223653 > > > Native no > > > Suspended no > > > Waited Count 3720 > > > Blocked Count 1599322 > > > Blocked Time 20 minutes > > > Lock Name java.util.Collections$SynchronizedSet@54e7a889 > > > Lock Class Name java.util.Collections$SynchronizedSet > > > Lock Identity Hash Code 1424468105 > > > Waiting for lock owned by 341435 - qtp132659602-341435 > > > > > > Stack trace: > > > "qtp132659602-223653" #223653 priority:5 State:Blocked > > > at > > java.util.Collections$SynchronizedSet.equals(Collections.java:2209) > > > at > > > > > > > > > javax.security.auth.SubjectDomainCombiner.combine(SubjectDomainCombiner.java:203) > > > at > > > > > > java.security.AccessControlContext.optimize(AccessControlContext.java:622) > > > at > > > java.security.AccessController.getContext(AccessController.java:1009) > > > at > > > > > > > > > org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerGuard.currentUserHasRole(ArtemisMBeanServerGuard.java:189) > > > at > > > > > > > > > org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerGuard.canInvoke(ArtemisMBeanServerGuard.java:146) > > > at > > > > > > > > > org.apache.activemq.artemis.core.server.management.impl.HawtioSecurityControlImpl.canInvoke(HawtioSecurityControlImpl.java:102) > > > at > > > > > > > > > org.apache.activemq.artemis.core.server.management.impl.HawtioSecurityControlImpl.canInvoke(HawtioSecurityControlImpl.java:127) > > > at jdk.internal.reflect.GeneratedMethodAccessor396.invoke(null) > > > at > > > > > > > > > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:569) > > > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > > > at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(null) > > > at > > > > > > > > > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:569) > > > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262) > > > at > > > > > > > > > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > > > at > > > > > > > > > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > > > at > > > > > > > > > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > > > at > com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > > > at > com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > > > at javax.management.StandardMBean.invoke(StandardMBean.java:405) > > > at > > > > > > > > > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:814) > > > at > > > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:802) > > > at > > org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98) > > > at > > org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40) > > > at > > > > > > > > > org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89) > > > at > > > > > > > > > org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109) > > > at > > > > > > > > > org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161) > > > at > > > > > > > > > org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99) > > > at > > > > > > > > > org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:429) > > > at > > > > org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158) > > > at > > > > > > > > > org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197) > > > at > > > > > > > > > org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:131) > > > at > > org.jolokia.http.AgentServlet$3.handleRequest(AgentServlet.java:493) > > > at > > org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:383) > > > at org.jolokia.http.AgentServlet.handle(AgentServlet.java:354) > > > at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:317) > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:523) > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) > > > at > > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) > > > at > > > > > > > > > io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > > > > io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > > > > > > > io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105) > > > at > > > > > > > > > io.hawt.web.auth.AuthenticationFilter$$Lambda$1423/0x00007fb77c82a000.run(null) > > > at > > > > > > java.security.AccessController.executePrivileged(AccessController.java:807) > > > at > > > java.security.AccessController.doPrivileged(AccessController.java:712) > > > at javax.security.auth.Subject.doAs(Subject.java:439) > > > at > > > > > > > > > io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) > > > at > > > > > > > > > io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:72) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > > > > io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:166) > > > at > > > > > > io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > > > > > > > org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50) > > > at > > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > > at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > > at > > > > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) > > > at > > > > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) > > > at > > > > > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) > > > at > > > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) > > > at > > > > > > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) > > > at > > > > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) > > > at > > > > > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > > > at org.eclipse.jetty.server.Server.handle(Server.java:563) > > > at > > > > > > > > > org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) > > > at > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) > > > at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) > > > at > > > > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) > > > at > > > org.eclipse.jetty.io > > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) > > > at org.eclipse.jetty.io > .FillInterest.fillable(FillInterest.java:100) > > > at > > > > > > > > > org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) > > > at > > > > org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) > > > at > > > > > > org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) > > > at org.eclipse.jetty.io > .FillInterest.fillable(FillInterest.java:100) > > > at > > > org.eclipse.jetty.io > > > .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) > > > at > > > > > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) > > > at java.lang.Thread.run(Thread.java:840) > > > > > > -- > > > Regards, > > > Alexander > > > > > >
