Thank you Justin for the quick response. I checked the thread "qtp132659602-341435" - it looks exactly the same (and also in the blocked state). Or maybe it has been changed while I searched for it by name. At the moment the load is not so high, and there are almost no blocked threads. I'll try to analyze the thread dump later.
This is a test instance of ActiveMQ Artemis where users can log on to the management console and check their queues. When logged on, each user performs a lot of operations to get information from management (mostly list*, get* and is* operations). It is performed when the browser tab with the management console is active. The production instance has a similar workload. I don't know the exact number of users, but I can check it by examining audit logs of the broker. вт, 28 янв. 2025 г. в 23:11, Justin Bertram <jbert...@apache.org>: > Based on the stack-trace you shared, there's a remote client making an > HTTPS connection for a management operation of some kind via Jolokia (i.e. > the JMX-HTTP bridge). This thread is blocked waiting on another thread to > release a lock that it needs. In order to diagnose the issue we first need > the stack-trace of whatever thread is holding that lock. It looks like the > lock is being held by a thread named "qtp132659602-341435". Can you provide > the appropriate stack-trace? > > Can you outline the specific use-case involved here and what a "successful" > outcome would be? > > Regarding your specific questions... > > > too many users logged in the management console... > > It's possible this is causing problems. > > > is there any metric to track it? > > I don't think so. > > > too many users working with the broker (number of connections, sessions, > consumers) > > The issue is related specifically to HTTP connections related to > management. Messaging connections/users are not involved based on the > information you've provided so far. > > > too many users that perform reconnection and re-authentication frequently > > If those users are using HTTP for management operations then that might be > contributing. > > > too many roles (active directory domain groups) assigned to each user, > most of which are not related to the management console access > > I don't see any evidence of that. > > > slow response of the domain controllers > > I don't see any evidence of that. > > > too many users trying to perform actions which are not permitted or some > other reason? > > It's possible this is contributing to the issue assuming the users are > using HTTP for management. > > > Can it be solved by increasing the size of the jetty thread pool, or some > other thread pool limit? > > There's not enough information to answer that question at this point. If > the problem is contention on a single resource then increasing the size of > a related thread pool could make the situation worse. > > > Justin > > On Tue, Jan 28, 2025 at 1:45 PM Alexander Milovidov <milovid...@gmail.com> > wrote: > > > Hi All, > > > > Sometimes we have a lot of threads in the blocked state in Artemis (up to > > 400 on one of the test servers). I have checked several blocked threads, > > and they look similar. > > When it happens, the management console becomes slow. When I checked the > > thread dump, I mentioned that there were many lines related to the > > management access control and user authorization. > > > > What can be the possible reason for that? > > - too many users logged in the management console (is there any metric to > > track it?) > > - too many users working with the broker (number of connections, > sessions, > > consumers) > > - too many users that perform reconnection and re-authentication > frequently > > - too many roles (active directory domain groups) assigned to each user, > > most of which are not related to the management console access > > - slow response of the domain controllers > > - too many users trying to perform actions which are not permitted > > or some other reason? > > > > Can it be solved by increasing the size of the jetty thread pool, or some > > other thread pool limit? > > > > Sorry for the huge amount of text. The details: > > > > Example blocked thread: > > ID 223653 > > State Blocked > > Name qtp132659602-223653 > > Native no > > Suspended no > > Waited Count 3720 > > Blocked Count 1599322 > > Blocked Time 20 minutes > > Lock Name java.util.Collections$SynchronizedSet@54e7a889 > > Lock Class Name java.util.Collections$SynchronizedSet > > Lock Identity Hash Code 1424468105 > > Waiting for lock owned by 341435 - qtp132659602-341435 > > > > Stack trace: > > "qtp132659602-223653" #223653 priority:5 State:Blocked > > at > java.util.Collections$SynchronizedSet.equals(Collections.java:2209) > > at > > > > > javax.security.auth.SubjectDomainCombiner.combine(SubjectDomainCombiner.java:203) > > at > > > java.security.AccessControlContext.optimize(AccessControlContext.java:622) > > at > > java.security.AccessController.getContext(AccessController.java:1009) > > at > > > > > org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerGuard.currentUserHasRole(ArtemisMBeanServerGuard.java:189) > > at > > > > > org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerGuard.canInvoke(ArtemisMBeanServerGuard.java:146) > > at > > > > > org.apache.activemq.artemis.core.server.management.impl.HawtioSecurityControlImpl.canInvoke(HawtioSecurityControlImpl.java:102) > > at > > > > > org.apache.activemq.artemis.core.server.management.impl.HawtioSecurityControlImpl.canInvoke(HawtioSecurityControlImpl.java:127) > > at jdk.internal.reflect.GeneratedMethodAccessor396.invoke(null) > > at > > > > > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:569) > > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > > at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(null) > > at > > > > > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:569) > > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262) > > at > > > > > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > > at > > > > > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > > at > > > > > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > > at javax.management.StandardMBean.invoke(StandardMBean.java:405) > > at > > > > > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:814) > > at > > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:802) > > at > org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98) > > at > org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40) > > at > > > > > org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89) > > at > > > > > org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109) > > at > > > > > org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161) > > at > > > > > org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99) > > at > > > > > org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:429) > > at > > org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158) > > at > > > > > org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197) > > at > > > > > org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:131) > > at > org.jolokia.http.AgentServlet$3.handleRequest(AgentServlet.java:493) > > at > org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:383) > > at org.jolokia.http.AgentServlet.handle(AgentServlet.java:354) > > at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:317) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:523) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) > > at > > > > > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) > > at > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) > > at > > > > > io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > > io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > > > > io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105) > > at > > > > > io.hawt.web.auth.AuthenticationFilter$$Lambda$1423/0x00007fb77c82a000.run(null) > > at > > > java.security.AccessController.executePrivileged(AccessController.java:807) > > at > > java.security.AccessController.doPrivileged(AccessController.java:712) > > at javax.security.auth.Subject.doAs(Subject.java:439) > > at > > > > > io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) > > at > > > > > io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:72) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > > io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:166) > > at > > > io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > > > > org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50) > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) > > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) > > at > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) > > at > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) > > at > > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) > > at > > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) > > at > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > > at org.eclipse.jetty.server.Server.handle(Server.java:563) > > at > > > > > org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) > > at > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) > > at > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) > > at > > org.eclipse.jetty.io > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) > > at > > > > > org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) > > at > > org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) > > at > > > org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) > > at > > org.eclipse.jetty.io > > .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) > > at > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) > > at > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) > > at > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) > > at > > > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) > > at > > > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) > > at java.lang.Thread.run(Thread.java:840) > > > > -- > > Regards, > > Alexander > > >
