Hi All,
Sometimes we have a lot of threads in the blocked state in Artemis (up to
400 on one of the test servers). I have checked several blocked threads,
and they look similar.
When it happens, the management console becomes slow. When I checked the
thread dump, I mentioned that there were many lines related to the
management access control and user authorization.
What can be the possible reason for that?
- too many users logged in the management console (is there any metric to
track it?)
- too many users working with the broker (number of connections, sessions,
consumers)
- too many users that perform reconnection and re-authentication frequently
- too many roles (active directory domain groups) assigned to each user,
most of which are not related to the management console access
- slow response of the domain controllers
- too many users trying to perform actions which are not permitted
or some other reason?
Can it be solved by increasing the size of the jetty thread pool, or some
other thread pool limit?
Sorry for the huge amount of text. The details:
Example blocked thread:
ID 223653
State Blocked
Name qtp132659602-223653
Native no
Suspended no
Waited Count 3720
Blocked Count 1599322
Blocked Time 20 minutes
Lock Name java.util.Collections$SynchronizedSet@54e7a889
Lock Class Name java.util.Collections$SynchronizedSet
Lock Identity Hash Code 1424468105
Waiting for lock owned by 341435 - qtp132659602-341435
Stack trace:
"qtp132659602-223653" #223653 priority:5 State:Blocked
at java.util.Collections$SynchronizedSet.equals(Collections.java:2209)
at
javax.security.auth.SubjectDomainCombiner.combine(SubjectDomainCombiner.java:203)
at
java.security.AccessControlContext.optimize(AccessControlContext.java:622)
at java.security.AccessController.getContext(AccessController.java:1009)
at
org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerGuard.currentUserHasRole(ArtemisMBeanServerGuard.java:189)
at
org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerGuard.canInvoke(ArtemisMBeanServerGuard.java:146)
at
org.apache.activemq.artemis.core.server.management.impl.HawtioSecurityControlImpl.canInvoke(HawtioSecurityControlImpl.java:102)
at
org.apache.activemq.artemis.core.server.management.impl.HawtioSecurityControlImpl.canInvoke(HawtioSecurityControlImpl.java:127)
at jdk.internal.reflect.GeneratedMethodAccessor396.invoke(null)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:569)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(null)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:569)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at javax.management.StandardMBean.invoke(StandardMBean.java:405)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:814)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:802)
at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)
at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)
at
org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)
at
org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)
at
org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161)
at
org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99)
at
org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:429)
at
org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)
at
org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)
at
org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:131)
at org.jolokia.http.AgentServlet$3.handleRequest(AgentServlet.java:493)
at org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:383)
at org.jolokia.http.AgentServlet.handle(AgentServlet.java:354)
at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:317)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:523)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
at
io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105)
at
io.hawt.web.auth.AuthenticationFilter$$Lambda$1423/0x00007fb77c82a000.run(null)
at
java.security.AccessController.executePrivileged(AccessController.java:807)
at
java.security.AccessController.doPrivileged(AccessController.java:712)
at javax.security.auth.Subject.doAs(Subject.java:439)
at
io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104)
at
io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:72)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:166)
at
io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.lang.Thread.run(Thread.java:840)
--
Regards,
Alexander
