Hi Team, We have ActiveMQ Artemis server in the test environment, and sometimes management console of this server becomes unresponsive. Users report that they get connection timeout when opening web page of management console. At the same time broker is working, messages are sent/consumed. When it happens, sometimes there is an exception logged with very long stracktrace (about 330 lines and 37 Kb). I don't know if this event is correlated to unavailability of embedded web server, but it usually happens more frequently when web server becomes unresponsive. But I did not found any other messages which can clarify the reason of web server connection timeouts.
The first lines are: 2024-03-13 13:28:15,812 INFO [io.hawt.web.auth.AuthenticationFilter] Failed to invoke action / due to: java.security.PrivilegedActionException: null at java.base/java.security.AccessController.doPrivileged(AccessController.java:716) ~[?:?] at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?] at io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) [hawtio-system-2.17.7.jar:2.17.7] at io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:72) [hawtio-system-2.17.7.jar:2.17.7] Full stacktrace is attached to the message (if this mailing list supports attachments). Does embedded web server have some limited resources like connection pool, thread pool (like in application servers)? If it has, can we monitor it? Is there any possible problems with authentication or authorization? Authentication is configured using LDAPLogin module. Which information can I gather when it happens to determine what could cause this problem? Version of Artemis is latest (2.32.0), but it also happened in previous versions. -- Best regards, Aleksandr ----------------------------------- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the sender immediately either by telephone or by e-mail and delete this message and any attachment from your system. Correspondence via e-mail is for information purposes only. AO Raiffeisenbank neither makes nor accepts legally binding statements by e-mail unless otherwise agreed. -----------------------------------
2024-03-13 13:28:15,812 INFO [io.hawt.web.auth.AuthenticationFilter] Failed to invoke action / due to: java.security.PrivilegedActionException: null at java.base/java.security.AccessController.doPrivileged(AccessController.java:716) ~[?:?] at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?] at io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) [hawtio-system-2.17.7.jar:2.17.7] at io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:72) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:166) [hawtio-system-2.17.7.jar:2.17.7] at io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50) [artemis-web-2.32.0.jar:2.32.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) [jetty-security-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) [jetty-util-10.0.16.jar:10.0.16] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:827) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:584) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1051) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1123) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:254) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:858) ~[jetty-server-10.0.16.jar:10.0.16] at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:142) ~[?:?] at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:223) ~[?:?] at java.base/java.io.Writer.write(Writer.java:249) ~[?:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:103) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:67) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:73) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:135) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:67) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:73) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:135) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONArray.writeJSONString(JSONArray.java:53) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONArray.writeJSONString(JSONArray.java:59) ~[json-simple-1.1.1.jar:?] at org.jolokia.util.IoUtil.streamResponseAndClose(IoUtil.java:34) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.sendStreamingResponse(AgentServlet.java:574) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.sendResponse(AgentServlet.java:557) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.handle(AgentServlet.java:369) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:317) ~[jolokia-core-1.7.2.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:523) ~[jetty-servlet-api-4.0.6.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68) ~[hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63) ~[hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105) ~[hawtio-system-2.17.7.jar:2.17.7] at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?] ... 76 more Suppressed: org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doShutdownOutput(SslConnection.java:1338) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doClose(SslConnection.java:1434) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:258) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:227) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1204) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:827) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:584) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1051) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1123) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:254) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:858) ~[jetty-server-10.0.16.jar:10.0.16] at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:142) ~[?:?] at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:223) ~[?:?] at java.base/java.io.Writer.write(Writer.java:249) ~[?:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:103) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:67) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:73) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:135) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:67) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:73) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:135) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONArray.writeJSONString(JSONArray.java:53) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONArray.writeJSONString(JSONArray.java:59) ~[json-simple-1.1.1.jar:?] at org.jolokia.util.IoUtil.streamResponseAndClose(IoUtil.java:34) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.sendStreamingResponse(AgentServlet.java:574) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.sendResponse(AgentServlet.java:557) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.handle(AgentServlet.java:369) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:317) ~[jolokia-core-1.7.2.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:523) ~[jetty-servlet-api-4.0.6.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68) ~[hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63) ~[hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105) ~[hawtio-system-2.17.7.jar:2.17.7] at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?] at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?] at io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) [hawtio-system-2.17.7.jar:2.17.7] at io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:72) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:166) [hawtio-system-2.17.7.jar:2.17.7] at io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60) [hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50) [artemis-web-2.32.0.jar:2.32.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) [jetty-security-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) [jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) [jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) [jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) [jetty-util-10.0.16.jar:10.0.16] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:?] at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66) ~[?:?] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:217) ~[?:?] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:153) ~[?:?] at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:563) ~[?:?] at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:642) ~[?:?] at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110) ~[jetty-io-10.0.16.jar:10.0.16] ... 130 more Caused by: java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:?] at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66) ~[?:?] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:217) ~[?:?] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:153) ~[?:?] at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:563) ~[?:?] at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:642) ~[?:?] at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386) ~[jetty-io-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:827) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243) ~[jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:584) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1051) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1123) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:254) ~[jetty-server-10.0.16.jar:10.0.16] at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:858) ~[jetty-server-10.0.16.jar:10.0.16] at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132) ~[?:?] at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:142) ~[?:?] at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:223) ~[?:?] at java.base/java.io.Writer.write(Writer.java:249) ~[?:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:103) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:67) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:73) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:135) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:67) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONObject.writeJSONString(JSONObject.java:73) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONValue.writeJSONString(JSONValue.java:135) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONArray.writeJSONString(JSONArray.java:53) ~[json-simple-1.1.1.jar:?] at org.json.simple.JSONArray.writeJSONString(JSONArray.java:59) ~[json-simple-1.1.1.jar:?] at org.jolokia.util.IoUtil.streamResponseAndClose(IoUtil.java:34) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.sendStreamingResponse(AgentServlet.java:574) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.sendResponse(AgentServlet.java:557) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.handle(AgentServlet.java:369) ~[jolokia-core-1.7.2.jar:?] at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:317) ~[jolokia-core-1.7.2.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:523) ~[jetty-servlet-api-4.0.6.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68) ~[hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63) ~[hawtio-system-2.17.7.jar:2.17.7] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.16.jar:10.0.16] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.16.jar:10.0.16] at io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105) ~[hawtio-system-2.17.7.jar:2.17.7] at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?] ... 76 more