iamsanjay commented on PR #2276: URL: https://github.com/apache/solr/pull/2276#issuecomment-2025117803
Used **LogLevel** annotation to generate the DEBUG logs from Jetty. However, the excessive logging reduce the likelihood of reproducing the failure. So I restrict the logging to one class. ``` @SuppressSSL // Currently, unknown why SSL does not work with this test @LogLevel("org.eclipse.jetty.http2.HTTP2Connection=DEBUG") public class TestHealthCheckHandlerLegacyMode extends SolrTestCaseJ4 { ``` Below is the new exception observed in the logs related to terminating the connection. > DEBUG (qtp803109855-19) [n: c: s: r: x: t:] o.e.j.h.HTTP2Connection Processing session failure on HTTP2ServerSession@1674feca{local:/127.0.0.1:50713,remote:/127.0.0.1:50719,sendWindow=938358,recvWindow=1048576,state=[streams=0,CLOSING,goAwayRecv=null,goAwaySent=GoAwayFrame@ca473bc{847/enhance_your_calm_error/invalid_rst_stream_frame_rate},failure=java.io.IOException: enhance_your_calm_error/invalid_rst_stream_frame_rate]} > 2> => java.io.IOException: enhance_your_calm_error/invalid_rst_stream_frame_rate > 2> at org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:633) > 2> java.io.IOException: enhance_your_calm_error/invalid_rst_stream_frame_rate > 2> at org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:633) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:2006) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:578) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:573) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.HTTP2Connection.onConnectionFailure(HTTP2Connection.java:303) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:218) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:210) [http2-common-10.0.20.jar:10.0.20] > 2> **at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:92) [http2-common-10.0.20.jar:10.0.20]** > 2> at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:240) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:167) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:126) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:350) [http2-common-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455) [jetty-util-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248) [jetty-util-10.0.20.jar:10.0.20] > 2> at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) [jetty-util-10.0.20.jar:10.0.20] ## Error org.eclipse.jetty.io.EofException: Close enhance_your_calm_error/ (invalid_rst_stream_frame_rate) As per RFC https://datatracker.ietf.org/doc/html/rfc9113#name-error-codes **ENHANCE_YOUR_CALM (0x0b):** The endpoint detected that its peer is exhibiting a behavior that might be generating excessive load. ### RST_STREAM The Client is sending RST_STREAM frame to terminate the connection. And on the server side there is a rateControl code to mitigate the HTTP/2 Rapid Reset attack https://github.com/jetty/jetty.project/blob/89c41b2550ed367a25d1664da8843f5a4e1019da/jetty-core/jetty-http2/jetty-http2-common/src/main/java/org/eclipse/jetty/http2/parser/ResetBodyParser.java#L88-L92 ``` private boolean onReset(ByteBuffer buffer, int error) { ResetFrame frame = new ResetFrame(getStreamId(), error); if (!rateControlOnEvent(frame)) return connectionFailure(buffer, ErrorCode.ENHANCE_YOUR_CALM_ERROR.code, "invalid_rst_stream_frame_rate"); reset(); notifyReset(frame); return true; } ``` ### The HTTP/2 Rapid Reset attack This attack is called Rapid Reset because it relies on the ability for an endpoint to send a RST_STREAM frame immediately after sending a request frame, which makes the other endpoint start working and then rapidly resets the request. The request is canceled, but leaves the HTTP/2 connection open. For more details https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack Jetty resolved it here https://github.com/jetty/jetty.project/issues/10679 ### What is the rate value? In Solr, we haven't configured any value and IMO we are using the default value - 128. https://github.com/jetty/jetty.project/blob/89c41b2550ed367a25d1664da8843f5a4e1019da/jetty-core/jetty-http2/jetty-http2-server/src/main/java/org/eclipse/jetty/http2/server/AbstractHTTP2ServerConnectionFactory.java#L76-L78 ### Next? 1. Find out why GetStream sending too many RST_STREAM frames? 2. Check out the HTTP2Client parameters to reduce the excessive load on server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org