iamsanjay commented on PR #2276:
URL: https://github.com/apache/solr/pull/2276#issuecomment-2025117803

   Used **LogLevel** annotation to generate the DEBUG logs from Jetty. However, 
the excessive logging reduce the likelihood of reproducing the failure. So I 
restrict the logging to one class.
   ```
   @SuppressSSL // Currently, unknown why SSL does not work with this test
   @LogLevel("org.eclipse.jetty.http2.HTTP2Connection=DEBUG")
   public class TestHealthCheckHandlerLegacyMode extends SolrTestCaseJ4 {
   ```
   
   Below is the new exception observed in the logs related to terminating the 
connection.
   
   > DEBUG (qtp803109855-19) [n: c: s: r: x: t:] o.e.j.h.HTTP2Connection 
Processing session failure on 
HTTP2ServerSession@1674feca{local:/127.0.0.1:50713,remote:/127.0.0.1:50719,sendWindow=938358,recvWindow=1048576,state=[streams=0,CLOSING,goAwayRecv=null,goAwaySent=GoAwayFrame@ca473bc{847/enhance_your_calm_error/invalid_rst_stream_frame_rate},failure=java.io.IOException:
 enhance_your_calm_error/invalid_rst_stream_frame_rate]}
   >   2>           => java.io.IOException: 
enhance_your_calm_error/invalid_rst_stream_frame_rate
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:633)
   >   2> java.io.IOException: 
enhance_your_calm_error/invalid_rst_stream_frame_rate
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:633) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:2006)
 [http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:578) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:573) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Connection.onConnectionFailure(HTTP2Connection.java:303)
 [http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:218)
 [http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:210)
 [http2-common-10.0.20.jar:10.0.20]
   >   2>       **at 
org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:92) 
[http2-common-10.0.20.jar:10.0.20]**
   >   2>       at 
org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:240) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:167) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:126) 
[http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:350)
 [http2-common-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
 [jetty-util-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
 [jetty-util-10.0.20.jar:10.0.20]
   >   2>       at 
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
 [jetty-util-10.0.20.jar:10.0.20]
   
   ## Error
   
   org.eclipse.jetty.io.EofException: Close enhance_your_calm_error/ 
(invalid_rst_stream_frame_rate) 
   
   As per RFC https://datatracker.ietf.org/doc/html/rfc9113#name-error-codes
   
   **ENHANCE_YOUR_CALM (0x0b):**
   The endpoint detected that its peer is exhibiting a behavior that might be 
generating excessive load.
   
   ### RST_STREAM
   
   The Client is sending RST_STREAM frame to terminate the connection. And on 
the server side there is a rateControl code to mitigate the HTTP/2 Rapid Reset 
attack
   
   
https://github.com/jetty/jetty.project/blob/89c41b2550ed367a25d1664da8843f5a4e1019da/jetty-core/jetty-http2/jetty-http2-common/src/main/java/org/eclipse/jetty/http2/parser/ResetBodyParser.java#L88-L92
   
   ```
   private boolean onReset(ByteBuffer buffer, int error)
       {
           ResetFrame frame = new ResetFrame(getStreamId(), error);
           if (!rateControlOnEvent(frame))
               return connectionFailure(buffer, 
ErrorCode.ENHANCE_YOUR_CALM_ERROR.code, "invalid_rst_stream_frame_rate");
           reset();
           notifyReset(frame);
           return true;
       }
   ```
   
   ### The HTTP/2 Rapid Reset attack
   
   This attack is called Rapid Reset because it relies on the ability for an 
endpoint to send a RST_STREAM frame immediately after sending a request frame, 
which makes the other endpoint start working and then rapidly resets the 
request. The request is canceled, but leaves the HTTP/2 connection open. For 
more details 
https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack
   
   Jetty resolved it here https://github.com/jetty/jetty.project/issues/10679 
   
   ### What is the rate value? 
   
   In Solr, we haven't configured any value and IMO we are using the default 
value - 128.
   
   
https://github.com/jetty/jetty.project/blob/89c41b2550ed367a25d1664da8843f5a4e1019da/jetty-core/jetty-http2/jetty-http2-server/src/main/java/org/eclipse/jetty/http2/server/AbstractHTTP2ServerConnectionFactory.java#L76-L78
   
   
   ### Next?
   
   1. Find out why GetStream sending too many RST_STREAM frames?
   2. Check out the HTTP2Client parameters to reduce the excessive load on 
server.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

Reply via email to