iamsanjay commented on PR #2276:
URL: https://github.com/apache/solr/pull/2276#issuecomment-2025117803
Used **LogLevel** annotation to generate the DEBUG logs from Jetty. However,
the excessive logging reduce the likelihood of reproducing the failure. So I
restrict the logging to one class.
```
@SuppressSSL // Currently, unknown why SSL does not work with this test
@LogLevel("org.eclipse.jetty.http2.HTTP2Connection=DEBUG")
public class TestHealthCheckHandlerLegacyMode extends SolrTestCaseJ4 {
```
Below is the new exception observed in the logs related to terminating the
connection.
> DEBUG (qtp803109855-19) [n: c: s: r: x: t:] o.e.j.h.HTTP2Connection
Processing session failure on
HTTP2ServerSession@1674feca{local:/127.0.0.1:50713,remote:/127.0.0.1:50719,sendWindow=938358,recvWindow=1048576,state=[streams=0,CLOSING,goAwayRecv=null,goAwaySent=GoAwayFrame@ca473bc{847/enhance_your_calm_error/invalid_rst_stream_frame_rate},failure=java.io.IOException:
enhance_your_calm_error/invalid_rst_stream_frame_rate]}
> 2> => java.io.IOException:
enhance_your_calm_error/invalid_rst_stream_frame_rate
> 2> at
org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:633)
> 2> java.io.IOException:
enhance_your_calm_error/invalid_rst_stream_frame_rate
> 2> at
org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:633)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:2006)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:578)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:573)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.HTTP2Connection.onConnectionFailure(HTTP2Connection.java:303)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:218)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:210)
[http2-common-10.0.20.jar:10.0.20]
> 2> **at
org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:92)
[http2-common-10.0.20.jar:10.0.20]**
> 2> at
org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:240)
[http2-common-10.0.20.jar:10.0.20]
> 2> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:167)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:126)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:350)
[http2-common-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
[jetty-util-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
[jetty-util-10.0.20.jar:10.0.20]
> 2> at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
[jetty-util-10.0.20.jar:10.0.20]
## Error
org.eclipse.jetty.io.EofException: Close enhance_your_calm_error/
(invalid_rst_stream_frame_rate)
As per RFC https://datatracker.ietf.org/doc/html/rfc9113#name-error-codes
**ENHANCE_YOUR_CALM (0x0b):**
The endpoint detected that its peer is exhibiting a behavior that might be
generating excessive load.
### RST_STREAM
The Client is sending RST_STREAM frame to terminate the connection. And on
the server side there is a rateControl code to mitigate the HTTP/2 Rapid Reset
attack
https://github.com/jetty/jetty.project/blob/89c41b2550ed367a25d1664da8843f5a4e1019da/jetty-core/jetty-http2/jetty-http2-common/src/main/java/org/eclipse/jetty/http2/parser/ResetBodyParser.java#L88-L92
```
private boolean onReset(ByteBuffer buffer, int error)
{
ResetFrame frame = new ResetFrame(getStreamId(), error);
if (!rateControlOnEvent(frame))
return connectionFailure(buffer,
ErrorCode.ENHANCE_YOUR_CALM_ERROR.code, "invalid_rst_stream_frame_rate");
reset();
notifyReset(frame);
return true;
}
```
### The HTTP/2 Rapid Reset attack
This attack is called Rapid Reset because it relies on the ability for an
endpoint to send a RST_STREAM frame immediately after sending a request frame,
which makes the other endpoint start working and then rapidly resets the
request. The request is canceled, but leaves the HTTP/2 connection open. For
more details
https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack
Jetty resolved it here https://github.com/jetty/jetty.project/issues/10679
### What is the rate value?
In Solr, we haven't configured any value and IMO we are using the default
value - 128.
https://github.com/jetty/jetty.project/blob/89c41b2550ed367a25d1664da8843f5a4e1019da/jetty-core/jetty-http2/jetty-http2-server/src/main/java/org/eclipse/jetty/http2/server/AbstractHTTP2ServerConnectionFactory.java#L76-L78
### Next?
1. Find out why GetStream sending too many RST_STREAM frames?
2. Check out the HTTP2Client parameters to reduce the excessive load on
server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
