hi yubiao : I have a question, why we need `httpClientRequestBufferSize ` in proxy, can you explain in detail?
Thanks, Bo Yubiao Feng <yubiao.f...@streamnative.io.invalid> 于2023年3月16日周四 00:11写道： > > Hi community > > I am starting a DISCUSS for "PIP-259: Make the config > httpMaxRequestHeaderSize of the pulsar web server configurable". > > ### Motivation > > We have two ways to manage pulsar's resources: > - By client API (Can manage some resources, such as `create topic`, `create > subscriber`, and so on) > - By admin API (Can manage all the resources) > > The `client API` has no limit on the request length. And the `admin API` > has a limit on the request length(such as HTTP request line and HTTP > request headers), this restriction is done by the built-in web container > Jetty. > > Almost resources can be created by two APIs, but can only be modified and > deleted by `admin API`. This causes us to be unable to modify or delete > resources created by `client API` with too long a name because it exceeds > Jetty's default HTTP request URI length limit. > > ### Goal > > #### 1. For web servers > Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration > (involves two servers: the web server in pulsar and the web server in > pulsar-proxy) > > #### 2.For the internal client in pulsar-proxy > Provide a way to modify Jetty-client's `httpClientRequestBufferSize` > configuration. > > Since the pulsar-proxy handles HTTP requests like this： `pulsar-admin.sh` > -> `proxy web server` -> `(highlight) internal client in proxy` -> `pulsar > web server`. > > When the internal client forwards a request, it forwards the request header > and the request body, and all the data passes through a buffer( we call it > Buf ), like this: > - Receive a request > - Put the request line and request headers input to the Buf. > - <strong>(highlight)</strong>Flush the Buf ( If the data in the request > line and request header exceeds the length of the buf, an error is reported > ) > - Put the request body input to the Buf. > - Flush the Buf if it is full. > > So we need a config to set the `buff size` of the Buf: > `pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the > internal client`. > > ### API Changes > > #### ServiceConfiguration.java > ```java > @FieldContext( > category = CATEGORY_HTTP, > doc = """ > The maximum size in bytes of the request header. > Larger headers will allow for more and/or larger cookies > plus larger form content encoded in a URL. > However, larger headers consume more memory and can make a > server more vulnerable to denial of service > attacks. > """ > ) > private int httpMaxRequestHeaderSize = 8 * 1024; > ``` > > #### ProxyConfiguration.java > > ```java > @FieldContext( > minValue = 1, > category = CATEGORY_HTTP, > doc = """ > The maximum size in bytes of the request header. > Larger headers will allow for more and/or larger cookies > plus larger form content encoded in a URL. > However, larger headers consume more memory and can make a > server more vulnerable to denial of service > attacks. > """ > ) > private int httpMaxRequestHeaderSize = 8 * 1024; > > @FieldContext( > minValue = 1, > category = CATEGORY_HTTP, > doc = """ > the size of the buffer used to write requests to Broker. > if "httpMaxRequestHeaderSize" is large than > "httpClientRequestBufferSize", will set > "httpClientRequestBufferSize" to the value of > "httpMaxRequestHeaderSize" > """ > ) > private int httpClientRequestBufferSize = httpMaxRequestHeaderSize; > ``` > > ### Anything else? > > This change should cherry-pick into the previous branches ( includes > `2.9~2.11` ) > > If the user uses the features `RETRY Topic` or `DLQ`, it is possible that > pulsar will automatically create some topics with names that are too long > and cannot be managed, the [scenario has been discussed in the email]( > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) before > > > Thanks > Yubiao Feng
