Hi Bo > I have a question, why we need `httpClientRequestBufferSize ` in > proxy, can you explain in detail?
Since The Pulsar-Proxy uses the tool `jetty-client` to forward HTTP requests from users to The Pulsar-Broker, if the proxy receives a request like this: ``` GET /admin/v2/public/default/tp.....long......long..../stats HTTP/1.1 ``` The internal client with forward this request like this: ``` ByteBuf buf = allocate( config.httpClientRequestBufferSize ) buf.write(requestLine); // (Highlight) we will get a BufferOverflowException if the request line is too long. ``` Therefore, in addition to ensuring that the proxy server can receive a long request line, the internal client must also process a long request line. And this problem can be solved by making configuration `httpClientRequestBufferSize` configurable. Thanks Yubiao Feng On Thu, Mar 16, 2023 at 8:12 PM 丛搏 <bog...@apache.org> wrote: > hi yubiao : > > I have a question, why we need `httpClientRequestBufferSize ` in > proxy, can you explain in detail? > > Thanks, > Bo > > Yubiao Feng <yubiao.f...@streamnative.io.invalid> 于2023年3月16日周四 00:11写道： > > > > > Hi community > > > > I am starting a DISCUSS for "PIP-259: Make the config > > httpMaxRequestHeaderSize of the pulsar web server configurable". > > > > ### Motivation > > > > We have two ways to manage pulsar's resources: > > - By client API (Can manage some resources, such as `create topic`, > `create > > subscriber`, and so on) > > - By admin API (Can manage all the resources) > > > > The `client API` has no limit on the request length. And the `admin API` > > has a limit on the request length(such as HTTP request line and HTTP > > request headers), this restriction is done by the built-in web container > > Jetty. > > > > Almost resources can be created by two APIs, but can only be modified and > > deleted by `admin API`. This causes us to be unable to modify or delete > > resources created by `client API` with too long a name because it exceeds > > Jetty's default HTTP request URI length limit. > > > > ### Goal > > > > #### 1. For web servers > > Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration > > (involves two servers: the web server in pulsar and the web server in > > pulsar-proxy) > > > > #### 2.For the internal client in pulsar-proxy > > Provide a way to modify Jetty-client's `httpClientRequestBufferSize` > > configuration. > > > > Since the pulsar-proxy handles HTTP requests like this： `pulsar-admin.sh` > > -> `proxy web server` -> `(highlight) internal client in proxy` -> > `pulsar > > web server`. > > > > When the internal client forwards a request, it forwards the request > header > > and the request body, and all the data passes through a buffer( we call > it > > Buf ), like this: > > - Receive a request > > - Put the request line and request headers input to the Buf. > > - <strong>(highlight)</strong>Flush the Buf ( If the data in the request > > line and request header exceeds the length of the buf, an error is > reported > > ) > > - Put the request body input to the Buf. > > - Flush the Buf if it is full. > > > > So we need a config to set the `buff size` of the Buf: > > `pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the > > internal client`. > > > > ### API Changes > > > > #### ServiceConfiguration.java > > ```java > > @FieldContext( > > category = CATEGORY_HTTP, > > doc = """ > > The maximum size in bytes of the request header. > > Larger headers will allow for more and/or larger cookies > > plus larger form content encoded in a URL. > > However, larger headers consume more memory and can make > a > > server more vulnerable to denial of service > > attacks. > > """ > > ) > > private int httpMaxRequestHeaderSize = 8 * 1024; > > ``` > > > > #### ProxyConfiguration.java > > > > ```java > > @FieldContext( > > minValue = 1, > > category = CATEGORY_HTTP, > > doc = """ > > The maximum size in bytes of the request header. > > Larger headers will allow for more and/or larger cookies > > plus larger form content encoded in a URL. > > However, larger headers consume more memory and can make > a > > server more vulnerable to denial of service > > attacks. > > """ > > ) > > private int httpMaxRequestHeaderSize = 8 * 1024; > > > > @FieldContext( > > minValue = 1, > > category = CATEGORY_HTTP, > > doc = """ > > the size of the buffer used to write requests to Broker. > > if "httpMaxRequestHeaderSize" is large than > > "httpClientRequestBufferSize", will set > > "httpClientRequestBufferSize" to the value of > > "httpMaxRequestHeaderSize" > > """ > > ) > > private int httpClientRequestBufferSize = httpMaxRequestHeaderSize; > > ``` > > > > ### Anything else? > > > > This change should cherry-pick into the previous branches ( includes > > `2.9~2.11` ) > > > > If the user uses the features `RETRY Topic` or `DLQ`, it is possible that > > pulsar will automatically create some topics with names that are too long > > and cannot be managed, the [scenario has been discussed in the email]( > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) before > > > > > > Thanks > > Yubiao Feng >
