Hi community I am starting a DISCUSS for "PIP-259: Make the config httpMaxRequestHeaderSize of the pulsar web server configurable".
### Motivation We have two ways to manage pulsar's resources: - By client API (Can manage some resources, such as `create topic`, `create subscriber`, and so on) - By admin API (Can manage all the resources) The `client API` has no limit on the request length. And the `admin API` has a limit on the request length(such as HTTP request line and HTTP request headers), this restriction is done by the built-in web container Jetty. Almost resources can be created by two APIs, but can only be modified and deleted by `admin API`. This causes us to be unable to modify or delete resources created by `client API` with too long a name because it exceeds Jetty's default HTTP request URI length limit. ### Goal #### 1. For web servers Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration (involves two servers: the web server in pulsar and the web server in pulsar-proxy) #### 2.For the internal client in pulsar-proxy Provide a way to modify Jetty-client's `httpClientRequestBufferSize` configuration. Since the pulsar-proxy handles HTTP requests like this: `pulsar-admin.sh` -> `proxy web server` -> `(highlight) internal client in proxy` -> `pulsar web server`. When the internal client forwards a request, it forwards the request header and the request body, and all the data passes through a buffer( we call it Buf ), like this: - Receive a request - Put the request line and request headers input to the Buf. - <strong>(highlight)</strong>Flush the Buf ( If the data in the request line and request header exceeds the length of the buf, an error is reported ) - Put the request body input to the Buf. - Flush the Buf if it is full. So we need a config to set the `buff size` of the Buf: `pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the internal client`. ### API Changes #### ServiceConfiguration.java ```java @FieldContext( category = CATEGORY_HTTP, doc = """ The maximum size in bytes of the request header. Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL. However, larger headers consume more memory and can make a server more vulnerable to denial of service attacks. """ ) private int httpMaxRequestHeaderSize = 8 * 1024; ``` #### ProxyConfiguration.java ```java @FieldContext( minValue = 1, category = CATEGORY_HTTP, doc = """ The maximum size in bytes of the request header. Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL. However, larger headers consume more memory and can make a server more vulnerable to denial of service attacks. """ ) private int httpMaxRequestHeaderSize = 8 * 1024; @FieldContext( minValue = 1, category = CATEGORY_HTTP, doc = """ the size of the buffer used to write requests to Broker. if "httpMaxRequestHeaderSize" is large than "httpClientRequestBufferSize", will set "httpClientRequestBufferSize" to the value of "httpMaxRequestHeaderSize" """ ) private int httpClientRequestBufferSize = httpMaxRequestHeaderSize; ``` ### Anything else? This change should cherry-pick into the previous branches ( includes `2.9~2.11` ) If the user uses the features `RETRY Topic` or `DLQ`, it is possible that pulsar will automatically create some topics with names that are too long and cannot be managed, the [scenario has been discussed in the email]( https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) before Thanks Yubiao Feng