I created PR https://github.com/apache/pulsar/pull/14320 to set numHttpServerThreads=200 . Please review
On 2022/02/16 12:39:34 Lari Hotari wrote: > On 2022/02/16 00:58:20 PengHui Li wrote: > > Which is a sync method. Ultimately this could lead to all the pulsar-web > > thread > > blocked. we'd better not introduce blocking calls if we use AsyncResponse. > > > > > What issue did you see? Please share more context. Thanks for the > > patience. > > > > It happened very earlier > > > > Here is the issue https://github.com/apache/pulsar/issues/4756 > > And here is also a related fix https://github.com/apache/pulsar/pull/10619 > > Penghui, Thank you for the patience, and thanks for sharing more context. I > happened to send a reply before reading your message, so please bear with me. > > So finally, I understand that "the problem" is that all HTTP server threads > are blocked and this makes the Pulsar Admin API unavailable. > > To support the blocking servlet API, Jetty uses a default thread pool that > can grow to up to 200 threads > (https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) > . > However this default of 200 maximum threads is not used in Pulsar. > > The problem is that Pulsar uses a low value that assumes asynchronous API > usage: > https://github.com/apache/pulsar/blob/5c3ddc26d6e07eb0473b11b5ecc8318c1efe414b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L201-L204 > Pulsar should be using a high value (for example 200) as long as there are > blocking calls in Admin APIs. > > The mitigation to the issue of all HTTP server threads getting blocked is > setting "numHttpServerThreads=200" in broker.conf. > > Regarding the "make async" changes, It is an optimization to migrate from the > blocking servlet api to the asynchronous servlet api. This work isn't urgent > since we can simply mitigate the HTTP server threads getting blocked by > setting "numHttpServerThreads=200" in broker.conf. "the problem" will be > resolved immediately without risks of regressions that are involved in making > the sync -> async changes. > > Penghui, would you mind adding a GitHub issue for the problem where all HTTP > threads get blocked and the Pulsar Admin API stops responding? > > I can follow up with a PR which updates the default for numHttpServerThreads > to 200. This is a maximum value and Jetty starts with 8 threads. We can agree > on the default value to use in the PR. > > Thank you for the great collaboration on sharing the context and describing > the problem patiently. > > BR, > > -Lari >
