My only concern is the same with Matteo: > Dedicating 200 threads to that would be > a massive waste of resources (CPU & memory).
And we should make it clear whether numHttpServerThread=200 could solve the existing problem like https://github.com/apache/pulsar/pull/13666 <https://github.com/apache/pulsar/pull/13666> Thanks, Yunze > 2022年2月17日 上午1:32，Lari Hotari <lhot...@apache.org> 写道： > > URL: https://github.com/apache/pulsar/issues/14329 > > Motivation > > Since Pulsar Admin API uses the blocking servlet API, all Jetty threads > might be occupied and this causes unavailability of the Pulsar Admin > API. The default value for the maximum number of threads for Jetty is > too low in Pulsar. That is the root cause of many problems where Pulsar > Admin API is unavailable when all threads are in use. > > Additional context > > - Examples of previous issues where Jetty threads have been occupied > and caused problems: #13666 #4756 #10619 > - Mailing list thread about “make async” changes: > https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l > > Implementation > > - Jetty defaults to 200 maximum threads, to prevent thread pool > starvation. Make Pulsar use the same default value by setting > numHttpServerThreads=200. > - Update the documentation for numHttpServerThreads > - The PR is already in place: > https://github.com/apache/pulsar/pull/14320 > - Set Jetty selectors and acceptors parameters to -1 so that Jetty > automatically chooses optimal values based on available cores. The > rationale is explained in the Q&A below. > - A separate PR will be made for this change. > > Q&A > > Q: What’s the reason of setting the default value to 200? If the node just > have one core, what will happen? > > These are threads. Jetty defaults to 200 maximum threads, to prevent > thread pool starvation. This is recommended when using blocking Servlet > API. The problem is that Pulsar uses the blocking servlet API and > doesn’t have a sufficient number of threads which are needed and > recommended. > > The value 200 doesn’t mean that there will be 200 threads to start with. > This is the maximum size for the thread pool. When the value is more > than 8, Jetty will start with 8 initial threads and add more threads to > the pool when all threads are occupied. > > Q: Do we need to take the number of system cores into consideration for the > maximum threads of the thread pool? > > No. Jetty is different from Netty in this aspect. In Netty, everything > should be asynchronous and “thou shall never block”. In Jetty, the > maximum number of threads for the thread pool should be set to 50-500 > threads and blocking operations are fine. > > The recommendation for the thread pool is explained in Jetty > documentation > https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_thread_pool >> Thread Pool > Configure with goal of limiting memory usage maximum > available. Typically this is >50 and <500 > > However, there are separate settings which should take the number of > available processors (cores) into account in Jetty. > > http port acceptor and selector count: > https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L88 > > https port acceptor and selector count: > https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L125 > > Jetty documentantion for acceptors: > Acceptors > The standard rule of > thumb for the number of Accepters to configure is one per CPU on a given > machine. > > Jetty documentation for selectors: > Selectors > The default number of > selectors is equal to half of the number of processors available to the > JVM, which should allow optimal performance even if all the connections > used are performing significant non-blocking work in the callback tasks. > > The settings in jetty are the “acceptor” and “selector” thread count > settings. These have been fixed to 1 in Pulsar. The acceptors and > selectors settings should be both set to -1. Jetty would pick the > recommended count based on cores in that case.
