URL: https://github.com/apache/pulsar/issues/14329
Motivation
Since Pulsar Admin API uses the blocking servlet API, all Jetty threads
might be occupied and this causes unavailability of the Pulsar Admin
API. The default value for the maximum number of threads for Jetty is
too low in Pulsar. That is the root cause of many problems where Pulsar
Admin API is unavailable when all threads are in use.
Additional context
- Examples of previous issues where Jetty threads have been occupied
and caused problems: #13666 #4756 #10619
- Mailing list thread about “make async” changes:
https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l
Implementation
- Jetty defaults to 200 maximum threads, to prevent thread pool
starvation. Make Pulsar use the same default value by setting
numHttpServerThreads=200.
- Update the documentation for numHttpServerThreads
- The PR is already in place:
https://github.com/apache/pulsar/pull/14320
- Set Jetty selectors and acceptors parameters to -1 so that Jetty
automatically chooses optimal values based on available cores. The
rationale is explained in the Q&A below.
- A separate PR will be made for this change.
Q&A
Q: What’s the reason of setting the default value to 200? If the node just have
one core, what will happen?
These are threads. Jetty defaults to 200 maximum threads, to prevent
thread pool starvation. This is recommended when using blocking Servlet
API. The problem is that Pulsar uses the blocking servlet API and
doesn’t have a sufficient number of threads which are needed and
recommended.
The value 200 doesn’t mean that there will be 200 threads to start with.
This is the maximum size for the thread pool. When the value is more
than 8, Jetty will start with 8 initial threads and add more threads to
the pool when all threads are occupied.
Q: Do we need to take the number of system cores into consideration for the
maximum threads of the thread pool?
No. Jetty is different from Netty in this aspect. In Netty, everything
should be asynchronous and “thou shall never block”. In Jetty, the
maximum number of threads for the thread pool should be set to 50-500
threads and blocking operations are fine.
The recommendation for the thread pool is explained in Jetty
documentation
https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_thread_pool
> Thread Pool > Configure with goal of limiting memory usage maximum
available. Typically this is >50 and <500
However, there are separate settings which should take the number of
available processors (cores) into account in Jetty.
http port acceptor and selector count:
https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L88
https port acceptor and selector count:
https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L125
Jetty documentantion for acceptors: > Acceptors > The standard rule of
thumb for the number of Accepters to configure is one per CPU on a given
machine.
Jetty documentation for selectors: > Selectors > The default number of
selectors is equal to half of the number of processors available to the
JVM, which should allow optimal performance even if all the connections
used are performing significant non-blocking work in the callback tasks.
The settings in jetty are the “acceptor” and “selector” thread count
settings. These have been fixed to 1 in Pulsar. The acceptors and
selectors settings should be both set to -1. Jetty would pick the
recommended count based on cores in that case.
