After my previous email, I received a question about Pulsar backpressure solutions: "Why can't we use Reactive frameworks here?"
That's a good question. Pulsar isn't implemented in a way where Reactive frameworks would be easily applicable or make sense in the Broker. (For implementing the REST API, that would be a different story.) There could also be performance reasons which make some Reactive frameworks not a good fit for implementing low-level network servers. Apple ServiceTalk is a framework which does support Reactive Streams and could be a reasonable solution for implementing network servers. quoting https://opensource.apple.com/projects/servicetalk/: "ServiceTalk is built by the team that leads the Netty project and is intended to be the next layer in the stack. ServiceTalk's asynchronous primitives follow the Reactive Streams specification and propagate backpressure/cancellation for your application. Asynchronous control flow can be challenging and not necessary for all use cases. For this reason ServiceTalk provides blocking APIs and allows you to choose if/when to evolve to asynchronous per-route in a safe manner." Reactive Streams provides a solution for non-blocking asynchronous backpressure. A plain CompletableFuture based programming model doesn't have that. Apple Servicetalk docs related to Reactive: https://apple.github.io/servicetalk//servicetalk-concurrent-api/SNAPSHOT/asynchronous-primitives.html This is a sidetrack. I think it would be more practical to focus on describing the current backpressure solution in Pulsar Broker than thinking about possible future solutions. -Lari On Wed, Sep 14, 2022 at 2:59 PM Lari Hotari <lhot...@apache.org> wrote: > > Thank you Mattison for starting the discussion. > > Just adding some more context. > > In general, every type of processing pipeline solution needs a solution > for backpressure. This could mean different types of solutions for > different goals, depending on the context. Usually a backpressure > solution is some type of flow control solution (wikipedia: > https://en.wikipedia.org/wiki/Flow_control_(data) ). > There are multiple backpressure solutions in Pulsar. > > For the broker, flow control is typically achieved using throttling > based on explicit rate limits and there's also the solution of > feedback-based flow control in the Pulsar binary protocol between the > broker and consumers, where "permits" are exchanged between the broker > and the consumer (document: > https://pulsar.apache.org/docs/develop-binary-protocol/#flow-control ) > > In Netty servers and clients, a common technical solution to > backpressure is to toggle the Netty "auto read" state of the TCP/IP > connection to pause or resume reading more input. It is also common in > Netty servers and clients to stop reading input from the TCP/IP > connection when the output buffer of the connection is full and resume > when the output buffer level goes below a threshold. > > Pulsar uses TCP/IP connection level controls for achieving backpressure > for producers. The challenge is that Pulsar can share a single TCP/IP > connection across multiple producers and consumers. Because of this, > there could be multiple producer and dispatcher rate limiters operating > on the same connection on the broker, and they will do conflicting > decisions, which results in inconsistent behavior. > > For backpressure between Pulsar and Bookkeepers, there are multiple > backpressure/flow control solutions. There's also an explicit > backpressure feature in Bookkeeper that could be enabled on server and > client side. There is an issue > https://github.com/apache/pulsar/issues/10439 to track enabling this > feature in Pulsar Broker and in the default configuration for Bookkeeper > as part of Pulsar deployments. > > Even without this explicit feature, there are forms of backpressure in > place between brokers and bookies. > > It would be helpful if the flow control between brokers and bookies > would be explained and documented in detail. This would help both Pulsar > users and developers to build understanding about the performance model > of Pulsar. > > One properly of flow control is that it can fail if the whole pipeline > isn't covered from end-to-end. Gaps in flow control could also lead to > amplification issues where the system has to cope with even more load > because retries that clients send when previous requests timeout or > fail. There could also be thundering herds issues where the system gets > high spikes of traffic when requests get buffered because of various > reasons and this could lead to cascading failures when the total load of > the system increases in unexpected ways because of amplification > effects. > > Touching upon Pulsar Admin REST API. The context for back pressure is a > lot different in the Pulsar Admin REST API. Before the PIP-149 async > changes, there was explicit backpressure in the REST API implementation. > The system could handle a limited amount of work and it would process > downstream work items one-by-one. > > With "PIP-149 Making the REST Admin API fully async" > (https://github.com/apache/pulsar/issues/14365), there are different > challenges related to backpressure. It is usually about how to limit the > in-progress work in the system. An async system will accept a lot of > work compared to the previous solution and this accepted work will get > processed in the async REST API backend eventually even when the clients > have already closed the connection and sent a new retry. One possible > solution to this issue is to limit incoming requests at the HTTP server > level with features that Jetty provides for limiting concurrency. PRs > https://github.com/apache/pulsar/pull/14353 and > https://github.com/apache/pulsar/pull/15637 added this support to > Pulsar. The values might have to be tuned to a lot lower values to > prevent issues in practice. This is not a complete solution for REST API > backend. It would be useful to also have a solution that would cancel > down stream requests that are for incoming HTTP requests that no longer > exist since the client stopped waiting for the response. The main down > stream requests are towards the metadata store. It might also be > necessary to limit the number of outstanding downstream requests. With > batching in metadata store, that might not be an issue. > > I hope it was useful to share some thoughts around backpressure in > Pulsar. I'm looking forward to learning more from others about the > design for backpressure in the Pulsar broker. Please share your thoughts > too! > > -Lari > > > > > > On Tue, Sep 13, 2022 at 7:36 AM mattison chao <mattisonc...@apache.org> wrote: > > > > Hi, All > > > > Since Pulsar has many asynchronous operations, we need to talk about > > backpressure. Because this is something to watch out for in async and I've > > seen comments about it in some other PRs. > > > > I think we need to open a separate discussion to collect the problems we > > are having and the solutions we need to provide. > > > > > > Best. > > Mattison > > > > FYI: > > https://github.com/apache/pulsar/issues/16998#issuecomment-1244833965 > > https://github.com/apache/pulsar/pull/17349#discussion_r964399426
