Re: [DISCUSS] Back pressure in Apache Pulsar

2022-12-02 Thread Asaf Mesika
Mattison, can you provide more context here? 1st link describes a complicated situation in which the BK node Pulsar is working against is overwhelmed, and we would like to transmit this back-pressure back to consumers/producers. This creates an issue as you need to notify the load balancer of this

Re: [DISCUSS] Back pressure in Apache Pulsar

2022-09-14 Thread Lari Hotari
After my previous email, I received a question about Pulsar backpressure solutions: "Why can't we use Reactive frameworks here?" That's a good question. Pulsar isn't implemented in a way where Reactive frameworks would be easily applicable or make sense in the Broker. (For implementing the REST A

Re: [DISCUSS] Back pressure in Apache Pulsar

2022-09-14 Thread Lari Hotari
Thank you Mattison for starting the discussion. Just adding some more context. In general, every type of processing pipeline solution needs a solution for backpressure. This could mean different types of solutions for different goals, depending on the context. Usually a backpressure solution is s