Re: [DISCUSS] Back pressure in Apache Pulsar

2022-12-02 Thread Asaf Mesika
Mattison, can you provide more context here? 1st link describes a complicated situation in which the BK node Pulsar is working against is overwhelmed, and we would like to transmit this back-pressure back to consumers/producers. This creates an issue as you need to notify the load balancer of this

Re: [DISCUSS] Back pressure in Apache Pulsar

2022-09-14 Thread Lari Hotari
After my previous email, I received a question about Pulsar backpressure solutions: "Why can't we use Reactive frameworks here?" That's a good question. Pulsar isn't implemented in a way where Reactive frameworks would be easily applicable or make sense in the Broker. (For implementing the REST A

Re: [DISCUSS] Back pressure in Apache Pulsar

2022-09-14 Thread Lari Hotari
Thank you Mattison for starting the discussion. Just adding some more context. In general, every type of processing pipeline solution needs a solution for backpressure. This could mean different types of solutions for different goals, depending on the context. Usually a backpressure solution is s

[DISCUSS] Back pressure in Apache Pulsar

2022-09-12 Thread mattison chao
Hi, All Since Pulsar has many asynchronous operations, we need to talk about backpressure. Because this is something to watch out for in async and I've seen comments about it in some other PRs. I think we need to open a separate discussion to collect the problems we are having and the solutions w