I think we are talking about 2 cases. 1. Protect the broker 2. Topic publish throughput/rate limitation
For the first case, It was originally to avoid brokers OOM, we don't have the publish buffer limitation at that time, frankly speaking, It's not a good solution for avoiding broker OOM. So, after we introduced publish buffer. If the purpose is to protect the broker, I think it's reasonable to stop reading data from the channel to wait for the broker to have the resources to handle new requests from clients, for both the producer and consumer, because the consumer requests might also consume memory. If we want to control the memory of producer and consumer separately, looks like we need to avoid reuse connection for producer and consumer. For the second case, I think a fail-fast policy makes sense. To tell the producer that reaches the max rate limitation. In this case, stopping reading from the channel is not a reasonable choice. Thanks, Penghui On Wed, Apr 13, 2022 at 3:32 PM Haiting Jiang <jianghait...@apache.org> wrote: > Hi Xiaolong, > > Similar to triggering the backlog threshold full processing method: > AFAIK, this is handled in creating producers. But with produce throttling, > it's more like a transient state. One second is throttling, the next is > not. > > IMO, throttling on connection brings more benefit for protecting broker. > As for most clients, they will do a retry when it's rate-limited. This > amplifies > the risk of crushing the broker or make it morse for heavy loaded brokers. > > So I suggest that if we want to introduce this feature. It have to come > with > some mandatory backoff policy to avoid the case mentioned above. > > Thanks, > Haiting > > On 2022/04/13 03:01:09 "r...@apache.org" wrote: > > Similar to triggering the backlog threshold full processing method: > > > > When the backlog hits the threshold, it is assumed that the user has set > > the producer_exception policy, and the Error of > > ProducerBlockedQuotaExceededError is returned to the client-side, so that > > the user can perceive this behavior. > > > > r...@apache.org <ranxiaolong...@gmail.com> 于2022年4月13日周三 10:59写道： > > > > > We also encountered the same problem. When the current limit is > triggered, > > > fast failure is an ideal way to deal with it. In this way, the > behavior of > > > the current limit itself can be notified to the client-side in time, > and > > > the user can feel the behavior of the current limit, instead of > letting the > > > user Send or receive a timeout. > > > > > > But for pulsar's own model, this thing doesn't seem to be handled very > > > well, because the current limit is based on the connection itself, as > > > Haiting and Lari said, if multiple topics reuse the same connection, > then > > > this situation will get tricky. > > > > > > So can we introduce other commands in Proto here, because the broker > knows > > > that the client has triggered various current-limiting thresholds? If > the > > > current-limiting threshold is triggered, the broker will notify the > > > client-side of the information. > > > > > > -- > > > Thanks > > > Xiaolong Ran > > > > > > Lari Hotari <l...@hotari.net> 于2022年4月8日周五 00:21写道： > > > > > >> > IMO, the more serious problem about `disableAutoread` is > > >> > Producer/Consumer isolation issue, > > >> > since they shares the connection. For example, if topic-A is > > >> rate-limited, > > >> > topic-B in the same client is also affected. > > >> > > >> Exactly. The rate-limiting won't even work properly since other rate > > >> limiters operating on the same connection will enable autoread and > this > > >> results in inconsistent behavior. > > >> > > >> -Lari > > >> > > >> On Thu, Apr 7, 2022 at 10:53 AM Haiting Jiang < > jianghait...@apache.org> > > >> wrote: > > >> > > >> > > send receipt to producer, the producer maybe timeout already. > > >> > > > >> > It's always possible that client got timeout, but the message is > > >> actually > > >> > successfully written, > > >> > causing by network issue. > > >> > > > >> > IMO, the more serious problem about `disableAutoread` is > > >> Producer/Consumer > > >> > isolation issue, > > >> > since they shares the connection. For example, if topic-A is > > >> rate-limited, > > >> > topic-B in the same > > >> > client is also affected. > > >> > > > >> > Thanks, > > >> > Haiting > > >> > > > >> > On 2022/04/07 07:13:50 Jiuming Tao wrote: > > >> > > Hi all: > > >> > > According to the issue ( > > >> > https://github.com/apache/pulsar/issues/15038), when producer > send a > > >> > message to broker and broker rate-limited at that time, the request > will > > >> > blocked in buffer, after broker read the message and send receipt to > > >> > producer, the producer maybe timeout already. > > >> > > So, in the case, do we need to provider a new flow-control > > >> > strategy(fail-fast)? It can be configurable, when broker in > rate-limit, > > >> > broker not `disableAutoread` and reply a `FlowControlException` > instead. > > >> > > > > >> > > Thanks, > > >> > > Tao Jiuming > > >> > > > >> > > > > > >
