Hi Joe, > The rate limits that are currently in place are there to protect the Pulsar > service, to manage multi-tenancy on the broker. This is not meant as a > feature to manage demand side throttling.
Yes, this is not only for demand side throttling. Consider the case of catch up consuming with multiply subscriber. Different subscriber of the same topic can be considered as different tenancies, and this subscription level dispatch rate limiter setting can server the purpose of managing multi-tenancy on the broker. For example, we have a topic with 1TB message data stored in Pulsar and two subscriber. The resource consumption is quite different if we allow two subscriber throttling both at 1000MB/s or one is 1000MB/s and the other is 10MB/s. > There is no need to add complexity For this point, I have to clarify that the complexity is not significant. The subscription level dispatch throttling is already implemented, only use the same value for all subscriptions of the same topic. Thanks, Haiting On 2022/04/12 17:58:43 Joe F wrote: > -1 > > The rate limits that are currently in place are there to protect the Pulsar > service, to manage multi-tenancy on the broker. This is not meant as a > feature to manage demand side throttling. > > In my opinion, this is best done as a client side feature. There is no > need to add complexity to the broker to manage demand side throttling. > Just throttle demand on the client side. > > > > -joe > > On Mon, Apr 11, 2022 at 7:54 PM Zixuan Liu <node...@gmail.com> wrote: > > > +1 > > > > Zixuan > > > > mattison chao <mattisonc...@gmail.com> 于2022年4月12日周二 09:24写道： > > > > > +1 > > > > > > Looks like a very useful feature. Thank you. > > > > > > Best, > > > Mattison > > > > > > On Mon, 11 Apr 2022 at 08:55, PengHui Li <peng...@apache.org> wrote: > > > > > > > +1 > > > > > > > > Penghui > > > > > > > > On Sat, Apr 9, 2022 at 4:24 PM Haiting Jiang <jianghait...@apache.org> > > > > wrote: > > > > > > > > > Hi Pulsar community, > > > > > > > > > > I created a PIP to add support for subscription level dispatch rate > > > > > limiter setting. > > > > > > > > > > The proposal can be found: > > > https://github.com/apache/pulsar/issues/15094 > > > > > > > > > > ---- > > > > > > > > > > ## Motivation > > > > > > > > > > Currently, for message dispatch rate limiter in a subscription , we > > > have > > > > 3 > > > > > level setting : > > > > > - Broker level setting: configured with > > > > > `dispatchThrottlingRatePerSubscriptionInMsg` and > > > > > `dispatchThrottlingRatePerSubscriptionInByte` in broker.conf > > > > > - Namespace level setting: configured with > > > > > > > `org.apache.pulsar.client.admin.Namespaces#setSubscriptionDispatchRate` > > > > > - Topic level setting: configured with > > > > > > > > > > > > > > `org.apache.pulsar.client.admin.TopicPolicies#setSubscriptionDispatchRate` > > > > > > > > > > As we all know, in the pub-sub messaging model, different subscriber > > of > > > > > the same topic process the messages for various purpose, and they may > > > > have > > > > > different requirement of message dispatch rate limiter. Here are some > > > use > > > > > case in my organization: > > > > > - On the client side, subscriptions have different > > > max-process-capacity. > > > > > If the dispatch rate is too large, they may crush their downstream > > > > services. > > > > > - We are billing base on the max message rate of the subscription. > > Some > > > > > are sensitive to budgets and willing to pay less for lower > > throughput. > > > > > > > > > > > > > > > ## Goal > > > > > > > > > > Support subscription level dispatch rate limiter setting. > > > > > > > > > > ## API Changes > > > > > > > > > > > > > > > 1. Add client api in org.apache.pulsar.client.admin.TopicPolicies. > > > > > ``` > > > > > void getSubscriptionDispatchRate(String topic, String sub) throws > > > > > PulsarAdminException; > > > > > void getSubscriptionDispatchRate(String topic, String sub, boolean > > > > > applied) throws PulsarAdminException; > > > > > void setSubscriptionDispatchRate(String topic, String sub, > > DispatchRate > > > > > dispatchRate) throws PulsarAdminException; > > > > > void removeSubscriptionDispatchRate(String topic, String sub) throws > > > > > PulsarAdminException; > > > > > > > > > > //And the async version of these methods. > > > > > > > > > > ``` > > > > > > > > > > 2. Add new admin API > > > > > > > > > > ``` > > > > > @PUT @DELETE @GET > > > > > @Path("/{tenant}/{namespace}/{topic}/{subName}/dispatchRate") > > > > > > > > > > ``` > > > > > > > > > > ## Implementation > > > > > > > > > > The rate limiter itself is already implemented with each > > subscription. > > > We > > > > > only need to update the rate limiter settings if subscription level > > > > config > > > > > is set. > > > > > I propose to just add a new field in > > > > > `org.apache.pulsar.common.policies.data.TopicPolicies` to store the > > > data. > > > > > ``` > > > > > private Map<String/*SubName*/, DispatchRateImpl> > > > > > subscriptionDispatchRateMap; > > > > > ``` > > > > > And subscription level rate limiter setting has higher priority than > > > > topic > > > > > level. We need to calculate the applied value when we create the > > > > > subscription or any level config is changed. > > > > > > > > > > ## Reject Alternatives > > > > > None yet. > > > > > > > > > > > > > > > Thanks, > > > > > Haiting > > > > > > > > > > > > > > >
