Hi Minjian, Frankly, I don't see any real cases that need to block consumers based on unacked messages on Failover or Exclusive subscription.
But I agree that always showing `0` unacked messages is confusing. Maybe we can consider removing the unacked messages from the topic stats for the Failover and Exclusive subscription. Thanks, Penghui On Fri, Jun 20, 2025 at 3:38 AM 蔡敏健 <xiaocair...@gmail.com> wrote: > Hi Penghui, ZIxuan > Thanks for discussion on it. It's really hard for me to assess whether to > add flow permits on exclusive/failover subscription. I want to share more > thought on the PIP. > I have noticed there is a precise dispatcher flow control on all > subscription type. Please refer to > https://github.com/apache/pulsar/pull/6719 and > https://github.com/apache/pulsar/pull/18553. I think the flow control has > same motivation which is: When there are many backlogs, bk can be protected > to avoid excessive I/O. Do you think the motivation is reasonable? > On the other hand, I have met difficulty when calculate the unacked count > by the index subscription way. The most important point is that the > markDeletedPosition is only updated after the transaction committed instead > of ack. This will cause the inaccurate unacked count again and different > behavior with shared subscription. It's hard to achieve accurate unacked > count without more memory usage. At the same time, it maybe need a > additional io to get index of message because we abstracted position as > ledgerId and entryId without index. And we need an additional variable to > keep the index of message if we optimize to get index without additional > IO. It maybe need more change on current code. So let's assess it again. > In conclution, I want to pickup the origin idea. If you have good idea > please take no hesitate to share it to me. I have been blocked here for > days. > > Regards, > Minjian > > PengHui Li <peng...@apache.org> 于2025年6月17日周二 06:23写道： > > > Hi Minjian, > > > > > Is it still need a > > pip if we fix the origin isue in a lightweight way which just calculated > > with a simple subtraction of index? > > > > Yes, I think so too. It would be good to have a clear conclusion > explaining > > why we chose the message index-based approach for Failover and Exclusive > > subscriptions. > > > > Alternatively, you can proceed with this proposal by introducing a flag > > that allows > > users to enable the max unacked messages limit, with the default set to > > false to prevent > > any breaking changes. This way, if more users later agree that the > > limitation should be enabled, > > we can consider switching the default to true in the future. It provides > a > > smooth path to > > roll out the change incrementally. > > > > Regards, > > Penghui > > > > On Mon, Jun 16, 2025 at 12:19 AM Zixuan Liu <zix...@apache.org> wrote: > > > > > This PIP looks good to me overall. > > > > > > However, I'd like to point out a concern regarding exclusive and > failover > > > subscriptions. Currently, the unacknowledged message count is always > > shown > > > as 0 for these types, which can be misleading. > > > > Failover and Exclusive subscriptions don't require a max > unacknowledged > > > > message limit > > > > because they operate more simply > > > > > > While it's true that exclusive and failover subscriptions are simpler > in > > > design, I believe we should still enforce a maximum unacknowledged > > message > > > limit for them. Enabling this limit would ensure fairness across all > > > consumer types and provide better consistency in system behavior. > > > > > > Unifying this behavior across all subscription types would be the right > > > approach. > > > > > > Thanks, > > > Zixuan > > > > > > 蔡敏健 <xiaocair...@gmail.com> 于2025年6月16日周一 12:26写道： > > > > > > > Thanks for detail explanation. It's really helpful. It will save lots > > of > > > > time if I have thought from the root like this. Since there isn't > > > > a real-world scenarios to add flow permit to exclusive or failover > > > > subscription, we reach an agreement that it's useless. Is it still > > need a > > > > pip if we fix the origin isue in a lightweight way which just > > calculated > > > > with a simple subtraction of index? IMO, we can reject and close this > > PIP > > > > and return back to fix the origin issue! > > > > > > > > PengHui Li <peng...@apache.org> 于2025年6月14日周六 14:06写道： > > > > > > > > > Hi Minjian, > > > > > > > > > > Thanks for sharing the proposal. Sharing my thoughts here: > > > > > > > > > > The maximum unacknowledged message limitation is crucial for > consumer > > > > load > > > > > balancing > > > > > and managing broker resources in Pulsar's Shared and Key_Shared > > > > > subscriptions: > > > > > > > > > > Consumer Load Balancing: The limit prevents any single consumer > from > > > > being > > > > > overwhelmed > > > > > with more messages than it can process. When a consumer hits this > > cap, > > > > > Pulsar stops > > > > > sending messages and, in a shared subscription, will dispatch new > > > > messages > > > > > to other > > > > > available consumers. This effectively balances the workload across > > the > > > > > consumer group, > > > > > preventing bottlenecks and ensuring smooth processing. > > > > > > > > > > Manages Broker Resources: Every message sent but not yet > acknowledged > > > > > consumes broker > > > > > memory and CPU to track its state. By capping the number of > > > > unacknowledged > > > > > messages > > > > > per consumer and subscription (ack state), the broker avoids being > > > > swamped > > > > > by slow or > > > > > unresponsive consumers. This conserves vital resources, preventing > > > > > performance degradation > > > > > and ensuring the stability of the entire Pulsar cluster. > > > > > > > > > > Failover and Exclusive subscriptions don't require a max > > unacknowledged > > > > > message limit > > > > > because they operate more simply: > > > > > > > > > > One Active Consumer: Only a single consumer receives messages at > any > > > > time. > > > > > This eliminates the need to load balance or protect individual > > > consumers > > > > > from being overwhelmed relative to others. > > > > > > > > > > Cumulative Acknowledgement: The consumer can acknowledge the last > > > message > > > > > it processed, > > > > > and Pulsar automatically marks all previous messages as > acknowledged. > > > > This > > > > > is highly > > > > > efficient and drastically reduces the number of unacknowledged > > messages > > > > the > > > > > broker needs > > > > > to track for that single consumer. > > > > > > > > > > I agree that there could be use cases for limiting the max unacked > > > > messages > > > > > for > > > > > Failover or Exclusive subscriptions. If you have any real-world > > > scenarios > > > > > where > > > > > this limitation is required, it would be great to include them in > the > > > > > proposal. > > > > > This would help users understand when and why they should use this > > > > feature. > > > > > > > > > > Since this proposal will change the behavior of the existing > Failover > > > or > > > > > Exclusive > > > > > subscriptions, please consider making it disabled by default to > avoid > > > > > breaking > > > > > existing users and having a flag for users to enable it. > > > > > > > > > > Regards, > > > > > Penghui > > > > > > > > > > On Tue, Jun 10, 2025 at 4:21 AM 蔡敏健 <xiaocair...@gmail.com> wrote: > > > > > > > > > > > Hi, Pulsar Community, > > > > > > > > > > > > I opened a new pip to enable consumer throttling and accurate > > > > > > unacknowledged message tracking for exclusive and failover > > > > subscriptions. > > > > > > I'm looking > > > > > > forward to your suggestions! > > > > > > > > > > > > link: https://github.com/apache/pulsar/pull/24400 > > > > > > > > > > > > Thanks, > > > > > > Minjian cai > > > > > > > > > > > > > > > > > > > > >
