Hi Penghui, ZIxuan Thanks for discussion on it. It's really hard for me to assess whether to add flow permits on exclusive/failover subscription. I want to share more thought on the PIP. I have noticed there is a precise dispatcher flow control on all subscription type. Please refer to https://github.com/apache/pulsar/pull/6719 and https://github.com/apache/pulsar/pull/18553. I think the flow control has same motivation which is: When there are many backlogs, bk can be protected to avoid excessive I/O. Do you think the motivation is reasonable? On the other hand, I have met difficulty when calculate the unacked count by the index subscription way. The most important point is that the markDeletedPosition is only updated after the transaction committed instead of ack. This will cause the inaccurate unacked count again and different behavior with shared subscription. It's hard to achieve accurate unacked count without more memory usage. At the same time, it maybe need a additional io to get index of message because we abstracted position as ledgerId and entryId without index. And we need an additional variable to keep the index of message if we optimize to get index without additional IO. It maybe need more change on current code. So let's assess it again. In conclution, I want to pickup the origin idea. If you have good idea please take no hesitate to share it to me. I have been blocked here for days.
Regards, Minjian PengHui Li <peng...@apache.org> 于2025年6月17日周二 06:23写道： > Hi Minjian, > > > Is it still need a > pip if we fix the origin isue in a lightweight way which just calculated > with a simple subtraction of index? > > Yes, I think so too. It would be good to have a clear conclusion explaining > why we chose the message index-based approach for Failover and Exclusive > subscriptions. > > Alternatively, you can proceed with this proposal by introducing a flag > that allows > users to enable the max unacked messages limit, with the default set to > false to prevent > any breaking changes. This way, if more users later agree that the > limitation should be enabled, > we can consider switching the default to true in the future. It provides a > smooth path to > roll out the change incrementally. > > Regards, > Penghui > > On Mon, Jun 16, 2025 at 12:19 AM Zixuan Liu <zix...@apache.org> wrote: > > > This PIP looks good to me overall. > > > > However, I'd like to point out a concern regarding exclusive and failover > > subscriptions. Currently, the unacknowledged message count is always > shown > > as 0 for these types, which can be misleading. > > > Failover and Exclusive subscriptions don't require a max unacknowledged > > > message limit > > > because they operate more simply > > > > While it's true that exclusive and failover subscriptions are simpler in > > design, I believe we should still enforce a maximum unacknowledged > message > > limit for them. Enabling this limit would ensure fairness across all > > consumer types and provide better consistency in system behavior. > > > > Unifying this behavior across all subscription types would be the right > > approach. > > > > Thanks, > > Zixuan > > > > 蔡敏健 <xiaocair...@gmail.com> 于2025年6月16日周一 12:26写道： > > > > > Thanks for detail explanation. It's really helpful. It will save lots > of > > > time if I have thought from the root like this. Since there isn't > > > a real-world scenarios to add flow permit to exclusive or failover > > > subscription, we reach an agreement that it's useless. Is it still > need a > > > pip if we fix the origin isue in a lightweight way which just > calculated > > > with a simple subtraction of index? IMO, we can reject and close this > PIP > > > and return back to fix the origin issue! > > > > > > PengHui Li <peng...@apache.org> 于2025年6月14日周六 14:06写道： > > > > > > > Hi Minjian, > > > > > > > > Thanks for sharing the proposal. Sharing my thoughts here: > > > > > > > > The maximum unacknowledged message limitation is crucial for consumer > > > load > > > > balancing > > > > and managing broker resources in Pulsar's Shared and Key_Shared > > > > subscriptions: > > > > > > > > Consumer Load Balancing: The limit prevents any single consumer from > > > being > > > > overwhelmed > > > > with more messages than it can process. When a consumer hits this > cap, > > > > Pulsar stops > > > > sending messages and, in a shared subscription, will dispatch new > > > messages > > > > to other > > > > available consumers. This effectively balances the workload across > the > > > > consumer group, > > > > preventing bottlenecks and ensuring smooth processing. > > > > > > > > Manages Broker Resources: Every message sent but not yet acknowledged > > > > consumes broker > > > > memory and CPU to track its state. By capping the number of > > > unacknowledged > > > > messages > > > > per consumer and subscription (ack state), the broker avoids being > > > swamped > > > > by slow or > > > > unresponsive consumers. This conserves vital resources, preventing > > > > performance degradation > > > > and ensuring the stability of the entire Pulsar cluster. > > > > > > > > Failover and Exclusive subscriptions don't require a max > unacknowledged > > > > message limit > > > > because they operate more simply: > > > > > > > > One Active Consumer: Only a single consumer receives messages at any > > > time. > > > > This eliminates the need to load balance or protect individual > > consumers > > > > from being overwhelmed relative to others. > > > > > > > > Cumulative Acknowledgement: The consumer can acknowledge the last > > message > > > > it processed, > > > > and Pulsar automatically marks all previous messages as acknowledged. > > > This > > > > is highly > > > > efficient and drastically reduces the number of unacknowledged > messages > > > the > > > > broker needs > > > > to track for that single consumer. > > > > > > > > I agree that there could be use cases for limiting the max unacked > > > messages > > > > for > > > > Failover or Exclusive subscriptions. If you have any real-world > > scenarios > > > > where > > > > this limitation is required, it would be great to include them in the > > > > proposal. > > > > This would help users understand when and why they should use this > > > feature. > > > > > > > > Since this proposal will change the behavior of the existing Failover > > or > > > > Exclusive > > > > subscriptions, please consider making it disabled by default to avoid > > > > breaking > > > > existing users and having a flag for users to enable it. > > > > > > > > Regards, > > > > Penghui > > > > > > > > On Tue, Jun 10, 2025 at 4:21 AM 蔡敏健 <xiaocair...@gmail.com> wrote: > > > > > > > > > Hi, Pulsar Community, > > > > > > > > > > I opened a new pip to enable consumer throttling and accurate > > > > > unacknowledged message tracking for exclusive and failover > > > subscriptions. > > > > > I'm looking > > > > > forward to your suggestions! > > > > > > > > > > link: https://github.com/apache/pulsar/pull/24400 > > > > > > > > > > Thanks, > > > > > Minjian cai > > > > > > > > > > > > > > >
