Hi PengHui, Thank you for your suggestion. I really appreciate your input on this, and I understand that without context, your proposed approach would indeed be the way forward.
In this particular case, though, there are compelling reasons why PIP-379 suggests replacing the "recently joined consumers" approach with "draining hashes" in Pulsar sooner rather than later. Unlike PIP-192 and PIP-195, where we were improving already functional implementations, here we're looking at addressing what appears to be a core functionality issue. An additional reason is that PIP-282 is already implemented in master branch and it adds significant runtime state complexity compared to the previous solution. PIP-282 didn't have a way to opt-out either and it is currently included in master branch. I'm more than happy to discuss the process aspects in more detail if you have any concerns about the proposed approach from that perspective. I'd really value your further feedback on the content and proposed design of PIP-379. Your insights would be incredibly helpful as we work on refining this solution. Looking forward to your thoughts! -Lari On 2024/09/15 23:52:48 PengHui Li wrote: > Hi Lari, > > I recommend creating a new implementation rather than directly replacing > the existing one. > This approach aligns with how we’ve handled several proposals in the past > and allows us to maintain stability while introducing improvements > > - PIP-192: New Pulsar Broker Load Balancer > - PIP-195: New bucket based delayed message tracker > > Once the new implementation proves to be stable, we can switch the default > implementation of the Key_Shared subscription to the new ‘draining hashes’ > solution. > > On Sat, Sep 14, 2024 at 8:40 AM Enrico Olivelli <eolive...@gmail.com> wrote: > > > Awesome proposal, no questions from my side > > > > +1 > > > > Enrico > > > > Il giorno sab 14 set 2024 alle ore 16:21 Lari Hotari <lhot...@apache.org> > > ha scritto: > > > > > Dear Pulsar Community, > > > > > > I'd like to propose a new improvement for Pulsar's Key_Shared > > > subscription mode, outlined in PIP-379. This proposal aims to address > > > several issues with the current implementation and introduce a more > > > efficient mechanism for managing message ordering. > > > > > > Problem: > > > The current Key_Shared implementation faces challenges including: > > > 1. Complex management of "recently joined consumers" > > > 2. Incomplete fulfillment of ordering guarantees > > > 3. Unnecessary message blocking > > > 4. Poor observability > > > > > > PIP-379 introduces a "draining hashes" concept to efficiently manage > > > message ordering by tracking affected hashes when consumer assignments > > > change. The high-level solution is drafted in the PIP document. > > > > > > Benefits: > > > 1. Improved message ordering guarantees > > > 2. Reduced unnecessary message blocking > > > 3. Better scalability and performance > > > 4. Enhanced observability > > > > > > This proposal would replace the existing "recently joined consumers" > > > mechanism, addressing its limitations while providing a more robust > > > solution. > > > > > > The full proposal can be found at: > > > https://github.com/apache/pulsar/pull/23309 > > > The direct link to the rendered version of the markdown file is: > > > https://github.com/lhotari/pulsar/blob/lh-pip-379/pip/pip-379.md > > > > > > I welcome your feedback and discussion on this proposal. Please share > > > your thoughts, concerns, or suggestions. > > > > > > -Lari > > > > > >
