Dear Pulsar Community, I'd like to propose a new improvement for Pulsar's Key_Shared subscription mode, outlined in PIP-379. This proposal aims to address several issues with the current implementation and introduce a more efficient mechanism for managing message ordering.
Problem: The current Key_Shared implementation faces challenges including: 1. Complex management of "recently joined consumers" 2. Incomplete fulfillment of ordering guarantees 3. Unnecessary message blocking 4. Poor observability PIP-379 introduces a "draining hashes" concept to efficiently manage message ordering by tracking affected hashes when consumer assignments change. The high-level solution is drafted in the PIP document. Benefits: 1. Improved message ordering guarantees 2. Reduced unnecessary message blocking 3. Better scalability and performance 4. Enhanced observability This proposal would replace the existing "recently joined consumers" mechanism, addressing its limitations while providing a more robust solution. The full proposal can be found at: https://github.com/apache/pulsar/pull/23309 The direct link to the rendered version of the markdown file is: https://github.com/lhotari/pulsar/blob/lh-pip-379/pip/pip-379.md I welcome your feedback and discussion on this proposal. Please share your thoughts, concerns, or suggestions. -Lari
