Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-25 Thread Lari Hotari
Here's the early draft implementation and proof-of-concept of PIP-379: https://github.com/apache/pulsar/pull/23352 Thanks for all the positive feedback about PIP-379. It has been very helpful in improving the initial idea. However, I haven't yet had time to address the review comments. I'll upda

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-17 Thread Apurva Telang
Hello Lari, Thanks for writing this proposal. The proposed solution looks good. Best regards, Apurva Telang On Mon, Sep 16, 2024 at 4:23 AM Lari Hotari wrote: > Hi Girish, > > Thank you for your feedback and for raising an important concern about the > potential memory impact of the draining h

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-16 Thread Lari Hotari
Hi Girish, Thank you for your feedback and for raising an important concern about the potential memory impact of the draining hashes hashmap, especially for large numbers of hashes. I'm glad to inform you that the memory impact related to the draining hashes state is not significant. On the br

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-16 Thread Girish Sharma
This is a good proposal and solves a big blocker that we have in using key_shared for a few certain use cases. My concern is on the memory impact of maintaining the draining hashes hashmap, especially when the hashes are in the order of 10,000s or more. Will there be checks or constraints to limit

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-16 Thread Lari Hotari
Hi PengHui, Thank you for your suggestion. I really appreciate your input on this, and I understand that without context, your proposed approach would indeed be the way forward. In this particular case, though, there are compelling reasons why PIP-379 suggests replacing the "recently joined co

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-15 Thread PengHui Li
Hi Lari, I recommend creating a new implementation rather than directly replacing the existing one. This approach aligns with how we’ve handled several proposals in the past and allows us to maintain stability while introducing improvements - PIP-192: New Pulsar Broker Load Balancer - PIP-195: Ne

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-14 Thread Enrico Olivelli
Awesome proposal, no questions from my side +1 Enrico Il giorno sab 14 set 2024 alle ore 16:21 Lari Hotari ha scritto: > Dear Pulsar Community, > > I'd like to propose a new improvement for Pulsar's Key_Shared > subscription mode, outlined in PIP-379. This proposal aims to address > several is

[DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-14 Thread Lari Hotari
Dear Pulsar Community, I'd like to propose a new improvement for Pulsar's Key_Shared subscription mode, outlined in PIP-379. This proposal aims to address several issues with the current implementation and introduce a more efficient mechanism for managing message ordering. Problem: The current Ke