Re: [DISCUSS] Cherry-pick PIP-370 configurable remote topic creation in geo-replication

2024-09-25 Thread Yubiao Feng
Hi Lari > Why did we cherry-pick to branch-3.0 > but skip branch-3.3? - I think `3.0` is a long-term supported version, since the PIP is very useful, I want to cherry-pick it into `branch-3.0` - `3.3` is not a long-term supported version, so the PIP will be released with the next feature release

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-25 Thread Rajan Dhabalia
+1 binding. >> Can we use a space-efficient probabilistic data structure, such as bloom filter to store the acked msg ids Here, in the problem statement it tries to dump and retrieve data for topic recovery, so we need a compute efficient solution for faster recovery with disk storage to store la

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-25 Thread Heesung Sohn
Sorry, we probably need to store "not acked" msg ids in the bloom filter. For all msg_Id from the min_acked, - if found in "not-acked msg id set"(possibly non-acked) -> re-send(msgs possibly duplicated). - if not found in "not-acked msg id set"(definitely acked) -> do not send. On Wed, Sep 25, 2

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-25 Thread Heesung Sohn
Hi team, Can we use a space-efficient probabilistic data structure, such as bloom filter to store the acked msg ids, if re-delivering acked msgs are not so strict? On Tue, Sep 24, 2024 at 5:28 PM Andrey Yegorov wrote: > > Penghui, > > Thank you for the thorough and detailed response. > > I have

Re: [VOTE] PIP-381: Handle large PositionInfo state

2024-09-25 Thread Apurva Telang
+1 (non-binding) On Tue, Sep 24, 2024 at 1:44 AM Lari Hotari wrote: > +1 (binding), with the condition that PIP-381 can be controlled with a > feature flag. I have provided that feedback in the discussion thread. > I'm expecting that it gets addressed before the PIP is merged. > > -Lari > > On M

Re: [DISCUSS] PIP-379: Key_Shared Draining Hashes for Improved Message Ordering

2024-09-25 Thread Lari Hotari
Here's the early draft implementation and proof-of-concept of PIP-379: https://github.com/apache/pulsar/pull/23352 Thanks for all the positive feedback about PIP-379. It has been very helpful in improving the initial idea. However, I haven't yet had time to address the review comments. I'll upda

[DISCUSS] PIP-382: Add a label named reason for topic_load_failed_total

2024-09-25 Thread Yubiao Feng
Hi all I want to start a discussion on PIP-382: Add a label named reason for topic_load_failed_total Proposal link: https://github.com/apache/pulsar/pull/23351 I'm looking forward to hearing from you. Thanks Yubiao Feng