Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-24 Thread Andrey Yegorov
Penghui, Thank you for the thorough and detailed response. I have added the feature toggle per Lari's comment on the implementation PR. Regarding the compatibility, if the user has 10MB cursor data somehow (in our testng large serialized position info resulted in a single large entry > 1MB that

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-24 Thread PengHui Li
Thanks for driving the proposal. I would like to share the related context that happened many years ago - https://lists.apache.org/thread/y0r9kk0968ydpxtf16x6ql3x6kwy7dc1 - https://lists.apache.org/thread/hfv18cg0yckt5cqd0fc66rp7tth036kf We have two major approaches: 1. Minimize the persistent

Re: Pulsar 4.0 LTS release timeline

2024-09-24 Thread Lari Hotari
BookKeeper 4.17.2 release discussion thread is https://lists.apache.org/thread/zhs8x1zz3g98998rsb68658qtgcgkw4r -Lari On 2024/09/24 21:23:38 Lari Hotari wrote: > Here's an update on the Pulsar 4.0 release timeline. > > We are currently slightly behind the original schedule; however, we > still

Re: Pulsar 4.0 LTS release timeline

2024-09-24 Thread Lari Hotari
Here's an update on the Pulsar 4.0 release timeline. We are currently slightly behind the original schedule; however, we still have a good chance of keeping the release schedule on track without major slippage. There are currently a few release blockers for 4.0 (tagged issues: https://github.com/

Re: Reverting some cherry-picks in branch-3.3 and branch-3.0

2024-09-24 Thread Lari Hotari
Kudos to Yubiao about that! -Lari On Tue, 24 Sept 2024, 20.22 Andrey Yegorov, wrote: > Thank you for catching the regression and finding the root cause. > > On Tue, Sep 24, 2024 at 7:59 AM Lari Hotari wrote: > > > Hi all, > > > > I'm reverting 3 cherry-pick PRs in branch-3.3 and branch-3.0 due

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

2024-09-24 Thread Dave Fisher
Hi - > On Sep 24, 2024, at 11:01 AM, Heesung Sohn wrote: > > Hi all, > > I realized that sampling outlier topic/subscription/consumer/producer > metrics can be useful to operate Pulsar. > > I noticed that Pulsar's performance issues are mostly caused by > outlier topics/subscription/consumers/

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

2024-09-24 Thread Heesung Sohn
Hi all, I realized that sampling outlier topic/subscription/consumer/producer metrics can be useful to operate Pulsar. I noticed that Pulsar's performance issues are mostly caused by outlier topics/subscription/consumers/producers. For example, disk usage can grow a lot when consumers are very s

Re: Reverting some cherry-picks in branch-3.3 and branch-3.0

2024-09-24 Thread Andrey Yegorov
Thank you for catching the regression and finding the root cause. On Tue, Sep 24, 2024 at 7:59 AM Lari Hotari wrote: > Hi all, > > I'm reverting 3 cherry-pick PRs in branch-3.3 and branch-3.0 due to a > performance regression: > - https://github.com/apache/pulsar/pull/23226 > - https://github.co

Re: [DISCUSS] Revisit PIP voting max time

2024-09-24 Thread Enrico Olivelli
Il giorno mar 24 set 2024 alle ore 17:31 Dave Fisher ha scritto: > > > > On Sep 23, 2024, at 8:07 PM, Yunze Xu wrote: > > > > I have similar concerns for it. Some PIPs might not get enough > > attention. Generally I agree with the proposal that a PIP should be > > treated as "approved" if > > -

Re: [DISCUSS] Revisit PIP voting max time

2024-09-24 Thread Dave Fisher
> On Sep 23, 2024, at 8:07 PM, Yunze Xu wrote: > > I have similar concerns for it. Some PIPs might not get enough > attention. Generally I agree with the proposal that a PIP should be > treated as "approved" if > - there is at least 1 binding +1 vote > - there is no binding -1 vote > - the vot

Re: Reverting some cherry-picks in branch-3.3 and branch-3.0

2024-09-24 Thread Yubiao Feng
Hi Lari +1, thanks for your hard work Thanks Yubiao Feng On Tue, Sep 24, 2024 at 10:59 PM Lari Hotari wrote: > Hi all, > > I'm reverting 3 cherry-pick PRs in branch-3.3 and branch-3.0 due to a > performance regression: > - https://github.com/apache/pulsar/pull/23226 > - https://github.com/apac

Reverting some cherry-picks in branch-3.3 and branch-3.0

2024-09-24 Thread Lari Hotari
Hi all, I'm reverting 3 cherry-pick PRs in branch-3.3 and branch-3.0 due to a performance regression: - https://github.com/apache/pulsar/pull/23226 - https://github.com/apache/pulsar/pull/23284 - https://github.com/apache/pulsar/pull/23340 These PRs address https://github.com/apache/pulsar/issues

Re: [DISCUSS] Revisit PIP voting max time

2024-09-24 Thread Lari Hotari
I agree that this is a problem. We need to resolve it. It's also good to note that Apache Pulsar is an Apache project run by volunteers. There's no paid staff processing PIPs. It's the responsibility of everyone in the community to contribute to the common administrative work of running the project

Re: [VOTE] PIP-381: Handle large PositionInfo state

2024-09-24 Thread Lari Hotari
+1 (binding), with the condition that PIP-381 can be controlled with a feature flag. I have provided that feedback in the discussion thread. I'm expecting that it gets addressed before the PIP is merged. -Lari On Mon, 23 Sept 2024 at 23:08, Enrico Olivelli wrote: > > +1 (binding) > > Enrico > >

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-24 Thread Lari Hotari
On Tue, 24 Sept 2024 at 05:01, Rajan Dhabalia wrote: > However, there are multiple other PRs related to key-shared sub, stats, > cursor performance, and other PRs are still blocked by others and people > just block it because they think they don't have this usecase. It's so > unfortunate that peop

Re: [DISCUSS] PIP-381: Handle large PositionInfo state

2024-09-24 Thread Lari Hotari
On Mon, 23 Sept 2024 at 20:37, Andrey Yegorov wrote: > @Lari: I think we can simply use managedLedgerMaxUnackedRangesToPersist as > a limiter, as the state is already persisted with the cursor. > Adding another config to allow single entry/multi entry storage of the > state feels like unnecessary