Backlog distribution problem with SubscriptionType.Key_Shared
Hi,
I am exploring the Key_Shared feature of Apache Pulsar using the Pulsar
2.10.1 client.
*UseCase Perspective:*
I am simulating an everyday use case in Kubernetes-based deployments.
when we deploy new code to the pod's consumers restart, the first consumer
that starts is picking all the backlog, and other consumers are idle until
new messages are produced to the topic.
*Below is the consumer code configuration.*
consumer = initClient()
.newConsumer(Schema.STRING)
.topic(topicName)
.subscriptionType(SubscriptionType.Key_Shared)
.subscriptionName("test123")
.keySharedPolicy(KeySharedPolicy.autoSplitHashRange()
.setAllowOutOfOrderDelivery(true))
.messageListener(this::readMessage)
.consumerName(UUID.randomUUID().toString())
.subscribe();
I have created a group of 3 consumers to consume data and followed the
below steps.
1. Produce messages with randomly generated key and value pair
2. Now, The keys are distributed to all my consumers while consuming. And
all the messages are acknowledged. This is working great, as expected
3. Now, stop all the consumers and create a backlog. About 1000 random key
values pairs
4. Now, start all your consumers.
5. The issue is entire backlog is mapped to first consumer, and all the
other consumers are just passed, not receiving any event
6. When I produce new messages. They are hashed, and all 3 consumers are
receiving new messages
*My conclusions on a high level:*
1. Messages that are already hashed to the first consumer are not
distributed to the newly added consumers. Even if the first consumer picks
them up slowly
2. But when this first consumer is taken down, both the other consumers are
receiving backlog events
3. when I used setAllowOutOfOrderDelivery(true) new messages are sent to
newly added consumers. But the backlog is still stuck to just one consumer.
4. Data produced after the consumers are added is hashed properly, but the
older data is just stuck to one consumer.
Also, I have not seen this case in test cases of
KeySharedSubscriptionTest, regarding the distribution percentage of already
produced events with setAllowOutOfOrderDelivery(true).
Thank you,
Tarun.
Re: [ANNOUNCE] New Committer: Lin Chen
Congrats! Best, Haiting On Mon, Nov 14, 2022 at 4:58 PM Zixuan Liu wrote: > > Congrats! Lin > > Best, > Zixuan > > Enrico Olivelli 于2022年11月14日周一 15:06写道： > > > Congratulations > > > > Enrico > > > > Il Lun 14 Nov 2022, 07:30 Qiang Huang ha > > scritto: > > > > > Congratulations! > > > > > > houxiaoyu 于2022年11月14日周一 13:40写道： > > > > > > > Congrats! > > > > > > > > Best, > > > > Xiaoyu Hou > > > > > > > > PengHui Li 于2022年11月14日周一 12:28写道： > > > > > > > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > > > > Lin Chen (https://github.com/lordcheng10) > > > > > to become a committer and we are pleased to announce that he has > > > > accepted. > > > > > > > > > > Being a committer enables easier contribution to the > > > > > project since there is no need to go via the patch > > > > > submission process. This should enable better productivity. > > > > > > > > > > Welcome and congratulations, Lin Chen! > > > > > > > > > > Please join us in congratulating and welcoming Lin Chen onboard! > > > > > > > > > > Best Regards, > > > > > Penghui on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > -- > > > BR, > > > Qiang Huang > > > > >
Re: Releasing current master as Pulsar 2.11.0 ?
If we drop the current branch-2.11 and release based on the master, why not release 3.0.0 based on the master branch directly according to the new release plan [1]. If we cut the master branch and release Pulsar 2.11.0, we will wait at least three months before we cut 3.0.0. [1] https://github.com/apache/pulsar/issues/15966 Thanks, Hang guo jiwei 于2022年11月14日周一 17:16写道： > > I found out that several PRs have been unable to cherry-pick to 2.11 today. > I agree to cut the new branch based on the master and turn off the > new/unstable features in branch-2.11. > > > > Regards > Tboy > > > On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher wrote: > > > Inline > > > > Sent from my iPhone > > > > > On Nov 3, 2022, at 6:55 AM, Enrico Olivelli wrote: > > > > > > PengHui, > > > > > >> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li > > >> ha scritto: > > >> > > >>> As it is, we already need to discuss EOL for 2.7 and 2.8. > > >> > > >> Agree. We should clarify this one. > > >> I think we can stop to provide new releases for 2.7 > > >> and only security or critical bugs for 2.8 (one more official release) > > >> > > >> https://github.com/apache/pulsar/issues/15966 will make the > > >> release strategy clear. > > >> > > >> LTS -> 36 months (24 + 12) > > >> Feature release -> 6 months (3+3) > > >> > > >> Thanks, > > >> Penghui > > >> > > >> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall > > >> wrote: > > >> > > >>> I am concerned that we have too many active release branches, and > > planning > > >>> to follow 2.11.0 with 3.0.0 soon after feels like it will make that > > problem > > >>> worse. As it is, we already need to discuss EOL for 2.7 and 2.8. > > >>> > > >>> Thanks, > > >>> Michael > > >>> > > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li > > wrote: > > >>> > > Releasing from the master branch will bring more uncertainty, no? > > We have fixed many regressions that were introduced to branch-2.11. > > If we cut a new branch-2.11 based on the master branch. Maybe new > > regressions > > will happen again. This may make us wait another month to have a > > 2.11.0 > > release. > > > > > > I am not sure. > > > I don't know if anyone is actively testing the 2.11 branch more than > > > the master branch. > > > On my side the (automated) testing that I do with my colleagues on > > > branch-2.11 is basically the same as for the master branch. > > > > > > I believe that if we want to cut a 2.11 release that is not branched > > > again from the master branch > > > we really must start the release as soon as BK 4.15.3 is released > > > > I understand that Bookkeeper issues have Ben what’s blocking 2.11 > > > > > > Many people contributed features to the master branch that cannot be > > > shipped with 2.11 because > > > they are considered "breaking changes". > > > But 2.11 was supposed to be released in August, more than 3 months ago. > > > > I think we can recognize that our past history has been that there are > > often 3 or 4 RCs for our 2.x.0 releases. > > > > Maybe we should be cherry picking some PRs on master to 2.11 before we > > start the process? It may or may not save an RC but it will give us time to > > be realistic about a reasonable cadence from 2.10.x to 2.11.x to 2.12.x … > > it’s hard to support many versions at once. The CVE announced today took > > months to be included in all of our current releases from 2.7.5 to 2.10.2. > > Separation of C++ and Pulsar client releases from Pulsar releases helps > > here, but it may not with the next security issue. > > > > Regards, > > Dave > > > > > > > > > Enrico > > > > > > > > > > IMO, we can start Pulsar 3.0 (follow > > https://github.com/apache/pulsar/issues/15966) > > after 2.11.0 is released instead of waiting for 3 more months. > > > > For https://github.com/apache/bookkeeper/issues/3466 > > I don't think it's a blocker for the Pulsar release for now. > > Yes, it is worth investigating more. We also tried a chaos test for > > that > > case. > > We haven't reproduced the problem on Pulsar. > > > > Now, we are just waiting for the new BookKeeper release 4.15.3 since > > >>> 4.15.2 > > has regressions [1] > > > > [1] https://github.com/apache/bookkeeper/pull/3523 > > > > Thanks, > > Penghui > > > > On Tue, Nov 1, 2022 at 3:10 AM Michael Marshall > > > > wrote: > > > > > I have not followed the branch-2.11 work closely, but I think it > > makes > > > sense to re-create branch-2.11 from the current master. > > > > > > We created branch-2.11 almost 3 months ago. Re-creating the branch > > > will prevent unnecessary delay on new features added over the past 3 > > > months. > > > > > > If we follow through with this proposal, we will need to clean up PR > > > tags and milestones to prevent confusion. > > > > > > Thanks, > > > Michael > > > > > > On Mon, Oct 31, 2022
Re: Releasing current master as Pulsar 2.11.0 ?
Il giorno mar 15 nov 2022 alle ore 15:26 Hang Chen ha scritto: > > If we drop the current branch-2.11 and release based on the master, > why not release 3.0.0 based on the master branch directly according to > the new release plan [1]. 3.0.0 means "breaking changes" current master is 100% compatible Enrico > > If we cut the master branch and release Pulsar 2.11.0, we will wait at > least three months before we cut 3.0.0. > > > [1] https://github.com/apache/pulsar/issues/15966 > > Thanks, > Hang > > guo jiwei 于2022年11月14日周一 17:16写道： > > > > I found out that several PRs have been unable to cherry-pick to 2.11 today. > > I agree to cut the new branch based on the master and turn off the > > new/unstable features in branch-2.11. > > > > > > > > Regards > > Tboy > > > > > > On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher wrote: > > > > > Inline > > > > > > Sent from my iPhone > > > > > > > On Nov 3, 2022, at 6:55 AM, Enrico Olivelli wrote: > > > > > > > > PengHui, > > > > > > > >> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li > > > >> ha scritto: > > > >> > > > >>> As it is, we already need to discuss EOL for 2.7 and 2.8. > > > >> > > > >> Agree. We should clarify this one. > > > >> I think we can stop to provide new releases for 2.7 > > > >> and only security or critical bugs for 2.8 (one more official release) > > > >> > > > >> https://github.com/apache/pulsar/issues/15966 will make the > > > >> release strategy clear. > > > >> > > > >> LTS -> 36 months (24 + 12) > > > >> Feature release -> 6 months (3+3) > > > >> > > > >> Thanks, > > > >> Penghui > > > >> > > > >> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall > > > >> wrote: > > > >> > > > >>> I am concerned that we have too many active release branches, and > > > planning > > > >>> to follow 2.11.0 with 3.0.0 soon after feels like it will make that > > > problem > > > >>> worse. As it is, we already need to discuss EOL for 2.7 and 2.8. > > > >>> > > > >>> Thanks, > > > >>> Michael > > > >>> > > > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li > > > wrote: > > > >>> > > > Releasing from the master branch will bring more uncertainty, no? > > > We have fixed many regressions that were introduced to branch-2.11. > > > If we cut a new branch-2.11 based on the master branch. Maybe new > > > regressions > > > will happen again. This may make us wait another month to have a > > > 2.11.0 > > > release. > > > > > > > > I am not sure. > > > > I don't know if anyone is actively testing the 2.11 branch more than > > > > the master branch. > > > > On my side the (automated) testing that I do with my colleagues on > > > > branch-2.11 is basically the same as for the master branch. > > > > > > > > I believe that if we want to cut a 2.11 release that is not branched > > > > again from the master branch > > > > we really must start the release as soon as BK 4.15.3 is released > > > > > > I understand that Bookkeeper issues have Ben what’s blocking 2.11 > > > > > > > > Many people contributed features to the master branch that cannot be > > > > shipped with 2.11 because > > > > they are considered "breaking changes". > > > > But 2.11 was supposed to be released in August, more than 3 months ago. > > > > > > I think we can recognize that our past history has been that there are > > > often 3 or 4 RCs for our 2.x.0 releases. > > > > > > Maybe we should be cherry picking some PRs on master to 2.11 before we > > > start the process? It may or may not save an RC but it will give us time > > > to > > > be realistic about a reasonable cadence from 2.10.x to 2.11.x to 2.12.x … > > > it’s hard to support many versions at once. The CVE announced today took > > > months to be included in all of our current releases from 2.7.5 to 2.10.2. > > > Separation of C++ and Pulsar client releases from Pulsar releases helps > > > here, but it may not with the next security issue. > > > > > > Regards, > > > Dave > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > IMO, we can start Pulsar 3.0 (follow > > > https://github.com/apache/pulsar/issues/15966) > > > after 2.11.0 is released instead of waiting for 3 more months. > > > > > > For https://github.com/apache/bookkeeper/issues/3466 > > > I don't think it's a blocker for the Pulsar release for now. > > > Yes, it is worth investigating more. We also tried a chaos test for > > > that > > > case. > > > We haven't reproduced the problem on Pulsar. > > > > > > Now, we are just waiting for the new BookKeeper release 4.15.3 since > > > >>> 4.15.2 > > > has regressions [1] > > > > > > [1] https://github.com/apache/bookkeeper/pull/3523 > > > > > > Thanks, > > > Penghui > > > > > > On Tue, Nov 1, 2022 at 3:10 AM Michael Marshall > > > > > > wrote: > > > > > > > I have not followed the branch-2.11 work closely, but I think it > > > makes > > > > sense to re-create branch-2.11 from the curren
Re: Releasing current master as Pulsar 2.11.0 ?
> 3.0.0 means "breaking changes" PIP 175 [0] proposes a different meaning to 3.0.0. The proposed meaning is that each major release is an LTS version. Here is the exact wording: > The major version bump will not carry any special meaning in terms of > "big features" included in the release or breaking API changes. > Instead, it would simply signal the type of the release. However, I don't remember a vote to adopt PIP 175. I see the discussion on the ML [1], but I don't see the vote. - Michael [0] https://github.com/apache/pulsar/issues/15966 [1] https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f On Tue, Nov 15, 2022 at 8:40 AM Enrico Olivelli wrote: > > Il giorno mar 15 nov 2022 alle ore 15:26 Hang Chen > ha scritto: > > > > If we drop the current branch-2.11 and release based on the master, > > why not release 3.0.0 based on the master branch directly according to > > the new release plan [1]. > > 3.0.0 means "breaking changes" > current master is 100% compatible > > Enrico > > > > > If we cut the master branch and release Pulsar 2.11.0, we will wait at > > least three months before we cut 3.0.0. > > > > > > [1] https://github.com/apache/pulsar/issues/15966 > > > > Thanks, > > Hang > > > > guo jiwei 于2022年11月14日周一 17:16写道： > > > > > > I found out that several PRs have been unable to cherry-pick to 2.11 > > > today. > > > I agree to cut the new branch based on the master and turn off the > > > new/unstable features in branch-2.11. > > > > > > > > > > > > Regards > > > Tboy > > > > > > > > > On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher wrote: > > > > > > > Inline > > > > > > > > Sent from my iPhone > > > > > > > > > On Nov 3, 2022, at 6:55 AM, Enrico Olivelli > > > > > wrote: > > > > > > > > > > PengHui, > > > > > > > > > >> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li > > > > >> ha scritto: > > > > >> > > > > >>> As it is, we already need to discuss EOL for 2.7 and 2.8. > > > > >> > > > > >> Agree. We should clarify this one. > > > > >> I think we can stop to provide new releases for 2.7 > > > > >> and only security or critical bugs for 2.8 (one more official > > > > >> release) > > > > >> > > > > >> https://github.com/apache/pulsar/issues/15966 will make the > > > > >> release strategy clear. > > > > >> > > > > >> LTS -> 36 months (24 + 12) > > > > >> Feature release -> 6 months (3+3) > > > > >> > > > > >> Thanks, > > > > >> Penghui > > > > >> > > > > >> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall > > > > >> > > > > >> wrote: > > > > >> > > > > >>> I am concerned that we have too many active release branches, and > > > > planning > > > > >>> to follow 2.11.0 with 3.0.0 soon after feels like it will make that > > > > problem > > > > >>> worse. As it is, we already need to discuss EOL for 2.7 and 2.8. > > > > >>> > > > > >>> Thanks, > > > > >>> Michael > > > > >>> > > > > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li > > > > wrote: > > > > >>> > > > > Releasing from the master branch will bring more uncertainty, no? > > > > We have fixed many regressions that were introduced to branch-2.11. > > > > If we cut a new branch-2.11 based on the master branch. Maybe new > > > > regressions > > > > will happen again. This may make us wait another month to have a > > > > 2.11.0 > > > > release. > > > > > > > > > > I am not sure. > > > > > I don't know if anyone is actively testing the 2.11 branch more than > > > > > the master branch. > > > > > On my side the (automated) testing that I do with my colleagues on > > > > > branch-2.11 is basically the same as for the master branch. > > > > > > > > > > I believe that if we want to cut a 2.11 release that is not branched > > > > > again from the master branch > > > > > we really must start the release as soon as BK 4.15.3 is released > > > > > > > > I understand that Bookkeeper issues have Ben what’s blocking 2.11 > > > > > > > > > > Many people contributed features to the master branch that cannot be > > > > > shipped with 2.11 because > > > > > they are considered "breaking changes". > > > > > But 2.11 was supposed to be released in August, more than 3 months > > > > > ago. > > > > > > > > I think we can recognize that our past history has been that there are > > > > often 3 or 4 RCs for our 2.x.0 releases. > > > > > > > > Maybe we should be cherry picking some PRs on master to 2.11 before we > > > > start the process? It may or may not save an RC but it will give us > > > > time to > > > > be realistic about a reasonable cadence from 2.10.x to 2.11.x to 2.12.x > > > > … > > > > it’s hard to support many versions at once. The CVE announced today took > > > > months to be included in all of our current releases from 2.7.5 to > > > > 2.10.2. > > > > Separation of C++ and Pulsar client releases from Pulsar releases helps > > > > here, but it may not with the next security issue. > > > > > > > > Regards, > > > > Dave > > > > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > >>>
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Why PersistentTopicInternalStats don't provide getter/setter method?
GitHub user michaeljmarshall added a comment to the discussion: Why PersistentTopicInternalStats don't provide getter/setter method? Seems like an oversight to me. There could definitely be getter methods for the fields in those classes, but I don't think we need setter methods. Similarly, `org.apache.pulsar.common.policies.data.PartitionedTopicStats` only has getters. GitHub link: https://github.com/apache/pulsar/discussions/18481#discussioncomment-4149075 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] crossoverJie added a comment to the discussion: Why PersistentTopicInternalStats don't provide getter/setter method?
GitHub user crossoverJie added a comment to the discussion: Why PersistentTopicInternalStats don't provide getter/setter method? Thank you for your answer, I think I can help fix this. GitHub link: https://github.com/apache/pulsar/discussions/18481#discussioncomment-4149098 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: Releasing current master as Pulsar 2.11.0 ?
Hi Enrico, It's okay for me to cut the current master as 2.11.0, but since many new PRs were merged recently, I'm afraid some regressions might be introduced. I found some flaky tests (like [1]) recently, not sure whether they are caused by bugs. And there is also a PR [2] that tries to solve a bug but it also brings a regression. See my fix here: [3] Generally, when there are more PRs between two major releases, there will be a higher possibility to introduce more unstable factors. So we must take it verfy carefully with the 2.11 release. [1] https://github.com/apache/pulsar/issues/18480 [2] https://github.com/apache/pulsar/pull/18454 [3] https://github.com/apache/pulsar/pull/18486 Thanks, Yunze On Wed, Nov 16, 2022 at 12:30 AM Michael Marshall wrote: > > > 3.0.0 means "breaking changes" > > PIP 175 [0] proposes a different meaning to 3.0.0. The proposed > meaning is that each major release is an LTS version. Here is the > exact wording: > > > The major version bump will not carry any special meaning in terms of > > "big features" included in the release or breaking API changes. > > Instead, it would simply signal the type of the release. > > However, I don't remember a vote to adopt PIP 175. I see the > discussion on the ML [1], but I don't see the vote. > > - Michael > > [0] https://github.com/apache/pulsar/issues/15966 > [1] https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f > > On Tue, Nov 15, 2022 at 8:40 AM Enrico Olivelli wrote: > > > > Il giorno mar 15 nov 2022 alle ore 15:26 Hang Chen > > ha scritto: > > > > > > If we drop the current branch-2.11 and release based on the master, > > > why not release 3.0.0 based on the master branch directly according to > > > the new release plan [1]. > > > > 3.0.0 means "breaking changes" > > current master is 100% compatible > > > > Enrico > > > > > > > > If we cut the master branch and release Pulsar 2.11.0, we will wait at > > > least three months before we cut 3.0.0. > > > > > > > > > [1] https://github.com/apache/pulsar/issues/15966 > > > > > > Thanks, > > > Hang > > > > > > guo jiwei 于2022年11月14日周一 17:16写道： > > > > > > > > I found out that several PRs have been unable to cherry-pick to 2.11 > > > > today. > > > > I agree to cut the new branch based on the master and turn off the > > > > new/unstable features in branch-2.11. > > > > > > > > > > > > > > > > Regards > > > > Tboy > > > > > > > > > > > > On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher > > > > wrote: > > > > > > > > > Inline > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > On Nov 3, 2022, at 6:55 AM, Enrico Olivelli > > > > > > wrote: > > > > > > > > > > > > PengHui, > > > > > > > > > > > >> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li > > > > > >> ha scritto: > > > > > >> > > > > > >>> As it is, we already need to discuss EOL for 2.7 and 2.8. > > > > > >> > > > > > >> Agree. We should clarify this one. > > > > > >> I think we can stop to provide new releases for 2.7 > > > > > >> and only security or critical bugs for 2.8 (one more official > > > > > >> release) > > > > > >> > > > > > >> https://github.com/apache/pulsar/issues/15966 will make the > > > > > >> release strategy clear. > > > > > >> > > > > > >> LTS -> 36 months (24 + 12) > > > > > >> Feature release -> 6 months (3+3) > > > > > >> > > > > > >> Thanks, > > > > > >> Penghui > > > > > >> > > > > > >> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall > > > > > >> > > > > > >> wrote: > > > > > >> > > > > > >>> I am concerned that we have too many active release branches, and > > > > > planning > > > > > >>> to follow 2.11.0 with 3.0.0 soon after feels like it will make > > > > > >>> that > > > > > problem > > > > > >>> worse. As it is, we already need to discuss EOL for 2.7 and 2.8. > > > > > >>> > > > > > >>> Thanks, > > > > > >>> Michael > > > > > >>> > > > > > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li > > > > > wrote: > > > > > >>> > > > > > Releasing from the master branch will bring more uncertainty, no? > > > > > We have fixed many regressions that were introduced to > > > > > branch-2.11. > > > > > If we cut a new branch-2.11 based on the master branch. Maybe new > > > > > regressions > > > > > will happen again. This may make us wait another month to have a > > > > > 2.11.0 > > > > > release. > > > > > > > > > > > > I am not sure. > > > > > > I don't know if anyone is actively testing the 2.11 branch more than > > > > > > the master branch. > > > > > > On my side the (automated) testing that I do with my colleagues on > > > > > > branch-2.11 is basically the same as for the master branch. > > > > > > > > > > > > I believe that if we want to cut a 2.11 release that is not branched > > > > > > again from the master branch > > > > > > we really must start the release as soon as BK 4.15.3 is released > > > > > > > > > > I understand that Bookkeeper issues have Ben what’s blocking 2.11 > > > > > > > > > > > > Many people contributed features to
Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic
Introducing new system concepts as code change PR is really not the way to go about making these changes. The semantics and use case discussion should come first, so that they can be discussed and the implications understood. It's not a good experience to put this out there without providing a meaningful explanation of the general concept, semantics and use cases that applies to a general Pulsar user. And how is this put out as a change with doc-not-needed ? >The current use case is to use the non-persistent topic to store the load data used by the new load manager. How will Pulsar users understand this?. Non-persistent topics are a well-defined concept.- lossy, totally unreliable, and no guarantee of anything, Table View is another well-defined concept - a much more reliable construct with compacted keys. As a general Pulsar user, what exactly does a Table View on a non-persistent topic conceptually mean? What are the semantics? And then there is the TTL - how does this all fit together? > The current use case is to use the non-persistent topic to store the load data used by the new load manager. So is the idea that no external user should use this new NP-TableView? This is a public API A Table with a totally random selection of events in it? How does this work for a general use case? With this construct it is entirely possible that two clients A & B will have totally different data in their table views at the same time - what does that mean?You have introduced state into what is essentially a stateless concept (N-P topic) in Pulsar. I am not debating the merits/demerits of the change. It's really about how we go about doing things. Ideally you want to make a case for N-P TableViews, what's the semantics, what are the use cases, limitations and anti-patterns for this change. The community then can provide meaningful feedback, discuss the merits and suggest improvements. You will get a better feature, with good user documentation and clarity about how and when applications should adopt/reject that in their solution design. That benefits everyone - the contributor, the community, Pulsar users, and Pulsar architecture On Mon, Nov 14, 2022 at 8:59 PM Kai Wang wrote: > Hi Joe, > > > I am not sure about the semantics of TableView on a non-persistent topic. > > What happens if the client crashes? What is the base state for the > table? > > If users use a non-persistent topic as the TableView topic, when the > client crashes, > the TableViews data will be lose. > > The current use case is to use the non-persistent topic to store the load > data used by the new load manager. It doesn't require strong consistency > ensure, and no need persistence. > > > Thanks, > Kai > > On 2022/11/14 23:03:13 Joe F wrote: > > I am not sure about the semantics of TableView on a non-persistent topic. > > > > Exactly how does this work? > > > > What happens if the client crashes? What is the base state for the > table? > > > > What exactly can I expect as a user from this? > > > > Joe > > > > On Sun, Nov 13, 2022 at 8:57 PM Kai Wang wrote: > > > > > Hi, pulsar-dev community, > > > > > > Since the non-persistent topic support doesn't require API changes. I > have > > > pushed a PR to implement it, which has already been merged. > > > > > > See: https://github.com/apache/pulsar/pull/18375 > > > > > > And this PIP title has been changed to `Make TableView support TTL`. > > > > > > PIP link: https://github.com/apache/pulsar/issues/18229 > > > > > > Thanks, > > > Kai > > > > > > On 2022/11/04 02:28:41 Kai Wang wrote: > > > > Hi, pulsar-dev community, > > > > > > > > I’ve opened a PIP to discuss : PIP-221: Make TableView support read > the > > > non-persistent topic. > > > > > > > > PIP link: https://github.com/apache/pulsar/issues/18229 > > > > > > > > Thanks, > > > > Kai > > > > > > > > > >
Re: Releasing current master as Pulsar 2.11.0 ?
My two cents, I think we should go forward with the current branch. All the cherry picks were done in a thoughtful way. My assumption is that if we go down recutting the branch from the current master, it will take at least a couple of weeks to get it in a stable condition. Other than that, there’s still the time needed for rc, vote, and so on. >From my understanding now the current branch is ready for an rc since we upgraded bookkeeper and all the perf doubts are gone away. If we start with an rc this week, we’ll be likely able to release 2.11 in December. (And we’re still 4 months late). Cheers, Nicolò Il giorno mar 15 nov 2022 alle 21:03 Yunze Xu ha scritto: > Hi Enrico, > > It's okay for me to cut the current master as 2.11.0, but since many new > PRs > were merged recently, I'm afraid some regressions might be introduced. I > found > some flaky tests (like [1]) recently, not sure whether they are caused > by bugs. And > there is also a PR [2] that tries to solve a bug but it also brings a > regression. See > my fix here: [3] > > Generally, when there are more PRs between two major releases, there will > be > a higher possibility to introduce more unstable factors. So we must > take it verfy > carefully with the 2.11 release. > > [1] https://github.com/apache/pulsar/issues/18480 > [2] https://github.com/apache/pulsar/pull/18454 > [3] https://github.com/apache/pulsar/pull/18486 > > Thanks, > Yunze > > > On Wed, Nov 16, 2022 at 12:30 AM Michael Marshall > wrote: > > > > > 3.0.0 means "breaking changes" > > > > PIP 175 [0] proposes a different meaning to 3.0.0. The proposed > > meaning is that each major release is an LTS version. Here is the > > exact wording: > > > > > The major version bump will not carry any special meaning in terms of > > > "big features" included in the release or breaking API changes. > > > Instead, it would simply signal the type of the release. > > > > However, I don't remember a vote to adopt PIP 175. I see the > > discussion on the ML [1], but I don't see the vote. > > > > - Michael > > > > [0] https://github.com/apache/pulsar/issues/15966 > > [1] https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f > > > > On Tue, Nov 15, 2022 at 8:40 AM Enrico Olivelli > wrote: > > > > > > Il giorno mar 15 nov 2022 alle ore 15:26 Hang Chen > > > ha scritto: > > > > > > > > If we drop the current branch-2.11 and release based on the master, > > > > why not release 3.0.0 based on the master branch directly according > to > > > > the new release plan [1]. > > > > > > 3.0.0 means "breaking changes" > > > current master is 100% compatible > > > > > > Enrico > > > > > > > > > > > If we cut the master branch and release Pulsar 2.11.0, we will wait > at > > > > least three months before we cut 3.0.0. > > > > > > > > > > > > [1] https://github.com/apache/pulsar/issues/15966 > > > > > > > > Thanks, > > > > Hang > > > > > > > > guo jiwei 于2022年11月14日周一 17:16写道： > > > > > > > > > > I found out that several PRs have been unable to cherry-pick to > 2.11 today. > > > > > I agree to cut the new branch based on the master and turn off the > > > > > new/unstable features in branch-2.11. > > > > > > > > > > > > > > > > > > > > Regards > > > > > Tboy > > > > > > > > > > > > > > > On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher > wrote: > > > > > > > > > > > Inline > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > On Nov 3, 2022, at 6:55 AM, Enrico Olivelli < > eolive...@gmail.com> wrote: > > > > > > > > > > > > > > PengHui, > > > > > > > > > > > > > >> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li > > > > > > >> ha scritto: > > > > > > >> > > > > > > >>> As it is, we already need to discuss EOL for 2.7 and 2.8. > > > > > > >> > > > > > > >> Agree. We should clarify this one. > > > > > > >> I think we can stop to provide new releases for 2.7 > > > > > > >> and only security or critical bugs for 2.8 (one more official > release) > > > > > > >> > > > > > > >> https://github.com/apache/pulsar/issues/15966 will make the > > > > > > >> release strategy clear. > > > > > > >> > > > > > > >> LTS -> 36 months (24 + 12) > > > > > > >> Feature release -> 6 months (3+3) > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Penghui > > > > > > >> > > > > > > >> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall < > mmarsh...@apache.org> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> I am concerned that we have too many active release > branches, and > > > > > > planning > > > > > > >>> to follow 2.11.0 with 3.0.0 soon after feels like it will > make that > > > > > > problem > > > > > > >>> worse. As it is, we already need to discuss EOL for 2.7 and > 2.8. > > > > > > >>> > > > > > > >>> Thanks, > > > > > > >>> Michael > > > > > > >>> > > > > > > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li < > peng...@apache.org> > > > > > > wrote: > > > > > > >>> > > > > > > Releasing from the master branch will bring more > uncertainty, no? > > > > > > We have fixed many regress
Re: [ANNOUNCE] New Committer: Lin Chen
Congrats! Nicolò Il giorno mar 15 nov 2022 alle 13:30 Haiting Jiang ha scritto: > Congrats! > > Best, > Haiting > > On Mon, Nov 14, 2022 at 4:58 PM Zixuan Liu wrote: > > > > Congrats! Lin > > > > Best, > > Zixuan > > > > Enrico Olivelli 于2022年11月14日周一 15:06写道： > > > > > Congratulations > > > > > > Enrico > > > > > > Il Lun 14 Nov 2022, 07:30 Qiang Huang ha > > > scritto: > > > > > > > Congratulations! > > > > > > > > houxiaoyu 于2022年11月14日周一 13:40写道： > > > > > > > > > Congrats! > > > > > > > > > > Best, > > > > > Xiaoyu Hou > > > > > > > > > > PengHui Li 于2022年11月14日周一 12:28写道： > > > > > > > > > > > The Project Management Committee (PMC) for Apache Pulsar has > invited > > > > > > Lin Chen (https://github.com/lordcheng10) > > > > > > to become a committer and we are pleased to announce that he has > > > > > accepted. > > > > > > > > > > > > Being a committer enables easier contribution to the > > > > > > project since there is no need to go via the patch > > > > > > submission process. This should enable better productivity. > > > > > > > > > > > > Welcome and congratulations, Lin Chen! > > > > > > > > > > > > Please join us in congratulating and welcoming Lin Chen onboard! > > > > > > > > > > > > Best Regards, > > > > > > Penghui on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > > > > > > -- > > > > BR, > > > > Qiang Huang > > > > > > > > -- Nicolò Boschi
[Vote] PIP-220 TransferShedder (only for PIP-192 New Broker Load Balancer)
Dear Pulsar Community, Please review and vote on this PIP. PIP link: https://github.com/apache/pulsar/issues/18215 Thank you, -Heesung
[GitHub] [pulsar-presto] dependabot[bot] opened a new pull request, #7: Bump jackson-databind from 2.11.1 to 2.12.7.1 in /presto-distribution
dependabot[bot] opened a new pull request, #7: URL: https://github.com/apache/pulsar-presto/pull/7 Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.11.1 to 2.12.7.1. Commits See full diff in https://github.com/FasterXML/jackson/commits";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar-presto/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-presto] dependabot[bot] commented on pull request #6: Bump jackson-databind from 2.11.1 to 2.13.4.1 in /presto-distribution
dependabot[bot] commented on PR #6: URL: https://github.com/apache/pulsar-presto/pull/6#issuecomment-1316162043 Superseded by #7. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-presto] dependabot[bot] closed pull request #6: Bump jackson-databind from 2.11.1 to 2.13.4.1 in /presto-distribution
dependabot[bot] closed pull request #6: Bump jackson-databind from 2.11.1 to 2.13.4.1 in /presto-distribution URL: https://github.com/apache/pulsar-presto/pull/6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-presto] tisonkun closed pull request #7: Bump jackson-databind from 2.11.1 to 2.12.7.1 in /presto-distribution
tisonkun closed pull request #7: Bump jackson-databind from 2.11.1 to 2.12.7.1 in /presto-distribution URL: https://github.com/apache/pulsar-presto/pull/7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-presto] dependabot[bot] commented on pull request #7: Bump jackson-databind from 2.11.1 to 2.12.7.1 in /presto-distribution
dependabot[bot] commented on PR #7: URL: https://github.com/apache/pulsar-presto/pull/7#issuecomment-1316165571 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: Releasing current master as Pulsar 2.11.0 ?
Most of you agree to use the previous branch-2.11 for release, I will start the release this week. Regards Jiwei Guo On Wed, Nov 16, 2022 at 4:23 AM Nicolò Boschi wrote: > My two cents, > > I think we should go forward with the current branch. All the cherry picks > were done in a thoughtful way. > > My assumption is that if we go down recutting the branch from the current > master, it will take at least a couple of weeks to get it in a stable > condition. Other than that, there’s still the time needed for rc, vote, and > so on. > > From my understanding now the current branch is ready for an rc since we > upgraded bookkeeper and all the perf doubts are gone away. > > If we start with an rc this week, we’ll be likely able to release 2.11 in > December. (And we’re still 4 months late). > > > Cheers, > Nicolò > > Il giorno mar 15 nov 2022 alle 21:03 Yunze Xu > > ha scritto: > > > Hi Enrico, > > > > It's okay for me to cut the current master as 2.11.0, but since many new > > PRs > > were merged recently, I'm afraid some regressions might be introduced. I > > found > > some flaky tests (like [1]) recently, not sure whether they are caused > > by bugs. And > > there is also a PR [2] that tries to solve a bug but it also brings a > > regression. See > > my fix here: [3] > > > > Generally, when there are more PRs between two major releases, there will > > be > > a higher possibility to introduce more unstable factors. So we must > > take it verfy > > carefully with the 2.11 release. > > > > [1] https://github.com/apache/pulsar/issues/18480 > > [2] https://github.com/apache/pulsar/pull/18454 > > [3] https://github.com/apache/pulsar/pull/18486 > > > > Thanks, > > Yunze > > > > > > On Wed, Nov 16, 2022 at 12:30 AM Michael Marshall > > wrote: > > > > > > > 3.0.0 means "breaking changes" > > > > > > PIP 175 [0] proposes a different meaning to 3.0.0. The proposed > > > meaning is that each major release is an LTS version. Here is the > > > exact wording: > > > > > > > The major version bump will not carry any special meaning in terms of > > > > "big features" included in the release or breaking API changes. > > > > Instead, it would simply signal the type of the release. > > > > > > However, I don't remember a vote to adopt PIP 175. I see the > > > discussion on the ML [1], but I don't see the vote. > > > > > > - Michael > > > > > > [0] https://github.com/apache/pulsar/issues/15966 > > > [1] https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f > > > > > > On Tue, Nov 15, 2022 at 8:40 AM Enrico Olivelli > > wrote: > > > > > > > > Il giorno mar 15 nov 2022 alle ore 15:26 Hang Chen > > > > ha scritto: > > > > > > > > > > If we drop the current branch-2.11 and release based on the master, > > > > > why not release 3.0.0 based on the master branch directly according > > to > > > > > the new release plan [1]. > > > > > > > > 3.0.0 means "breaking changes" > > > > current master is 100% compatible > > > > > > > > Enrico > > > > > > > > > > > > > > If we cut the master branch and release Pulsar 2.11.0, we will wait > > at > > > > > least three months before we cut 3.0.0. > > > > > > > > > > > > > > > [1] https://github.com/apache/pulsar/issues/15966 > > > > > > > > > > Thanks, > > > > > Hang > > > > > > > > > > guo jiwei 于2022年11月14日周一 17:16写道： > > > > > > > > > > > > I found out that several PRs have been unable to cherry-pick to > > 2.11 today. > > > > > > I agree to cut the new branch based on the master and turn off > the > > > > > > new/unstable features in branch-2.11. > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > Tboy > > > > > > > > > > > > > > > > > > On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher < > wave4d...@comcast.net> > > wrote: > > > > > > > > > > > > > Inline > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > On Nov 3, 2022, at 6:55 AM, Enrico Olivelli < > > eolive...@gmail.com> wrote: > > > > > > > > > > > > > > > > PengHui, > > > > > > > > > > > > > > > >> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li > > > > > > > >> ha scritto: > > > > > > > >> > > > > > > > >>> As it is, we already need to discuss EOL for 2.7 and 2.8. > > > > > > > >> > > > > > > > >> Agree. We should clarify this one. > > > > > > > >> I think we can stop to provide new releases for 2.7 > > > > > > > >> and only security or critical bugs for 2.8 (one more > official > > release) > > > > > > > >> > > > > > > > >> https://github.com/apache/pulsar/issues/15966 will make the > > > > > > > >> release strategy clear. > > > > > > > >> > > > > > > > >> LTS -> 36 months (24 + 12) > > > > > > > >> Feature release -> 6 months (3+3) > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> Penghui > > > > > > > >> > > > > > > > >> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall < > > mmarsh...@apache.org> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >>> I am concerned that we have too many active release > > branches, and > > > > > > > planning > > > > > > > >>> to f
