[GitHub] [pulsar-helm-chart] dragonls commented on pull request #208: Fix ci error caused by wrong block of if clause
dragonls commented on pull request #208: URL: https://github.com/apache/pulsar-helm-chart/pull/208#issuecomment-1019435385 @michaeljmarshall PTAL, thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.
HI, Michael Marshall
> I am concerned that this design will not scale and that it will be very
expensive on installations with many topics.
I very much agree with your concern, but currently we already have some
topic names to use ``endWith()`` and ``startWith()`` to determine.
> It would also improve the speed of operations on namespaces because
filtering out system topics would only require a quick check for
`startsWith("__")`.
Currently, we don't have some system topics that users can consume, I think
it's a good chance to normalise the system topic naming rule.
but "__" is more common, we need more complex naming rules to reduce
restrictions on users. Maybe we can use some prefix like "__pulsar_sys_"
etc.
Others, some plugins like "Protocol handler" want to use system topics or
pulsar-manager can monitor how many system topics we have. I think we also
need to support registering system topic names, because currently we can
only rely on system topic names to create system topics. (Maybe changing
the internal api to support creating system topic would also be a good
option. But then it's harder to monitor).
Do you have any great ideas ?
Best,
Mattison
On Sun, 23 Jan 2022 at 14:46, Michael Marshall wrote:
> This is a really important thread. I think our system topic design
> needs more definition, structure, and documentation. Thanks for
> starting the thread, Bo Cong.
>
> > If the user uses it, it means that the user has considered the result of
> the clear system sub backlog
>
> I think we need to discuss authorization for operating on system
> topics, especially when topic level policies are not enabled.
>
> From reading the PIP [0], it looks like we already have many different
> ways to designate a system topic (prefixes, suffixes, and total string
> equality). I am concerned that this design will not scale and that it
> will be very expensive on installations with many topics.
>
> Is it possible for us to pivot our implementation so that system
> topics and only system topics start with two underscores? This could
> simplify system topic authorization. It would also improve the speed
> of operations on namespaces because filtering out system topics would
> only require a quick check for `startsWith("__")`.
>
> While it's possible that users already have topics that start with
> "__", I think it is unlikely, and I think it is essential that we
> design system topics so that they are highly scalable.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/issues/13794
>
>
> On Thu, Jan 13, 2022 at 10:27 AM PengHui Li wrote:
> >
> > I agree with the point, we should avoid the `clearNamespaceBacklog(String
> > namespace)` to apply to the internal topic or internal cursor.
> > It will introduce abnormal behaviors, e.g. clear a replication cursor
> > backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.
> >
> > I would suggest letting the `clearNamespaceBacklog(String namespace)`
> > method does not apply to the Pulsar internal topics and cursors.
> >
> > Thanks,
> > Penghui
> >
> > On Thu, Jan 13, 2022 at 4:32 PM 丛搏 wrote:
> >
> > > Hi :
> > >
> > > issue link :
> https://github.com/streamnative/support-tickets/issues/408 <
> > > https://github.com/streamnative/support-tickets/issues/408>
> > >
> > > # Background
> > > Now we have many clear namespace backlog interfaces etc.
> > > ```
> > > clearNamespaceBacklog(String namespace)
> > >
> > > clearNamespaceBacklogAsync(String namespace)
> > >
> > > clearNamespaceBacklogForSubscription(String namespace, String
> subscription)
> > >
> > > clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> > > subscription)
> > >
> > > clearNamespaceBundleBacklog(String namespace, String bundle)
> > >
> > > clearNamespaceBundleBacklogAsync(String namespace, String bundle)
> > >
> > > clearNamespaceBundleBacklogForSubscription(String namespace, String
> > > bundle, String subscription)
> > >
> > > clearNamespaceBundleBacklogForSubscriptionAsync(String namespace,
> String
> > > bundle, String subscription)
> > > ```
> > > There are two types of interfaces:
> > > 1. clear namespace backlog with subscriptionName
> > > 2. clear namespace backlog no subscriptionName
> > >
> > > But there have a problem, users do not know that there are many system
> > > topics in the namespace.
> > >
> > > ```
> > >
> > > /**
> > > * Local topic name for the namespace events.
> > > */
> > > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > > "__change_events";
> > >
> > > /**
> > > * Local topic name for the transaction buffer snapshot.
> > > */
> > > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > > "__transaction_buffer_snapshot";
> > >
> > > public static final String SUBSCRIPTION_NAME =
> > > "transaction-buffer-sub";
> > > ```
> > > User only want to clear the backlog witch they own created and sub. But
> > > now we have clear the system topic sub backlog when user don't spe
[VOTE] Pulsar Release 2.9.2 Candidate 1
This is the first release candidate for Apache Pulsar, version 2.9.2. It fixes the many issues and the log4j security problem. *** Please download, test, and vote on this release. This vote will stay open for at least 72 hours *** Note that we are voting upon the source (tag), binaries are provided for convenience. Source and binary files: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-1/ SHA-512 checksums: ccecec6ed53e9aea8882be0ee8d3e2dd353cde48033dad329d907580e2e472ac4b63e5467ccc5ec218ed7095a275526d78f7b9efb0dc467af4113848c2462673 ./apache-pulsar-2.9.2-bin.tar.gz ebc4422c7a293aaa2aa32cb1dbd0e7a96e1848ae0a6ad70fe14a672359704085e98a40d9f97af472306f4eba798615836485782f8975738a6771087f9dfb3cbe ./apache-pulsar-2.9.2-src.tar.gz Maven staging repo: https://repository.apache.org/content/repositories/orgapachepulsar-1132/ The tag to be voted upon: v2.9.2-candidate-1 (8a5d2253b888b3b865a2aedf635d672821c7) https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-1 Pulsar's KEYS file containing PGP keys we use to sign the release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS Please download the source package, and follow the README to build and run the Pulsar standalone service.
Re: [DISCUSS] Apache Pulsar 2.9.2 release
Thanks for your reminder. Currently, the new added PR will be added the label `branch-x.x`, the label indicate the PR should be added to which branch. After the PR is merged, the PR will be cherry-picked to the specific branch. After cherry-pick, a label `cherry-picked/branch-x.x` will be added. I think this is a good way because it's hard to check all PRs before releasing a new version. Maybe we have a better choice, we could start a discussion. On 2022/01/05 17:43:17 Enrico Olivelli wrote: > BTW +1 > > I hope we will review carefully what we are including in the release > > And that we will let branch-2.9 stabilise by not adding any other commits > that are not strictly required: > - security related issues > - data loss/data corruption cases > > > Enrico > > > Thank you Ran Gao for starting this thread. > > > > Enrico > > Il Mer 5 Gen 2022, 17:29 Enrico Olivelli ha scritto: > > > 171 commits? > > > > Why? > > > > This is too much in my opinion for a point release. > > > > I am pretty sure that we don't need all that changes. > > > > Every change we cherry pick has a good chance to break the stability > > > > Enrico > > > > Il Mer 5 Gen 2022, 16:30 mattison chao ha > > scritto: > > > >> +1 > >> > >> regards, > >> Mattisonchao > >> > >> On Wed, 5 Jan 2022 at 23:04, Lan Liang wrote: > >> > >> > +1,Thanks for your work. > >> > > >> > > >> > > >> > > >> > > >> > > >> > Best Regards, > >> > Lan Liang > >> > On 1/5/2022 23:01，Hang Chen wrote： > >> > +1 > >> > > >> > Best, > >> > Hang > >> > > >> > 陳智弘 于2022年1月5日周三 21:30写道： > >> > > >> > +1 > >> > > >> > Lari Hotari 於 2022年1月5日 週三 20:56 寫道： > >> > > >> > +1 > >> > > >> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao wrote: > >> > > >> > Hello, Pulsar community: > >> > > >> > I'd like to propose that we release Apache Pulsar 2.9.2. > >> > > >> > Currently, compared to 2.9.1, branch-2.9 already merged 171 > >> commits(refer > >> > to [0]), they contain the log4j security patch and many important fixes. > >> > > >> > I am happy to volunteer to be the release manager. > >> > > >> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to > >> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are > >> > labeled release/2.9.2, I'll follow them to make sure important fix could > >> > be > >> > merged in 2.9.2. > >> > > >> > > >> > Best, > >> > Ran Gao > >> > > >> > > >> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9 > >> > > >> > > >> > > >> > > >
Re: [DISCUSS] Apache Pulsar 2.9.2 release
Sorry, late response, please take a look at the previous reply. If you have good ideas, we could start a discussion. On 2022/01/05 18:31:38 Michael Marshall wrote: > +1 - thanks for starting this thread. > > >171 commits? > > Why? > > This is too much in my opinion for a point release. > > I agree that this is a lot of commits, and especially since we only > just released 2.9.1. Note though that this is not a new phenomenon: > 2.8.2 had 251 commits on top of 2.8.1. [0] > > > Every change we cherry pick has a good chance to break the stability > > This is one of the challenges to our current cherry-pick based git > workflow. It's hard to decide which commits to cherry-pick, and it > requires a high level of effort to later verify the correctness of > what was cherry-picked. This is one of the reasons I am interested in > exploring a merge based git workflow. > > Our project has grown a ton and will continue to grow. Our git process > should prioritize the stability of our existing release branches. > > > And that we will let branch-2.9 stabilise by not adding any other commits > > that are not strictly required: > > - security related issues > > - data loss/data corruption cases > > If we don't change our git workflow, we should at least add a > cherry-picking guide to the wiki. We have added many new committers in > the past year, but we don't provide them with much documented guidance > on how to exercise their new rights. > > Thanks, > Michael > > [0] https://github.com/apache/pulsar/compare/v2.8.1...v2.8.2 > > > On Wed, Jan 5, 2022 at 11:44 AM Enrico Olivelli wrote: > > > > BTW +1 > > > > I hope we will review carefully what we are including in the release > > > > And that we will let branch-2.9 stabilise by not adding any other commits > > that are not strictly required: > > - security related issues > > - data loss/data corruption cases > > > > > > Enrico > > > > > > Thank you Ran Gao for starting this thread. > > > > > > > > Enrico > > > > Il Mer 5 Gen 2022, 17:29 Enrico Olivelli ha scritto: > > > > > 171 commits? > > > > > > Why? > > > > > > This is too much in my opinion for a point release. > > > > > > I am pretty sure that we don't need all that changes. > > > > > > Every change we cherry pick has a good chance to break the stability > > > > > > Enrico > > > > > > Il Mer 5 Gen 2022, 16:30 mattison chao ha > > > scritto: > > > > > >> +1 > > >> > > >> regards, > > >> Mattisonchao > > >> > > >> On Wed, 5 Jan 2022 at 23:04, Lan Liang wrote: > > >> > > >> > +1,Thanks for your work. > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > Best Regards, > > >> > Lan Liang > > >> > On 1/5/2022 23:01，Hang Chen wrote： > > >> > +1 > > >> > > > >> > Best, > > >> > Hang > > >> > > > >> > 陳智弘 于2022年1月5日周三 21:30写道： > > >> > > > >> > +1 > > >> > > > >> > Lari Hotari 於 2022年1月5日 週三 20:56 寫道： > > >> > > > >> > +1 > > >> > > > >> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao wrote: > > >> > > > >> > Hello, Pulsar community: > > >> > > > >> > I'd like to propose that we release Apache Pulsar 2.9.2. > > >> > > > >> > Currently, compared to 2.9.1, branch-2.9 already merged 171 > > >> commits(refer > > >> > to [0]), they contain the log4j security patch and many important > > >> > fixes. > > >> > > > >> > I am happy to volunteer to be the release manager. > > >> > > > >> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to > > >> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are > > >> > labeled release/2.9.2, I'll follow them to make sure important fix > > >> > could > > >> > be > > >> > merged in 2.9.2. > > >> > > > >> > > > >> > Best, > > >> > Ran Gao > > >> > > > >> > > > >> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9 > > >> > > > >> > > > >> > > > >> > > > >
[GitHub] [pulsar-manager] mattisonchao commented on issue #441: Support HerdDB in the standard docker image.
mattisonchao commented on issue #441: URL: https://github.com/apache/pulsar-manager/issues/441#issuecomment-1019492858 @michaeljmarshall Yes, That's exactly what I thought. I think we can add ``herddb image`` in **docker-compose** and remove raw database logic from pulsar-manager to improve user **experience** and **observability**. Am I understanding right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ
> This is true of the DLQ producer, but by using the metadata map, won't we be exposing this feature to end users implicitly because they could add the field to the metadata map for other producers? (I am okay with giving users this functionality, but I know that Matteo said in a community meeting that this feature should be limited to the DLQ producer.) Yes, but it doesn't seem like a big deal. We just need to implement it like that `ORIGINAL_MESSAGE_ID` in the retryLetterTopic. On Sun, Jan 23, 2022 at 2:38 PM Michael Marshall wrote: > > > users will not touch the metadata of the producer of a DLQ > > This is true of the DLQ producer, but by using the metadata map, won't > we be exposing this feature to end users implicitly because they could > add the field to the metadata map for other producers? (I am okay with > giving users this functionality, but I know that Matteo said in a > community meeting that this feature should be limited to the DLQ producer.) > > > we also introduce some system message properties such as > > __original_message_id > > in retry letter topic. > > Thanks for this context. I didn't know we were already doing this. > > Thanks, > Michael > > > On Fri, Jan 21, 2022 at 3:35 AM Zike Yang > wrote: > > > > +1 for adding init_subscription filed to the metadata of CommandProducer. > > > > > > > > On Thu, Jan 20, 2022 at 2:52 PM PengHui Li wrote: > > > > > > > What is the justification for avoiding the new protobuf field? If we > > > add a structured field to a map of , we are still > > > modifying the protocol, even if we aren't modifying the protobuf. > > > > > > Not all cases need a new field named init_subscription when creating > > > producer, > > > and users will not touch the metadata of the producer of a DLQ, so I think > > > we can use the metadata to achieve the purpose more flexibly. > > > > > > Yes, it modifies the protocol, but in different ways. It like the message > > > properties, > > > we also introduce some system message properties such as > > > __original_message_id > > > in retry letter topic. > > > > > > Penghui > > > > > > On Thu, Jan 20, 2022 at 2:05 PM Michael Marshall > > > wrote: > > > > > > > > I think that, good or bad, the impression that users have that the DLQ > > > > > is not a "normal" topic > > > > > > > > Thanks for your perspective, Matteo. I still prefer my alternative > > > > design that bypasses subscription creation, but it seems there > > > > is insufficient interest in it, so we should move forward > > > > discussing a DLQ specific feature and its implementation. > > > > > > > > > 1. Instead of modifying the current protocol, we can use producer > > > > > metadata to carry the init subscription > > > > > > > > What is the justification for avoiding the new protobuf field? If we > > > > add a structured field to a map of , we are still > > > > modifying the protocol, even if we aren't modifying the protobuf. > > > > > > > > Thanks, > > > > Michael > > > > > > > > > > > > On Tue, Jan 18, 2022 at 8:38 AM PengHui Li wrote: > > > > > > > > > > +1 for adding the DLQ_init_sub to producer metadata so that we don't > > > > > need to introduce a new field in CommandProducer, and the new field > > > > > looks a little confusing > > > > > > > > > > Thanks, > > > > > Penghui > > > > > > > > > > On Mon, Jan 17, 2022 at 10:19 PM Hang Chen > > > > > wrote: > > > > > > > > > > > Thanks for creating this proposal Zike Yang. I have two ideas about > > > > > > it. > > > > > > 1. Instead of modifying the current protocol, we can use producer > > > > > > metadata to carry the init subscription > > > > > > 2. Please add auth for subscription creation when create topic by > > > > > > producer, otherwise, it will be easily attacked. > > > > > > > > > > > > Thanks, > > > > > > Hang > > > > > > > > > > > > Matteo Merli 于2022年1月12日周三 15:13写道： > > > > > > > > > > > > > > > If we want to hold that the DLQ is not a normal topic, then I > > > > > > > > can > > > > see > > > > > > > > why we would have a DLQ specific feature here. > > > > > > > > > > > > > > I think that, good or bad, the impression that users have that the > > > > DLQ > > > > > > > is not a "normal" topic comes from 2 factors: > > > > > > > 1. The experience with traditional messaging systems JMS and > > > > > > > others > > > > > > > where the DLQ are handled in slightly different ways, compared to > > > > > > > other topics > > > > > > > 2. The name "DLQ" which in a way it's implying a "queue"... which > > > > can > > > > > > > be implemented on topic, using a subscription.. > > > > > > > > > > > > > > > > > > -- > > Zike Yang -- Zike Yang
