Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
+1 (binding) Penghui On Tue, Jan 31, 2023 at 3:52 PM Nitin Goyal wrote: > +1 (non-binding) > > On Tue, Jan 31, 2023 at 12:29 PM guo jiwei wrote: > > > +1 (binding) > > > > > > Regards > > Jiwei Guo (Tboy) > > > > On Tue, Jan 31, 2023 at 2:36 PM Yunze Xu > > wrote: > > > > > > +1 (binding) > > > > > > Thanks, > > > Yunze > > > > > > On Tue, Jan 31, 2023 at 6:57 AM wrote: > > > > > > > > Hello everyone. > > > > > > > > I would like to start the vote for PIP-242 > > https://github.com/apache/pulsar/issues/19239, > > > > Please let me know if you have any concerns or questions. > > > > > > > > Best, > > > > Mattison > > > > > > > > --- Paste original PIP content to help quote -- > > > > > > > > ### Motivation > > > > > > > > Currently, the Apache Pulsar broker allows users to create a topic > > name that includes `-partition-`, which is confusing for our developers > to > > identify whether this is a partition of a partitioned topic. Plus, we > need > > to add more logic to be compatible with this special topic name. for > > example: > > > > > > > > - https://github.com/apache/pulsar/pull/19240 > > > > - https://github.com/apache/pulsar/pull/19230 > > > > - https://github.com/apache/pulsar/pull/19171 > > > > - https://github.com/apache/pulsar/pull/19086 > > > > - ... > > > > > > > > ### Goal > > > > This proposal wants `-partition-` to be a topic name keyword. Users > > can only create a topic with it if the topic is partitioned. For the > > compatibility reason, we want to Introduce a new configuration - > > `enableStrictTopicName` for the broker to help reject creating a topic in > > the following cases: > > > > 1. Create a partitioned topic that includes `-partition-`. > > > > 2. Create a topic which is not a partitioned topic. > > > > > > > > **Create a topic:** > > > > _no corresponding partitioned topic_ > > > > > > > > - persistent://public/default/local-name (passed) > > > > - persistent://public/default/local-name-partition-z (rejected by > > keyword) > > > > - persistent://public/default/local-name-partition-0 (rejected by > > keyword) > > > > > > > > _Has corresponding partitioned topic, **partitions=2** and topic > > partition name is **persistent://public/default/local-name**_ > > > > > > > > - persistent://public/default/local-name-partition-0 (passed, Because > > it is the partition topic's sub-partition) > > > > - persistent://public/default/local-name-partition-z (rejected by > > keyword) > > > > - persistent://public/default/local-name-partition-4 (rejected, > > Because it exceeds the number of maximum partitions) > > > > > > > > **Create a partitioned topic(topic metadata)** > > > > > > > > - persistent://public/default/local-name (passed) > > > > - persistent://public/default/local-name-partition-z (rejected by > > keyword) > > > > - persistent://public/default/local-name-partition-0 (rejected by > > keyword) > > > > > > > > > > > > ### API Changes > > > > > > > > Add a new configuration, `enableStrictTopicName=false`. > > > > > > > > ### Implementation > > > > > > > > 1. Add configuration `enableStrictTopicName=false`. > > > > 2. Add rejection logic when the user enables `enableStrictTopicName`. > > > > 4. Add warning logs to inform users that we do not recommend creating > > non-partitioned topics with the keyword `-partition-`. > > > > 5. Make `enableStrictTopicName=true` in the next major release. > > > > > -- > Regards > Nitin Goyal >
Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
+1 (non-binding) Thanks ZhangJian He On Tue, 31 Jan 2023 at 16:51, PengHui Li wrote: > +1 (binding) > > Penghui > > On Tue, Jan 31, 2023 at 3:52 PM Nitin Goyal > wrote: > > > +1 (non-binding) > > > > On Tue, Jan 31, 2023 at 12:29 PM guo jiwei wrote: > > > > > +1 (binding) > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > On Tue, Jan 31, 2023 at 2:36 PM Yunze Xu > > > > wrote: > > > > > > > > +1 (binding) > > > > > > > > Thanks, > > > > Yunze > > > > > > > > On Tue, Jan 31, 2023 at 6:57 AM wrote: > > > > > > > > > > Hello everyone. > > > > > > > > > > I would like to start the vote for PIP-242 > > > https://github.com/apache/pulsar/issues/19239, > > > > > Please let me know if you have any concerns or questions. > > > > > > > > > > Best, > > > > > Mattison > > > > > > > > > > --- Paste original PIP content to help quote -- > > > > > > > > > > ### Motivation > > > > > > > > > > Currently, the Apache Pulsar broker allows users to create a topic > > > name that includes `-partition-`, which is confusing for our developers > > to > > > identify whether this is a partition of a partitioned topic. Plus, we > > need > > > to add more logic to be compatible with this special topic name. for > > > example: > > > > > > > > > > - https://github.com/apache/pulsar/pull/19240 > > > > > - https://github.com/apache/pulsar/pull/19230 > > > > > - https://github.com/apache/pulsar/pull/19171 > > > > > - https://github.com/apache/pulsar/pull/19086 > > > > > - ... > > > > > > > > > > ### Goal > > > > > This proposal wants `-partition-` to be a topic name keyword. Users > > > can only create a topic with it if the topic is partitioned. For the > > > compatibility reason, we want to Introduce a new configuration - > > > `enableStrictTopicName` for the broker to help reject creating a topic > in > > > the following cases: > > > > > 1. Create a partitioned topic that includes `-partition-`. > > > > > 2. Create a topic which is not a partitioned topic. > > > > > > > > > > **Create a topic:** > > > > > _no corresponding partitioned topic_ > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > keyword) > > > > > - persistent://public/default/local-name-partition-0 (rejected by > > > keyword) > > > > > > > > > > _Has corresponding partitioned topic, **partitions=2** and topic > > > partition name is **persistent://public/default/local-name**_ > > > > > > > > > > - persistent://public/default/local-name-partition-0 (passed, > Because > > > it is the partition topic's sub-partition) > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > keyword) > > > > > - persistent://public/default/local-name-partition-4 (rejected, > > > Because it exceeds the number of maximum partitions) > > > > > > > > > > **Create a partitioned topic(topic metadata)** > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > keyword) > > > > > - persistent://public/default/local-name-partition-0 (rejected by > > > keyword) > > > > > > > > > > > > > > > ### API Changes > > > > > > > > > > Add a new configuration, `enableStrictTopicName=false`. > > > > > > > > > > ### Implementation > > > > > > > > > > 1. Add configuration `enableStrictTopicName=false`. > > > > > 2. Add rejection logic when the user enables > `enableStrictTopicName`. > > > > > 4. Add warning logs to inform users that we do not recommend > creating > > > non-partitioned topics with the keyword `-partition-`. > > > > > 5. Make `enableStrictTopicName=true` in the next major release. > > > > > > > > > -- > > Regards > > Nitin Goyal > > >
Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
+1 (binding) Thanks, Bo ZhangJian He 于2023年1月31日周二 16:52写道： > > +1 (non-binding) > > Thanks > ZhangJian He > > > On Tue, 31 Jan 2023 at 16:51, PengHui Li wrote: > > > +1 (binding) > > > > Penghui > > > > On Tue, Jan 31, 2023 at 3:52 PM Nitin Goyal > > wrote: > > > > > +1 (non-binding) > > > > > > On Tue, Jan 31, 2023 at 12:29 PM guo jiwei wrote: > > > > > > > +1 (binding) > > > > > > > > > > > > Regards > > > > Jiwei Guo (Tboy) > > > > > > > > On Tue, Jan 31, 2023 at 2:36 PM Yunze Xu > > > > > > wrote: > > > > > > > > > > +1 (binding) > > > > > > > > > > Thanks, > > > > > Yunze > > > > > > > > > > On Tue, Jan 31, 2023 at 6:57 AM wrote: > > > > > > > > > > > > Hello everyone. > > > > > > > > > > > > I would like to start the vote for PIP-242 > > > > https://github.com/apache/pulsar/issues/19239, > > > > > > Please let me know if you have any concerns or questions. > > > > > > > > > > > > Best, > > > > > > Mattison > > > > > > > > > > > > --- Paste original PIP content to help quote -- > > > > > > > > > > > > ### Motivation > > > > > > > > > > > > Currently, the Apache Pulsar broker allows users to create a topic > > > > name that includes `-partition-`, which is confusing for our developers > > > to > > > > identify whether this is a partition of a partitioned topic. Plus, we > > > need > > > > to add more logic to be compatible with this special topic name. for > > > > example: > > > > > > > > > > > > - https://github.com/apache/pulsar/pull/19240 > > > > > > - https://github.com/apache/pulsar/pull/19230 > > > > > > - https://github.com/apache/pulsar/pull/19171 > > > > > > - https://github.com/apache/pulsar/pull/19086 > > > > > > - ... > > > > > > > > > > > > ### Goal > > > > > > This proposal wants `-partition-` to be a topic name keyword. Users > > > > can only create a topic with it if the topic is partitioned. For the > > > > compatibility reason, we want to Introduce a new configuration - > > > > `enableStrictTopicName` for the broker to help reject creating a topic > > in > > > > the following cases: > > > > > > 1. Create a partitioned topic that includes `-partition-`. > > > > > > 2. Create a topic which is not a partitioned topic. > > > > > > > > > > > > **Create a topic:** > > > > > > _no corresponding partitioned topic_ > > > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > > keyword) > > > > > > - persistent://public/default/local-name-partition-0 (rejected by > > > > keyword) > > > > > > > > > > > > _Has corresponding partitioned topic, **partitions=2** and topic > > > > partition name is **persistent://public/default/local-name**_ > > > > > > > > > > > > - persistent://public/default/local-name-partition-0 (passed, > > Because > > > > it is the partition topic's sub-partition) > > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > > keyword) > > > > > > - persistent://public/default/local-name-partition-4 (rejected, > > > > Because it exceeds the number of maximum partitions) > > > > > > > > > > > > **Create a partitioned topic(topic metadata)** > > > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > > keyword) > > > > > > - persistent://public/default/local-name-partition-0 (rejected by > > > > keyword) > > > > > > > > > > > > > > > > > > ### API Changes > > > > > > > > > > > > Add a new configuration, `enableStrictTopicName=false`. > > > > > > > > > > > > ### Implementation > > > > > > > > > > > > 1. Add configuration `enableStrictTopicName=false`. > > > > > > 2. Add rejection logic when the user enables > > `enableStrictTopicName`. > > > > > > 4. Add warning logs to inform users that we do not recommend > > creating > > > > non-partitioned topics with the keyword `-partition-`. > > > > > > 5. Make `enableStrictTopicName=true` in the next major release. > > > > > > > > > > > > > -- > > > Regards > > > Nitin Goyal > > > > >
Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
+1 (non-binding) Thanks, Zike Yang On Tue, Jan 31, 2023 at 5:32 PM 丛搏 wrote: > > +1 (binding) > > Thanks, > Bo > > ZhangJian He 于2023年1月31日周二 16:52写道： > > > > +1 (non-binding) > > > > Thanks > > ZhangJian He > > > > > > On Tue, 31 Jan 2023 at 16:51, PengHui Li wrote: > > > > > +1 (binding) > > > > > > Penghui > > > > > > On Tue, Jan 31, 2023 at 3:52 PM Nitin Goyal > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > On Tue, Jan 31, 2023 at 12:29 PM guo jiwei wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > Regards > > > > > Jiwei Guo (Tboy) > > > > > > > > > > On Tue, Jan 31, 2023 at 2:36 PM Yunze Xu > > > > > > > > wrote: > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Thanks, > > > > > > Yunze > > > > > > > > > > > > On Tue, Jan 31, 2023 at 6:57 AM wrote: > > > > > > > > > > > > > > Hello everyone. > > > > > > > > > > > > > > I would like to start the vote for PIP-242 > > > > > https://github.com/apache/pulsar/issues/19239, > > > > > > > Please let me know if you have any concerns or questions. > > > > > > > > > > > > > > Best, > > > > > > > Mattison > > > > > > > > > > > > > > --- Paste original PIP content to help quote -- > > > > > > > > > > > > > > ### Motivation > > > > > > > > > > > > > > Currently, the Apache Pulsar broker allows users to create a topic > > > > > name that includes `-partition-`, which is confusing for our > > > > > developers > > > > to > > > > > identify whether this is a partition of a partitioned topic. Plus, we > > > > need > > > > > to add more logic to be compatible with this special topic name. for > > > > > example: > > > > > > > > > > > > > > - https://github.com/apache/pulsar/pull/19240 > > > > > > > - https://github.com/apache/pulsar/pull/19230 > > > > > > > - https://github.com/apache/pulsar/pull/19171 > > > > > > > - https://github.com/apache/pulsar/pull/19086 > > > > > > > - ... > > > > > > > > > > > > > > ### Goal > > > > > > > This proposal wants `-partition-` to be a topic name keyword. > > > > > > > Users > > > > > can only create a topic with it if the topic is partitioned. For the > > > > > compatibility reason, we want to Introduce a new configuration - > > > > > `enableStrictTopicName` for the broker to help reject creating a topic > > > in > > > > > the following cases: > > > > > > > 1. Create a partitioned topic that includes `-partition-`. > > > > > > > 2. Create a topic which is not a partitioned topic. > > > > > > > > > > > > > > **Create a topic:** > > > > > > > _no corresponding partitioned topic_ > > > > > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > > > keyword) > > > > > > > - persistent://public/default/local-name-partition-0 (rejected by > > > > > keyword) > > > > > > > > > > > > > > _Has corresponding partitioned topic, **partitions=2** and topic > > > > > partition name is **persistent://public/default/local-name**_ > > > > > > > > > > > > > > - persistent://public/default/local-name-partition-0 (passed, > > > Because > > > > > it is the partition topic's sub-partition) > > > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > > > keyword) > > > > > > > - persistent://public/default/local-name-partition-4 (rejected, > > > > > Because it exceeds the number of maximum partitions) > > > > > > > > > > > > > > **Create a partitioned topic(topic metadata)** > > > > > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > > > - persistent://public/default/local-name-partition-z (rejected by > > > > > keyword) > > > > > > > - persistent://public/default/local-name-partition-0 (rejected by > > > > > keyword) > > > > > > > > > > > > > > > > > > > > > ### API Changes > > > > > > > > > > > > > > Add a new configuration, `enableStrictTopicName=false`. > > > > > > > > > > > > > > ### Implementation > > > > > > > > > > > > > > 1. Add configuration `enableStrictTopicName=false`. > > > > > > > 2. Add rejection logic when the user enables > > > `enableStrictTopicName`. > > > > > > > 4. Add warning logs to inform users that we do not recommend > > > creating > > > > > non-partitioned topics with the keyword `-partition-`. > > > > > > > 5. Make `enableStrictTopicName=true` in the next major release. > > > > > > > > > > > > > > > > > -- > > > > Regards > > > > Nitin Goyal > > > > > > >
Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
+1 (binding) But I DON'T think that this implementation should go into released versions. This is not a big problem for users and we MUST hold off adding code to released branches if the change is not strictly necessary: - bug that causes data corruption/data loss - security issue Any changes committed to released branches are likely to make the release less stable So the feature must be turned on and we could keep the flag to disable it. Please note that having this flag will force us to maintain compatibility forever Enrico Il giorno mar 31 gen 2023 alle ore 10:58 Zike Yang ha scritto: > > +1 (non-binding) > > Thanks, > Zike Yang > > On Tue, Jan 31, 2023 at 5:32 PM 丛搏 wrote: > > > > +1 (binding) > > > > Thanks, > > Bo > > > > ZhangJian He 于2023年1月31日周二 16:52写道： > > > > > > +1 (non-binding) > > > > > > Thanks > > > ZhangJian He > > > > > > > > > On Tue, 31 Jan 2023 at 16:51, PengHui Li wrote: > > > > > > > +1 (binding) > > > > > > > > Penghui > > > > > > > > On Tue, Jan 31, 2023 at 3:52 PM Nitin Goyal > > > > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > On Tue, Jan 31, 2023 at 12:29 PM guo jiwei > > > > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > Regards > > > > > > Jiwei Guo (Tboy) > > > > > > > > > > > > On Tue, Jan 31, 2023 at 2:36 PM Yunze Xu > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > Thanks, > > > > > > > Yunze > > > > > > > > > > > > > > On Tue, Jan 31, 2023 at 6:57 AM wrote: > > > > > > > > > > > > > > > > Hello everyone. > > > > > > > > > > > > > > > > I would like to start the vote for PIP-242 > > > > > > https://github.com/apache/pulsar/issues/19239, > > > > > > > > Please let me know if you have any concerns or questions. > > > > > > > > > > > > > > > > Best, > > > > > > > > Mattison > > > > > > > > > > > > > > > > --- Paste original PIP content to help quote -- > > > > > > > > > > > > > > > > ### Motivation > > > > > > > > > > > > > > > > Currently, the Apache Pulsar broker allows users to create a > > > > > > > > topic > > > > > > name that includes `-partition-`, which is confusing for our > > > > > > developers > > > > > to > > > > > > identify whether this is a partition of a partitioned topic. Plus, > > > > > > we > > > > > need > > > > > > to add more logic to be compatible with this special topic name. for > > > > > > example: > > > > > > > > > > > > > > > > - https://github.com/apache/pulsar/pull/19240 > > > > > > > > - https://github.com/apache/pulsar/pull/19230 > > > > > > > > - https://github.com/apache/pulsar/pull/19171 > > > > > > > > - https://github.com/apache/pulsar/pull/19086 > > > > > > > > - ... > > > > > > > > > > > > > > > > ### Goal > > > > > > > > This proposal wants `-partition-` to be a topic name keyword. > > > > > > > > Users > > > > > > can only create a topic with it if the topic is partitioned. For the > > > > > > compatibility reason, we want to Introduce a new configuration - > > > > > > `enableStrictTopicName` for the broker to help reject creating a > > > > > > topic > > > > in > > > > > > the following cases: > > > > > > > > 1. Create a partitioned topic that includes `-partition-`. > > > > > > > > 2. Create a topic which is not a partitioned topic. > > > > > > > > > > > > > > > > **Create a topic:** > > > > > > > > _no corresponding partitioned topic_ > > > > > > > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > > > > - persistent://public/default/local-name-partition-z (rejected > > > > > > > > by > > > > > > keyword) > > > > > > > > - persistent://public/default/local-name-partition-0 (rejected > > > > > > > > by > > > > > > keyword) > > > > > > > > > > > > > > > > _Has corresponding partitioned topic, **partitions=2** and topic > > > > > > partition name is **persistent://public/default/local-name**_ > > > > > > > > > > > > > > > > - persistent://public/default/local-name-partition-0 (passed, > > > > Because > > > > > > it is the partition topic's sub-partition) > > > > > > > > - persistent://public/default/local-name-partition-z (rejected > > > > > > > > by > > > > > > keyword) > > > > > > > > - persistent://public/default/local-name-partition-4 (rejected, > > > > > > Because it exceeds the number of maximum partitions) > > > > > > > > > > > > > > > > **Create a partitioned topic(topic metadata)** > > > > > > > > > > > > > > > > - persistent://public/default/local-name (passed) > > > > > > > > - persistent://public/default/local-name-partition-z (rejected > > > > > > > > by > > > > > > keyword) > > > > > > > > - persistent://public/default/local-name-partition-0 (rejected > > > > > > > > by > > > > > > keyword) > > > > > > > > > > > > > > > > > > > > > > > > ### API Changes > > > > > > > > > > > > > > > > Add a new configuration, `enableStrictTopicName=false`. > > > > > > > > > > > > > > > > ### Implementation > > > > > > > > > > > > > > > > 1. Add configuration `enableSt
Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic
> - Have we considered a metadata store to persist and dedup deletion > requests instead of the system topic? Why is the system topic the better > choice than a metadata store for this problem? If we use the metadata store to store the middle step ledger, we need to operate the metadata store after deletion every time. And we need a trigger to trigger deletion. In the broker, it may have lots of topics, the ledger deletion is also much. Using the metadata store to store it may be a bottleneck. Using pub/sub is easy to implement, and it is a good trigger to trigger deletion. > - How does Pulsar deduplicate deletion requests(error out to users) while > the deletion request is running? The user only can invoke `truncateTopic`, it's not for a particular ledger. The note: "The truncate operation will move all cursors to the end of the topic and delete all inactive ledgers." It's just a trigger for the user. > - How do users track async deletion flow status? (do we expose any > describeDeletion API to show the deletion status?) Why need to track the async deletion flow status? The ledger deletion is transparent for pulsarClient. In the broker, deleting a ledger will print the log `delete ledger xx successfully `. If delete failed, it print the log `delete ledger xxx failed.`
Re: [VOTE] Reactive Java client for Apache Pulsar 0.2.0 Candidate 1 (updated)
The KEYS file is here: https://dist.apache.org/repos/dist/release/pulsar/KEYS +1 (binding) I checked - signatures. - RAT/license headers - LICENSE - NOTICE (please update copyright to 2023) I tried to compile, but am having issues with `gradle build` For the next release please update the README.adoc with the new version in advance and provide clear, reproducible build instructions. Best, Dave > On Jan 30, 2023, at 4:39 PM, Christophe Bornet wrote: > > New voting thread with links to artifacts signed with an rsa4096 key. > Please ignore the previous mails and vote on this one. > > This is the release candidate 1 for the Reactive Java client for Apache > Pulsar, version 0.2.0. > > *** Please download, test and vote on this release. This vote will stay > open for at least 72 hours *** > > Note that we are voting upon the source (tag). Binaries in the Maven > repository are provided for convenience. > > Source package: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-reactive-0.2.0-candidate-1/ > > SHA-512 checksums: > c6953d145415ff6a9e991ee74d736511a663cdf9e739c89d72d74b1fef95047550b459a36871c73a771ed09d15172765dd279c22a4a0fb98d14c90d7f8365ac7 > pulsar-client-reactive-0.2.0-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1205/ > > The tag to be voted upon: > v0.2.0-candidate-1 (20ced033a34af1cfdc4cbf61ea91e0e8a613edad) > https://github.com/apache/pulsar-client-reactive/releases/tag/v0.2.0-candidate-1 > > Please download the source package, and follow detailed instructions for > pulsar-client-reactive release validation at > https://github.com/apache/pulsar-client-reactive/wiki/Release-process#release-validation > . > > Best regards > > Christophe
Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic
On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote:
> > - Have we considered a metadata store to persist and dedup deletion
> > requests instead of the system topic? Why is the system topic the better
> > choice than a metadata store for this problem?
> If we use the metadata store to store the middle step ledger, we need to
> operate the metadata store after deletion every time.
>
> And we need a trigger to trigger deletion. In the broker, it may have lots
> of topics, the ledger deletion is also much. Using the metadata store to
> store it may be a bottleneck.
> Using pub/sub is easy to implement, and it is a good trigger to trigger
> deletion.
>
We can group the multiple resource deletions to a single record in the
metadata store. Also, we can use the metadata store watcher to trigger the
deletion.
I can see that a similar transactional operation(using metadata store) can
be done like the following.
Alternatively,
1. A broker receives a resource(ledger) deletion request from a client.
2. If the target resource is available, the broker persists a transaction
lock(/transactions/broker-id/delete_ledger/ledger_id) into a metadata
store(state:pending, createdAt:now).
2.1 If there is no target resource, error
out(ResourceDoesNotExistException).
2.2 If the lock already exists, error out(OperationInProgressExeception).
3. The broker returns success to the client.
4. The transaction watcher(metadata store listener) on the same broker-id
is notified.
5. The transaction watcher runs the deletion process with an x min timeout.
5.1 The transaction watcher updates the lock state (state: running,
startedAt: now)
5.2 Run step 1 ... n (periodically update the lock state and
updatedAt:now every x secs)
5.3 Delete the lock.
6. The orphan transaction monitor runs any orphan jobs by retrying step 5.
(If the watcher fails in the middle at step 5, the lock state will be
orphan(state:running and startedAt : > x min))
7. The leader monitor(on the leader broker) manages orphan jobs if brokers
are gone or unavailable.
We can have multiple types of transaction locks(or generic lock) depending
on the operations types. This will reduce the number of locks to
create/update if there are multiple target resources to operate on for a
single transaction.
- Single ledger deletion: /transactions/broker-id/delete_ledger/ledger_id
- Mult-ledger deletion: /transactions/broker-id/delete_ledgers/ledgers :
{ledger_ids[a,b,c,d], last_deleted_ledger_index:3}
//last_deleted_ledger_index could be periodically updated every min. This
can help to resume the deletion when retrying.
- Topic deletion : /transactions/broker-id/delete_topic/topic_name
> > - How does Pulsar deduplicate deletion requests(error out to users) while
> > the deletion request is running?
> The user only can invoke `truncateTopic`, it's not for a particular
> ledger. The note: "The truncate operation will move all cursors to the end
> of the topic and delete all inactive ledgers."
> It's just a trigger for the user.
>
What if the admin concurrently requests `truncateTopic` many times for the
same topic while one truncation job is running? How does Pulsar currently
deduplicate these requests? And how does this proposal handle this
situation?
>
> > - How do users track async deletion flow status? (do we expose any
> > describeDeletion API to show the deletion status?)
> Why need to track the async deletion flow status? The ledger deletion is
> transparent for pulsarClient. In the broker, deleting a ledger will print
> the log `delete ledger xx successfully `.
> If delete failed, it print the log `delete ledger xxx failed.`
>
IMHO, relying on logs to check the system state is not a good practice.
Generally, every async user/admin API(long-running async workflow API)
needs the corresponding describe* API to return the current running state.
Regards,
Heesung
Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
Passed the voting by 8 +1 (5 binding and 3 non-binding) Closed. Best, Mattison On Jan 31, 2023, 06:57 +0800, mattisonc...@gmail.com, wrote: > Hello everyone. > > I would like to start the vote for PIP-242 > https://github.com/apache/pulsar/issues/19239, > Please let me know if you have any concerns or questions. > > Best, > Mattison > > --- Paste original PIP content to help quote -- > > ### Motivation > > Currently, the Apache Pulsar broker allows users to create a topic name that > includes `-partition-`, which is confusing for our developers to identify > whether this is a partition of a partitioned topic. Plus, we need to add more > logic to be compatible with this special topic name. for example: > > - https://github.com/apache/pulsar/pull/19240 > - https://github.com/apache/pulsar/pull/19230 > - https://github.com/apache/pulsar/pull/19171 > - https://github.com/apache/pulsar/pull/19086 > - ... > > ### Goal > This proposal wants `-partition-` to be a topic name keyword. Users can only > create a topic with it if the topic is partitioned. For the compatibility > reason, we want to Introduce a new configuration - `enableStrictTopicName` > for the broker to help reject creating a topic in the following cases: > 1. Create a partitioned topic that includes `-partition-`. > 2. Create a topic which is not a partitioned topic. > > **Create a topic:** > _no corresponding partitioned topic_ > > - persistent://public/default/local-name (passed) > - persistent://public/default/local-name-partition-z (rejected by keyword) > - persistent://public/default/local-name-partition-0 (rejected by keyword) > > _Has corresponding partitioned topic, **partitions=2** and topic partition > name is **persistent://public/default/local-name**_ > > - persistent://public/default/local-name-partition-0 (passed, Because it is > the partition topic's sub-partition) > - persistent://public/default/local-name-partition-z (rejected by keyword) > - persistent://public/default/local-name-partition-4 (rejected, Because it > exceeds the number of maximum partitions) > > **Create a partitioned topic(topic metadata)** > > - persistent://public/default/local-name (passed) > - persistent://public/default/local-name-partition-z (rejected by keyword) > - persistent://public/default/local-name-partition-0 (rejected by keyword) > > > ### API Changes > > Add a new configuration, `enableStrictTopicName=false`. > > ### Implementation > > 1. Add configuration `enableStrictTopicName=false`. > 2. Add rejection logic when the user enables `enableStrictTopicName`. > 4. Add warning logs to inform users that we do not recommend creating > non-partitioned topics with the keyword `-partition-`. > 5. Make `enableStrictTopicName=true` in the next major release.
Pulsar Python client 3.1.0 release
There is a serious regression bug in Pulsar C++ client 3.1.0 [1], on which the Python client is based. And the Python client 3.0.0 also introduces a deadlock problem [2]. In addition, now the Python client supports Python 3.11. I'll start a 3.1.0 release for Pulsar Python client. [1] https://lists.apache.org/thread/81dwlq1vb21pq1jt83okqln8oyc3c2wj [2] https://github.com/apache/pulsar-client-python/pull/87 Thanks, Yunze
Re: [ANNOUNCE] New Committer: Baodi Shi
Congratulations! Max Xu 于2023年1月30日周一 12:34写道： > Congratulations! Baodi > > Best, > Max Xu > > > On Wed, Jan 18, 2023 at 9:36 PM Yunze Xu wrote: > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > Baodi Shi (https://github.com/shibd) to become a committer and we are > > pleased to announce that he has accepted. > > > > Being a committer enables easier contribution to the project since > > there is no need to go via the patch submission process. This should > > enable better productivity. > > > > Welcome and congratulations, Baodi Shi! > > > > Please join us in congratulating and welcoming Baodi Shi onboard! > > > > Thanks, > > Yunze on behalf of the Pulsar PMC > > > -- BR, Qiang Huang
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congrats!!! Max Xu 于2023年1月30日周一 12:33写道： > Congratulations! Bo > > Best, > Max Xu > > > On Wed, Jan 18, 2023 at 9:50 PM PengHui Li wrote: > > > Hi all, > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > > (https://github.com/congbobo184) as a member of the PMC and we are > > pleased to announce that he has accepted. > > > > He is very active in the community in the past few years and made a lot > of > > great contributions > > such as transactions and schemas. > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > Best Regards, > > Penghui on behalf of the Pulsar PMC > > > -- BR, Qiang Huang
Re: [ANNOUNCE] Nicolò Boschi as new PMC member in Apache Pulsar
Congratulations !!! Zike Yang 于2023年1月30日周一 16:41写道： > Congratulations! > > BR, > Zike Yang > > On Mon, Jan 30, 2023 at 3:16 PM houxiaoyu wrote: > > > > Congratulations ! > > > > Best > > Xiaoyu Hou > > > > Max Xu 于2023年1月30日周一 12:34写道： > > > > > Congratulations! Nicolò > > > > > > Best, > > > Max Xu > > > > > > > > > On Fri, Jan 20, 2023 at 8:36 PM Lari Hotari > wrote: > > > > > > > Dear Community, > > > > > > > > We are thrilled to announce that Nicolò Boschi > > > > (https://github.com/nicoloboschi) has been invited and has accepted > the > > > > role of member of the Apache Pulsar Project Management Committee > (PMC). > > > > > > > > Nicolò Boschi has been a vital asset to our community, consistently > > > > demonstrating his dedication and active participation through > > > > significant contributions such as the development of the Pulsar Shell > > > > and numerous bug fixes, security enhancements, and improvements to > > > > Pulsar and its CI pipeline. In addition to his technical > contributions, > > > > Nicolò also plays an important role in reviewing pull requests and > > > > ensuring the overall quality of our project. We look forward to his > > > > continued contributions. > > > > > > > > On behalf of the Pulsar PMC, we extend a warm welcome and > > > > congratulations to Nicolò Boschi. > > > > > > > > Best regards, Lari, on behalf of the Pulsar PMC > > > > > > > > -- BR, Qiang Huang
Pulsar Nodejs client 1.8.1 release
As discussed in [0], there is a regression bug in Pulsar C++ client 3.1.0 on which the Nodejs client is based. The C++ client 3.1.1 has been released to fix this bug [1]. We need to release a new Nodejs client to get this fix. I volunteer to be the release manager. The new release would be mainly the C++ client upgrade. So this release version would be 1.8.1 according to the Version Rule of the Nodejs client [2]. [0] https://lists.apache.org/thread/81dwlq1vb21pq1jt83okqln8oyc3c2wj [1] https://lists.apache.org/thread/0z6fjmv2cp3ds0gyfwr4cq36p79tw2lc [2] https://github.com/apache/pulsar-client-node/blob/master/docs/release-process.md#versioning BR, Zike Yang
[DISCUSS] Release Pulsar 2.10.4
Hello, Pulsar community: I'd like to propose releasing Apache Pulsar 2.10.4. It's been about one month since 2.10.3 was released. There are 45 PRs [0] needed to cherry-pick in branch-2.10. I will cherry-pick these PRs for branch-2.10. Exclude some PRs that merge directly into branch-2.10. There are 21 PRs [1] opened. I'll follow up on each of those PRs to see if they will be completed soon or will need to be pushed to 2.10.4 If you have any important fixes or any questions, please reply to this email, and we will evaluate whether to include them in 2.10.4 Thanks, Xiangying [0] - https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.4+-label%3Acherry-picked%2Fbranch-2.10+ [1] - https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.4+-label%3Acherry-picked%2Fbranch-2.10+
Re: [DISCUSS] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
If we are going to create a flag named "enableStrictTopicName", we should discuss more requirements on topic names than just "partitioned". At the moment, topic names are not validated using the `NamedEntity` class's this logic [0]. Enabling stricting topic names should enable this validation call. We already run this validation on tenant and namespace names. Further, I think we should reserve topics that start with __ for system topics. In this case, we still need a way to create these topics, so it would be sufficient to let super users create these topics. If we keep the scope as is, I don't think we should introduce this feature flag. Thanks, Michael [0] https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java#L28-L41 On Mon, Jan 30, 2023 at 3:26 PM Nicolò Boschi wrote: > > +1 for going with default = true in the next release (3.0/2.12) > > Nicolò Boschi > > > Il giorno lun 30 gen 2023 alle ore 05:31 ha > scritto: > > > Hi, Asaf, Yunze > > > You mean to say, if the topic is partitioned, the word "partition" can > > notappear in the submitted topic name, in the topic creation API? > > It's a little bit confusing. I will give some examples to help explain: > > > > Create a topic: > > > > • no corresponding partitioned topic > > • persistent://public/default/local-name (passed) > > • persistent://public/default/local-name-partition-z (rejected by > > keyword) > > • persistent://public/default/local-name-partition-0 (rejected by > > keyword) > > • Has corresponding partitioned topic, partitions=2 and topic partition > > name is persistent://public/default/local-name > > • persistent://public/default/local-name-partition-0 (passed, Because > > it is the partition topic's sub-partition) > > • persistent://public/default/local-name-partition-z (rejected by > > keyword) > > • persistent://public/default/local-name-partition-4 (rejected, > > Because it exceeds the number of maximum partitions) > > > > > > Create a partitioned topic(topic metadata) > > > > • persistent://public/default/local-name (passed) > > • persistent://public/default/local-name-partition-z (rejected by keyword) > > • persistent://public/default/local-name-partition-0 (rejected by keyword) > > > > > I think this PIP should go close to this end to end, meaning the last > > stepbeing making it default true.Otherwise, we end up having so many > > "feature flags" turned off, it's hardto navigate and improve Pulsar. > > Yes, we will add a warning log-in step 3 at the current release and enable > > it by default in the next major release(3.0?). > > > > Thanks a lot! > > > > Best, > > Mattison > > On Jan 30, 2023, 00:17 +0800, Asaf Mesika , wrote: > > > You mean to say, if the topic is partitioned, the word "partition" can > > not appear in the submitted topic name, in the topic creation API? > >
Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.
We did not have this discussion or vote open long enough. The discussion started on a Saturday and then the vote started on Monday. The vote was closed 24 hours later. I just sent a note to the discussion thread. I would like to discuss increasing the scope of this PIP before we officially close this vote. Thanks, Michael On Tue, Jan 31, 2023 at 4:29 PM wrote: > > Passed the voting by 8 +1 (5 binding and 3 non-binding) > > Closed. > > Best, > Mattison > On Jan 31, 2023, 06:57 +0800, mattisonc...@gmail.com, wrote: > > Hello everyone. > > > > I would like to start the vote for PIP-242 > > https://github.com/apache/pulsar/issues/19239, > > Please let me know if you have any concerns or questions. > > > > Best, > > Mattison > > > > --- Paste original PIP content to help quote -- > > > > ### Motivation > > > > Currently, the Apache Pulsar broker allows users to create a topic name > > that includes `-partition-`, which is confusing for our developers to > > identify whether this is a partition of a partitioned topic. Plus, we need > > to add more logic to be compatible with this special topic name. for > > example: > > > > - https://github.com/apache/pulsar/pull/19240 > > - https://github.com/apache/pulsar/pull/19230 > > - https://github.com/apache/pulsar/pull/19171 > > - https://github.com/apache/pulsar/pull/19086 > > - ... > > > > ### Goal > > This proposal wants `-partition-` to be a topic name keyword. Users can > > only create a topic with it if the topic is partitioned. For the > > compatibility reason, we want to Introduce a new configuration - > > `enableStrictTopicName` for the broker to help reject creating a topic in > > the following cases: > > 1. Create a partitioned topic that includes `-partition-`. > > 2. Create a topic which is not a partitioned topic. > > > > **Create a topic:** > > _no corresponding partitioned topic_ > > > > - persistent://public/default/local-name (passed) > > - persistent://public/default/local-name-partition-z (rejected by keyword) > > - persistent://public/default/local-name-partition-0 (rejected by keyword) > > > > _Has corresponding partitioned topic, **partitions=2** and topic partition > > name is **persistent://public/default/local-name**_ > > > > - persistent://public/default/local-name-partition-0 (passed, Because it is > > the partition topic's sub-partition) > > - persistent://public/default/local-name-partition-z (rejected by keyword) > > - persistent://public/default/local-name-partition-4 (rejected, Because it > > exceeds the number of maximum partitions) > > > > **Create a partitioned topic(topic metadata)** > > > > - persistent://public/default/local-name (passed) > > - persistent://public/default/local-name-partition-z (rejected by keyword) > > - persistent://public/default/local-name-partition-0 (rejected by keyword) > > > > > > ### API Changes > > > > Add a new configuration, `enableStrictTopicName=false`. > > > > ### Implementation > > > > 1. Add configuration `enableStrictTopicName=false`. > > 2. Add rejection logic when the user enables `enableStrictTopicName`. > > 4. Add warning logs to inform users that we do not recommend creating > > non-partitioned topics with the keyword `-partition-`. > > 5. Make `enableStrictTopicName=true` in the next major release.
