Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
Hi Bo, If we have the `seek` API that accepts a topic name, how to use seek for a single topic consumer and multi-topics consumer will be different. ```java var msg = singleTopicConsumer.receive(); var msgId = singleTopicConsumer.getMessageId(); consumer.seek(msgId); ``` ```java var msg = multiTopicsConsumer.receive(); var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); consumer.seek(msgId.getOwnerTopic(), msgId); ``` It's not as clear as you have thought. A question could come from the code above: since we can get the key (topic name) from `msgId` itself, why do we need another argument? What's worse is that users have to specify the correct topic name. For a partitioned topic, if users specified another partition, the `seek` operation would fail. If they specified something like `multiTopicsConsumer.getTopic()`, it would also fail because other APIs like `Consumer#getTopic()` doesn't return the correct topic name. If there is only one correct topic name for a given TopicMessageId, what's the meaning of making it as a required argument? BTW, let's see Kafka client's commit API: ```java public void commitSync(Map offsets) ``` What's different is that the offset in Kafka can represent a position of ANY partition, while the MessageId in Pulsar can only represent the position of A SPECIFIC partition. And in Pulsar, we also do not expose the partition concept, if we introduce the seek API with the topic name as the argument, we have to explain in detail about what's the topic name for a partition. It could be a very confusing thing from my experience when I explained the "partition" concept in community. Thanks, Yunze On Wed, Dec 21, 2022 at 3:20 PM 丛搏 wrote: > > Hi Yunze, > > add `TopicMessageId ` will couple messageID and `topic name` together, > which is very unclear for non-partition-topic. > > ``` > void seek(String topicName, MessageId messageId) throws PulsarClientException; > List> getLastTopicMessageId() throws > PulsarClientException; > ``` > If the interface is designed in this way, it may be simpler, easier to > understand, and more intuitive for users, and MessageID will not be > coupled with TopicName. > > because this PIP has already initiated a VOTE, so I will sync this > reply to PIP-224-VOTE[0] > > Thanks, > Bo > [0] https://lists.apache.org/thread/mbrpjsgrgwrlkdpvkk738jxnlk7rf4qk > > Yunze Xu 于2022年12月9日周五 14:33写道： > > > > Hi Jiaqi, > > > > Let's move to > > https://lists.apache.org/thread/mbrpjsgrgwrlkdpvkk738jxnlk7rf4qk > > for the vote. > > > > Thanks, > > Yunze > > > > On Fri, Dec 9, 2022 at 1:54 PM Jiaqi Shen wrote: > > > > > > This is make sense to me, +1 > > > > > > Thanks, > > > Jiaqi Shen > > > > > > > > > Yunze Xu 于2022年12月7日周三 13:51写道： > > > > > > > Hi Baodi, > > > > > > > > I decided not to change the behavior of the `negativeAcknowledge` > > > > method. I just checked again that there is no exception signature for > > > > this method and there is no asynchronous version like > > > > `negativeAcknowledgeAsync`. To keep the API compatible, we should not > > > > add an exception signature, which would be required if a > > > > `PulsarClientException` was thrown. > > > > > > > > Thanks, > > > > Yunze > > > > > > > > On Tue, Nov 29, 2022 at 10:12 PM Baodi Shi > > > > > > > > wrote: > > > > > > > > > > Hi, Yunze: > > > > > > > > > > Thanks for your proposal. That Looks good to me. > > > > > > > > > > `negativeAcknowledge` also needs to add the same checks as the new > > > > acknowledge interface. > > > > > > > > > > > This interface doesn't add any acknowledge overload because the > > > > overloads are already too many. But it will make the behavior clear. > > > > > I think since we exposed the TopicMessageId, it would be better to add > > > > overloaded interfaces (even if the overloads are a lot). This can users > > > > to > > > > clearly associate the use cases of MultiTopicConsumer and > > > > TopicMessageId. > > > > > > > > > > Also, while it's okay to use TopicMessageId param on a single > > > > > consumer, > > > > I guess we shouldn't allow users to use it. > > > > > > > > > > In this way, users are clearly aware that TopicMessageId is used when > > > > using MultiTopicConsumer and MessageId is used when using > > > > SingleTopicConsumer.(Maybe it's not a good idea) > > > > > > > > > > > > > > > Thanks, > > > > > Baodi Shi > > > > > > > > > > > 2022年11月29日 15:57，Yunze Xu 写道： > > > > > > > > > > > >> Is there a case where the user uses the messageId returned by the > > > > > > producer to seek in the consumer? Is this a good behavior? > > > > > > > > > > > > Yes. I think it should be acceptable. To correct my previous point, > > > > > > now I think the MessageId returned by send should also be able to be > > > > > > applied for seek or acknowledge. > > > > > > > > > > > >> even with the > > > > > > current proposal, it may return null when getting the topic from > > > > > > TopicMessageId for backward compatibility. > > > > > > > > > > > > No.
[DISCUSS] [GO-Client] The behavior of Seek is different with Java Client
Hi all. I noticed that the behavior of go client Seek is different with Java client, which may confuse the users. More details here: https://github.com/apache/pulsar-client-go/issues/919. Overall, if go client executes `Seek(id)`, it will reset its position in `id` instead of `id+1` by default. It's different with Java Client. I think we should modify it to make sure that Go Client `Seek()` has the same behavior with Java Client. Even though it will bring the breaking change. Please feel free to leave your comments. Thanks, Jiaqi Shen
Re: [DISCUSS] Change the default IO threads and listener threads of Java Client
I have a concern about the message ordering. If we have more than 1 listener thread by default, could messages from the same topic be passed to different listener threads? Thanks, Yunze On Wed, Dec 21, 2022 at 11:12 AM 丛搏 wrote: > > +1 > Our default configuration is best for most users. Multiple clients are > a few cases. > > Thanks, > Bo > > houxiaoyu 于2022年12月20日周二 16:02写道： > > > > +1 > > > > This change might bring thread number increment in case users create many > > clients, but too many pulsar clients run in one machine is not a good use > > case I think, so this change looks good to me. > > > > Thanks, > > Xiaoyu Hou > > > > 于2022年12月20日周二 12:25写道： > > > > > +1 > > > > > > My concern is whether this change will affect some users who are creating > > > many clients. I think we can wait for other users to confirm it. (If this > > > will be affected, maybe we can give it a max_io_thread_num and then expand > > > the size from 1 to max_io_thread_num when adding a new consumer or > > > producer) > > > > > > > > > Best, > > > Mattison > > > On Dec 20, 2022, 11:17 +0800, PengHui Li , wrote: > > > > Hi all, > > > > > > > > I noticed the Java Client (I haven't checked other clients) uses 1 IO > > > > thread and 1 listener > > > > thread by default. It will require users to update the thread > > > configuration > > > > if they have > > > > multiple cores and desired high throughput. > > > > > > > > Here is the example that we change to 16 IO threads in > > > > openmessaging benchmark > > > > > > > https://github.com/openmessaging/benchmark/blob/master/driver-pulsar/pulsar.yaml#L22 > > > > > > > > We can apply the configuration of the threads based on the CPU cores. So > > > > that for the > > > > most common cases, users don't need to touch the thread configuration. > > > > > > > > ``` > > > > private int numIoThreads = Runtime.getRuntime().availableProcessors(); > > > > private int numListenerThreads = > > > Runtime.getRuntime().availableProcessors(); > > > > ``` > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Penghui > > >
Re: [DISCUSS] Introduce oshi library to sensory OS resources
+1 Yubiao Feng 于2022年12月21日周三 11:59写道： > +1 > > On Mon, Dec 19, 2022 at 6:19 PM wrote: > > > > > Hi, All > > > > I would like to introduce a new library oshi[1] to help Apache Pulsar > > sensory OS resources. It can help us to get away from the complex file > > manipulation and cross-platform compatibility issues in some operating > > systems. > > > > code example: https://github.com/apache/pulsar/pull/18984 > > > > Please feel free to left comments. > > > > > > Best, > > Mattison > > > > > > [1] https://github.com/oshi/oshi > > >
Re: [DISCUSS] [GO-Client] The behavior of Seek is different with Java Client
+1 to me. It's a breaking change, but I think it's more like a wrong implementation in Go client. Thanks, Yunze On Wed, Dec 21, 2022 at 5:00 PM Jiaqi Shen wrote: > > Hi all. > > I noticed that the behavior of go client Seek is different with Java > client, which may confuse the users. > > More details here: https://github.com/apache/pulsar-client-go/issues/919. > > Overall, if go client executes `Seek(id)`, it will reset its position in > `id` instead of `id+1` by default. It's different with Java Client. > > I think we should modify it to make sure that Go Client `Seek()` has the > same behavior with Java Client. Even though it will bring the breaking > change. > > Please feel free to leave your comments. > > Thanks, > Jiaqi Shen
Re: [DISCUSS] Introduce oshi library to sensory OS resources
+1 Thanks, Yunze On Wed, Dec 21, 2022 at 5:02 PM houxiaoyu wrote: > > +1 > > Yubiao Feng 于2022年12月21日周三 11:59写道： > > > +1 > > > > On Mon, Dec 19, 2022 at 6:19 PM wrote: > > > > > > > > Hi, All > > > > > > I would like to introduce a new library oshi[1] to help Apache Pulsar > > > sensory OS resources. It can help us to get away from the complex file > > > manipulation and cross-platform compatibility issues in some operating > > > systems. > > > > > > code example: https://github.com/apache/pulsar/pull/18984 > > > > > > Please feel free to left comments. > > > > > > > > > Best, > > > Mattison > > > > > > > > > [1] https://github.com/oshi/oshi > > > > >
[VOTE] Pulsar Node.js Client Release 1.8.0 Candidate 1
Hi everyone, This is the first release candidate for the Apache Pulsar Node.js client, version 1.8.0. It fixes the following issues: https://github.com/apache/pulsar-client-node/milestone/13?closed=1 Please download the source files and review this release candidate: - Download the source package, verify shasum and asc - Follow the README.md to build and run the Pulsar Node.js client. I have created a repo here to help you verify it more easily: https://github.com/RobertIndie/pulsar-client-node-validation. Please read the README file for more information. The automatic release validation has been passed: https://github.com/RobertIndie/pulsar-client-node-validation/actions/runs/3747965037 The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source files: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.0-candidate-1/ Pulsar's KEYS file containing PGP keys we use to sign the release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS SHA-512 checksum: 98326718eb7de20b3ebb12a4908ae195ec56e506b8c465b14eaa2e4191a2778bda6dbdbd1e2ceb654a1fdc7bb6d908a3bd8018e9f49e40215d3534a568105aff pulsar-client-node-1.8.0.tar.gz The tag to be voted upon: v1.8.0-rc.1 https://github.com/apache/pulsar-client-node/releases/tag/v1.8.0-rc.1 Please review and vote on the release candidate #1 for the version 1.8.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Thanks, Zike Yang
Re: [DISCUSS] [GO-Client] The behavior of Seek is different with Java Client
Hi Jiaqi, Thanks for raising up this discussion. This is the wrong behavior for the go client, and it will make it confusing for the go client user. +1 for fixing this wrong implementation even though it will introduce breaking changes. Thanks, Zike Yang On Wed, Dec 21, 2022 at 5:05 PM Yunze Xu wrote: > > +1 to me. It's a breaking change, but I think it's more like a wrong > implementation in Go client. > > Thanks, > Yunze > > On Wed, Dec 21, 2022 at 5:00 PM Jiaqi Shen wrote: > > > > Hi all. > > > > I noticed that the behavior of go client Seek is different with Java > > client, which may confuse the users. > > > > More details here: https://github.com/apache/pulsar-client-go/issues/919. > > > > Overall, if go client executes `Seek(id)`, it will reset its position in > > `id` instead of `id+1` by default. It's different with Java Client. > > > > I think we should modify it to make sure that Go Client `Seek()` has the > > same behavior with Java Client. Even though it will bring the breaking > > change. > > > > Please feel free to leave your comments. > > > > Thanks, > > Jiaqi Shen
Re: [DISCUSS] Introduce oshi library to sensory OS resources
+1 Thanks, Zike Yang On Wed, Dec 21, 2022 at 5:06 PM Yunze Xu wrote: > > +1 > > Thanks, > Yunze > > On Wed, Dec 21, 2022 at 5:02 PM houxiaoyu wrote: > > > > +1 > > > > Yubiao Feng 于2022年12月21日周三 11:59写道： > > > > > +1 > > > > > > On Mon, Dec 19, 2022 at 6:19 PM wrote: > > > > > > > > > > > Hi, All > > > > > > > > I would like to introduce a new library oshi[1] to help Apache Pulsar > > > > sensory OS resources. It can help us to get away from the complex file > > > > manipulation and cross-platform compatibility issues in some operating > > > > systems. > > > > > > > > code example: https://github.com/apache/pulsar/pull/18984 > > > > > > > > Please feel free to left comments. > > > > > > > > > > > > Best, > > > > Mattison > > > > > > > > > > > > [1] https://github.com/oshi/oshi > > > > > > >
Re: [VOTE] Pulsar Client Python Release 3.0.0 Candidate 3
+1 - Checked the signature - Install the python .whl file (pip3 install pulsar_client-3.0.0-cp38-cp38-macosx_10_15_universal2.whl) on macOS 13.0.1 (22A400) - Start the standalone (build from the master branch) - Start consumer (python3 ./examples/consumer.py) - Start producer (python3 ./examples/producer.py) -- Penghui On Sat, Dec 17, 2022 at 10:35 AM Matteo Merli wrote: > +1 > > * From candidate-2 I checked the source tarball for licenses, notice > and no binaries included. > -- > Matteo Merli > > > On Thu, Dec 15, 2022 at 11:30 PM Yunze Xu > wrote: > > > > This is the third release candidate for Apache Pulsar Client Python, > > version 3.0.0. > > > > It fixes the following issues: > > https://github.com/apache/pulsar-client-python/milestone/1?closed=1 > > > > *** Please download, test and vote on this release. This vote will > > stay open for at least 72 hours *** > > > > Python wheels: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.0.0-candidate-3/ > > > > The supported python versions are 3.7, 3.8, 3.9 and 3.10. The > > supported platforms and architectures are: > > - Windows x86_64 (windows/) > > - glibc-based Linux x86_64 (linux-glibc-x86_64/) > > - glibc-based Linux arm64 (linux-glibc-arm64/) > > - musl-based Linux x86_64 (linux-musl-x86_64/) > > - musl-based Linux arm64 (linux-musl-arm64/) > > - macOS universal 2 (macos/) > > > > The tag to be voted upon: v3.0.0-candidate-3 > > (46acc487ad16fdc0aeea9dae64484030e62c1b96) > > > https://github.com/apache/pulsar-client-python/releases/tag/v3.0.0-candidate-3 > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Please download the Python wheels and follow the README to test. > > > > Thanks, > > Yunze >
Re: [VOTE] Pulsar Release 2.9.4 Candidate 3
+1 (non-binding) - Checked the signature - Build from source - Start standalone with KoP 2.9.3.20 - Verified Pulsar client 2.9.4, master (05e6f5e3), and Kafka clients 3.3.1 (through KoP) Thanks, Yunze On Mon, Dec 19, 2022 at 11:13 AM Xiangying Meng wrote: > > +1 (non-binding) > > - Start standalone > - Validate Pub/Sub and Java Functions > - Validate Cassandra connector > - Validate Stateful Functions > > Thanks, > Xiangying > > On Thu, Dec 15, 2022 at 10:25 AM PengHui Li wrote: > > > +1 (binding) > > > > - Checked the signature > > - Start standalone > > - Publish and consume messages > > - Verified Function and State Function > > - Verified Cassandra connector > > - Build from the source package > > > > Thanks, > > Penghui > > > > On Tue, Dec 13, 2022 at 7:49 PM 丛搏 wrote: > > > > > This is the third release candidate for Apache Pulsar, version 2.9.4. > > > > > > > > > This release contains 319 commits by 69 contributors. > > > https://github.com/apache/pulsar/compare/v2.9.3...v2.9.4-candidate-3 > > > > > > *** Please download, test and vote on this release. This vote will stay > > > open > > > for at least 72 hours *** > > > > > > Note that we are voting upon the source (tag), binaries are provided for > > > convenience. > > > > > > Source and binary files: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.4-candidate-3/ > > > > > > SHA-512 checksums: > > > > > > > > 85cd920c8fedcec2551867e1ea89052c8578634e95226f92c4114d17587e7d2821f8033ef6fc70103e0b21dd3f8f9b907c68209cdc2cb74eca08f0a3ae6bd98c > > > apache-pulsar-2.9.4-bin.tar.gz > > > > > > > > da6ee53ffc66e4d9f60c74935c3ed0d85b26f5a629cb50fdfc02f535d66492297932256e4e44c8d4a08d20a85c4f490b7d7b3e169756bc246690bedfe582892b > > > apache-pulsar-2.9.4-src.tar.gz > > > > > > Maven staging repo: > > > https://repository.apache.org/content/repositories/orgapachepulsar-1198/ > > > > > > The tag to be voted upon: > > > v2.9.4-candidate-3 (e949f18a20c6f8f5b6f326cd4afb814d0fb3b8be) > > > https://github.com/apache/pulsar/releases/tag/v2.9.4-candidate-3 > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > Docker images: > > > > > > > > > > > > > > https://hub.docker.com/layers/congbobo184/pulsar/2.9.4/images/sha256-72272e9b7ce5c568575bacbddf7565fd570d27b486f2f47cafaa0938ec56e1ef > > > > > > > > > > > > > > https://hub.docker.com/layers/congbobo184/pulsar-all/2.9.4/images/sha256-c17d42831a882028996627abe56e71e067b905fdaac91ca3bdc933d51ce5b73b > > > > > > > > > Please download the source package, and follow the README to build > > > and run the Pulsar standalone service. > > > > >
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
Hi, Yunze: < ```java < var msg = multiTopicsConsumer.receive(); < var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); < consumer.seek(msgId.getOwnerTopic(), msgId); < ``` the code can be like this: ```java var msg = anyConsumer.receive(); var msgId = anyConsume.getMessageId(); consumer.seek(msg, msgId); ``` If messageID does not contain `TopicName`, the `TopicName` is best get from msg. < What's different is that the offset in Kafka can represent a position < of ANY partition, while the MessageId in Pulsar can only represent the < position of A SPECIFIC partition. Although MessageId in Pulsar can only represent the position of A SPECIFIC partition, but it still needs a TopicName. `LedgerID` and `EntryID` do not mean that this `MessageID` belongs to a topic (although it does belong), but it still cannot avoid `TopicName` for marking this `MessageID` belongs to this topic. > And in Pulsar, we also do not expose > the partition concept, if we introduce the seek API with the topic > name as the argument, we have to explain in detail about what's the > topic name for a partition. It could be a very confusing thing from my > experience when I explained the "partition" concept in community. if using `TopicMessageId` also has the same problem, why we need to use `TopicMessageId` not `MessageId` Thanks, Bo Yunze Xu 于2022年12月21日周三 16:59写道： > > Hi Bo, > > If we have the `seek` API that accepts a topic name, how to use seek > for a single topic consumer and multi-topics consumer will be > different. > > ```java > var msg = singleTopicConsumer.receive(); > var msgId = singleTopicConsumer.getMessageId(); > consumer.seek(msgId); > ``` > > ```java > var msg = multiTopicsConsumer.receive(); > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > consumer.seek(msgId.getOwnerTopic(), msgId); > ``` > > It's not as clear as you have thought. A question could come from the > code above: since we can get the key (topic name) from `msgId` itself, > why do we need another argument? > > What's worse is that users have to specify the correct topic name. For > a partitioned topic, if users specified another partition, the `seek` > operation would fail. If they specified something like > `multiTopicsConsumer.getTopic()`, it would also fail because other > APIs like `Consumer#getTopic()` doesn't return the correct topic name. > > If there is only one correct topic name for a given TopicMessageId, > what's the meaning of making it as a required argument? > > BTW, let's see Kafka client's commit API: > > ```java > public void commitSync(Map offsets) > ``` > > What's different is that the offset in Kafka can represent a position > of ANY partition, while the MessageId in Pulsar can only represent the > position of A SPECIFIC partition. And in Pulsar, we also do not expose > the partition concept, if we introduce the seek API with the topic > name as the argument, we have to explain in detail about what's the > topic name for a partition. It could be a very confusing thing from my > experience when I explained the "partition" concept in community. > > Thanks, > Yunze > > > On Wed, Dec 21, 2022 at 3:20 PM 丛搏 wrote: > > > > Hi Yunze, > > > > add `TopicMessageId ` will couple messageID and `topic name` together, > > which is very unclear for non-partition-topic. > > > > ``` > > void seek(String topicName, MessageId messageId) throws > > PulsarClientException; > > List> getLastTopicMessageId() throws > > PulsarClientException; > > ``` > > If the interface is designed in this way, it may be simpler, easier to > > understand, and more intuitive for users, and MessageID will not be > > coupled with TopicName. > > > > because this PIP has already initiated a VOTE, so I will sync this > > reply to PIP-224-VOTE[0] > > > > Thanks, > > Bo > > [0] https://lists.apache.org/thread/mbrpjsgrgwrlkdpvkk738jxnlk7rf4qk > > > > Yunze Xu 于2022年12月9日周五 14:33写道： > > > > > > Hi Jiaqi, > > > > > > Let's move to > > > https://lists.apache.org/thread/mbrpjsgrgwrlkdpvkk738jxnlk7rf4qk > > > for the vote. > > > > > > Thanks, > > > Yunze > > > > > > On Fri, Dec 9, 2022 at 1:54 PM Jiaqi Shen wrote: > > > > > > > > This is make sense to me, +1 > > > > > > > > Thanks, > > > > Jiaqi Shen > > > > > > > > > > > > Yunze Xu 于2022年12月7日周三 13:51写道： > > > > > > > > > Hi Baodi, > > > > > > > > > > I decided not to change the behavior of the `negativeAcknowledge` > > > > > method. I just checked again that there is no exception signature for > > > > > this method and there is no asynchronous version like > > > > > `negativeAcknowledgeAsync`. To keep the API compatible, we should not > > > > > add an exception signature, which would be required if a > > > > > `PulsarClientException` was thrown. > > > > > > > > > > Thanks, > > > > > Yunze > > > > > > > > > > On Tue, Nov 29, 2022 at 10:12 PM Baodi Shi > > > > > > > > > > wrote: > > > > > > > > > > > > Hi, Yunze: > > > > > > > > > > > > Thanks for your proposal. That Looks good to me. > > > > >
Re: [VOTE] Pulsar Client Python Release 3.0.0 Candidate 3
+1 (non-binding) python version: 3.7.6 - Checked the signature - Install the python .whl file (pip3 install pulsar_client-3.0.0-cp37-cp37m-macosx_10_15_universal2.whl) on macOS 12.3.1 - Start the standalone (2.10.2) - Start consumer (python3 ./examples/consumer.py) - Start producer (python3 ./examples/producer.py ) Thanks, Bo PengHui Li 于2022年12月21日周三 18:01写道： > > +1 > > - Checked the signature > - Install the python .whl file (pip3 install > pulsar_client-3.0.0-cp38-cp38-macosx_10_15_universal2.whl) on macOS 13.0.1 > (22A400) > - Start the standalone (build from the master branch) > - Start consumer (python3 ./examples/consumer.py) > - Start producer (python3 ./examples/producer.py) > > -- > Penghui > > On Sat, Dec 17, 2022 at 10:35 AM Matteo Merli > wrote: > > > +1 > > > > * From candidate-2 I checked the source tarball for licenses, notice > > and no binaries included. > > -- > > Matteo Merli > > > > > > On Thu, Dec 15, 2022 at 11:30 PM Yunze Xu > > wrote: > > > > > > This is the third release candidate for Apache Pulsar Client Python, > > > version 3.0.0. > > > > > > It fixes the following issues: > > > https://github.com/apache/pulsar-client-python/milestone/1?closed=1 > > > > > > *** Please download, test and vote on this release. This vote will > > > stay open for at least 72 hours *** > > > > > > Python wheels: > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.0.0-candidate-3/ > > > > > > The supported python versions are 3.7, 3.8, 3.9 and 3.10. The > > > supported platforms and architectures are: > > > - Windows x86_64 (windows/) > > > - glibc-based Linux x86_64 (linux-glibc-x86_64/) > > > - glibc-based Linux arm64 (linux-glibc-arm64/) > > > - musl-based Linux x86_64 (linux-musl-x86_64/) > > > - musl-based Linux arm64 (linux-musl-arm64/) > > > - macOS universal 2 (macos/) > > > > > > The tag to be voted upon: v3.0.0-candidate-3 > > > (46acc487ad16fdc0aeea9dae64484030e62c1b96) > > > > > https://github.com/apache/pulsar-client-python/releases/tag/v3.0.0-candidate-3 > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > Please download the Python wheels and follow the README to test. > > > > > > Thanks, > > > Yunze > >
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
> If messageID does not contain `TopicName`, the `TopicName` is best get from > msg. If you mean `msg.getTopicName()`, how can you declare it's better than `msgId.getOwnerTopic()`? > but it still cannot avoid `TopicName` for marking this `MessageID` belongs to > this topic. It can. Because the `TopicMessageIdImpl` already contains the correct topic. That's the point. > if using `TopicMessageId` also has the same problem, why we need to use > `TopicMessageId` not `MessageId` Because `TopicMessageId` is constructed by the Pulsar Client library itself, which can guarantee `getOwnerTopic()` returns the correct topic name. The benefit of passing a `TopicMessageId` rather than the combination of a topic name and a `MessageId` is, users won't need to care about how to get the correct topic name for a given partition by themselves. The key point is that if there is only one valid value for an argument, which relies on the other argument, then the API design is bad. Assume you need to use the Pulsar client like: ``` // numberOfMessages must be the same with msgIds.size(), otherwise, an exception will be thrown consumer.acknowledge(numberOfMessages, msgIds); ``` With the API of this proposal, users don't need to care much about how to call `seek` correctly, except the MessageId is returned by Producer#send. `consumer.seek(msg.getMessageId())` works for all cases. With the `seek(String, MessageId)` API, you have to write more explanations like: 1. If the consumer only subscribes to a topic, use `consumer.seek(msg.getMessageId())`. 2. If the consumer subscribes to multiple topics, use `consumer.seek(topic, msg.getMessageId())`. The topic must be what the message belongs to, so you have to use the correct topic like `consumer.seek(msg.getTopicName(), msg.getMessageId()`. Otherwise, seek would fail. I don't know what you're thinking about using `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single topic consumer. If it's accepted, and you want to unify the use case of `seek`, the original `seek` API should be deprecated and much existing code could be affected. If it's not accepted, users have to distinguish if a consumer is a multi-topics consumer. Thanks, Yunze On Wed, Dec 21, 2022 at 8:50 PM 丛搏 wrote: > > Hi, Yunze: > > < ```java > < var msg = multiTopicsConsumer.receive(); > < var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > < consumer.seek(msgId.getOwnerTopic(), msgId); > < ``` > > the code can be like this: > ```java > var msg = anyConsumer.receive(); > var msgId = anyConsume.getMessageId(); > consumer.seek(msg, msgId); > ``` > If messageID does not contain `TopicName`, the `TopicName` is best get from > msg. > > < What's different is that the offset in Kafka can represent a position > < of ANY partition, while the MessageId in Pulsar can only represent the > < position of A SPECIFIC partition. > > Although MessageId in Pulsar can only represent the position of A > SPECIFIC partition, but it still needs a TopicName. `LedgerID` and > `EntryID` do not mean that this `MessageID` belongs to a topic > (although it does belong), but it still cannot avoid `TopicName` for > marking this `MessageID` belongs to this topic. > > > And in Pulsar, we also do not expose > > the partition concept, if we introduce the seek API with the topic > > name as the argument, we have to explain in detail about what's the > > topic name for a partition. It could be a very confusing thing from my > > experience when I explained the "partition" concept in community. > > if using `TopicMessageId` also has the same problem, why we need to > use `TopicMessageId` not `MessageId` > > Thanks, > Bo > > Yunze Xu 于2022年12月21日周三 16:59写道： > > > > Hi Bo, > > > > If we have the `seek` API that accepts a topic name, how to use seek > > for a single topic consumer and multi-topics consumer will be > > different. > > > > ```java > > var msg = singleTopicConsumer.receive(); > > var msgId = singleTopicConsumer.getMessageId(); > > consumer.seek(msgId); > > ``` > > > > ```java > > var msg = multiTopicsConsumer.receive(); > > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > > consumer.seek(msgId.getOwnerTopic(), msgId); > > ``` > > > > It's not as clear as you have thought. A question could come from the > > code above: since we can get the key (topic name) from `msgId` itself, > > why do we need another argument? > > > > What's worse is that users have to specify the correct topic name. For > > a partitioned topic, if users specified another partition, the `seek` > > operation would fail. If they specified something like > > `multiTopicsConsumer.getTopic()`, it would also fail because other > > APIs like `Consumer#getTopic()` doesn't return the correct topic name. > > > > If there is only one correct topic name for a given TopicMessageId, > > what's the meaning of making it as a required argument? > > > > BTW, let's see Kafka client's commit API: > > > > ```java > > public void commitSync(Map
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
> If you mean `msg.getTopicName()`, how can you declare it's better than > `msgId.getOwnerTopic()`? > It can. Because the `TopicMessageIdImpl` already contains the correct > topic. That's the point. ``` var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); ``` if `msgId.getOwnerTopic()` is the interface of `MessageId`, I have no problem. but it needs to cast the `TopicMessageId` from `MessageId`, which is very user-unfriendly. And it doesn't make sense. > I don't know what you're thinking about using > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > topic consumer. If it's accepted, and you want to unify the use case > of `seek`, the original `seek` API should be deprecated and much > existing code could be affected. If it's not accepted, users have to > distinguish if a consumer is a multi-topics consumer. the same as `consumer.seek(TopicMessageId topicMessageId)` for a single topic consumer is also strange. My point is either TopicName belongs to MessageId or separate the two. it's not a good interface implementation to couple them together. Very unclear. Thanks, Bo Yunze Xu 于2022年12月21日周三 22:46写道： > > > If messageID does not contain `TopicName`, the `TopicName` is best get from > > msg. > > If you mean `msg.getTopicName()`, how can you declare it's better than > `msgId.getOwnerTopic()`? > > > but it still cannot avoid `TopicName` for marking this `MessageID` belongs > > to this topic. > > It can. Because the `TopicMessageIdImpl` already contains the correct > topic. That's the point. > > > if using `TopicMessageId` also has the same problem, why we need to use > > `TopicMessageId` not `MessageId` > > Because `TopicMessageId` is constructed by the Pulsar Client library > itself, which can guarantee `getOwnerTopic()` returns the correct > topic name. The benefit of passing a `TopicMessageId` rather than the > combination of a topic name and a `MessageId` is, users won't need to > care about how to get the correct topic name for a given partition by > themselves. > > The key point is that if there is only one valid value for an > argument, which relies on the other argument, then the API design is > bad. Assume you need to use the Pulsar client like: > > ``` > // numberOfMessages must be the same with msgIds.size(), otherwise, an > exception will be thrown > consumer.acknowledge(numberOfMessages, msgIds); > ``` > > With the API of this proposal, users don't need to care much about how > to call `seek` correctly, except the MessageId is returned by > Producer#send. `consumer.seek(msg.getMessageId())` works for all > cases. > > With the `seek(String, MessageId)` API, you have to write more > explanations like: > 1. If the consumer only subscribes to a topic, use > `consumer.seek(msg.getMessageId())`. > 2. If the consumer subscribes to multiple topics, use > `consumer.seek(topic, msg.getMessageId())`. The topic must be what the > message belongs to, so you have to use the correct topic like > `consumer.seek(msg.getTopicName(), msg.getMessageId()`. Otherwise, > seek would fail. > > I don't know what you're thinking about using > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > topic consumer. If it's accepted, and you want to unify the use case > of `seek`, the original `seek` API should be deprecated and much > existing code could be affected. If it's not accepted, users have to > distinguish if a consumer is a multi-topics consumer. > > Thanks, > Yunze > > On Wed, Dec 21, 2022 at 8:50 PM 丛搏 wrote: > > > > Hi, Yunze: > > > > < ```java > > < var msg = multiTopicsConsumer.receive(); > > < var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > > < consumer.seek(msgId.getOwnerTopic(), msgId); > > < ``` > > > > the code can be like this: > > ```java > > var msg = anyConsumer.receive(); > > var msgId = anyConsume.getMessageId(); > > consumer.seek(msg, msgId); > > ``` > > If messageID does not contain `TopicName`, the `TopicName` is best get from > > msg. > > > > < What's different is that the offset in Kafka can represent a position > > < of ANY partition, while the MessageId in Pulsar can only represent the > > < position of A SPECIFIC partition. > > > > Although MessageId in Pulsar can only represent the position of A > > SPECIFIC partition, but it still needs a TopicName. `LedgerID` and > > `EntryID` do not mean that this `MessageID` belongs to a topic > > (although it does belong), but it still cannot avoid `TopicName` for > > marking this `MessageID` belongs to this topic. > > > > > And in Pulsar, we also do not expose > > > the partition concept, if we introduce the seek API with the topic > > > name as the argument, we have to explain in detail about what's the > > > topic name for a partition. It could be a very confusing thing from my > > > experience when I explained the "partition" concept in community. > > > > if using `TopicMessageId` also has the same problem, why we need to > > use `TopicMessageId` not `MessageId` > > >
Re: [VOTE] Pulsar Release 2.11.0 Candidate-3
+1 (non-binding) - Checked the signature - Build from source - Start standalone with KoP master (1e5433d) - Verify Pulsar client 2.11.0, Kafka client 3.3.1 (through KoP) - Verify standalone with ZooKeeper as the metadata store - Verify Docker images Thanks, Yunze On Mon, Dec 19, 2022 at 12:47 PM PengHui Li wrote: > > +1 (binding) > > - Checked the signature > - Build from source > - Start standalone with zookeeper > - Done some simple performance test( publish, consume, and drain backlog) > - Checked function > - Checked Cassandra connector > - Checked stateful function > > Thanks, > Penghui > > On Sun, Dec 18, 2022 at 8:33 PM 丛搏 wrote: > > > +1 (non-binding) > > > > system: mac os 12.6, Apple M1 > > maven: 3.8.5 > > java: OpenJDK 17.0.3 > > > > - Checked the signature > > - Start standalone with zookeeper stream storage > > - Publish and consume messages > > - Verified Function and State Function > > - Verified Cassandra connector > > - Build from the source package > > > > Thanks, > > Bo > > > > guo jiwei 于2022年12月17日周六 11:39写道： > > > > > > This is the third release candidate for Apache Pulsar, version 2.11.0. > > > > > > This release contains 1605 commits by 62 contributors. > > > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-3 > > > > > > CI for this release candidate > > > https://github.com/Technoboy-/pulsar/pull/21 > > > > > > *** Please download, test and vote on this release. This vote will stay > > open > > > for at least 72 hours *** > > > > > > Note that we are voting upon the source (tag), binaries are provided for > > > convenience. > > > > > > Source and binary files: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-3 > > > > > > SHA-512 checksums: > > > > > > > > 096014c7a1bb7975c9eb7b7796d225e69cac066ca2e228019a7fffbde08a7a48d377932e5d7a2dca6adf51428e2aa52b1e47035624a554b9e30da024e7933256 > > > > > > ./apache-pulsar-2.11.0-bin.tar.gz > > > > > > > > b8e9bb39f6687190c05b6186bdc80cb0499b6600e8e9a29b942c91e650a64614d5205bf527fa47c1005aa57da96d0a9c2506b8ff2f51332fa8e8beef7bdaf7b3 > > > > > > ./apache-pulsar-2.11.0-src.tar.gz > > > > > > Maven staging repo: > > > https://repository.apache.org/content/repositories/orgapachepulsar-1199/ > > > > > > The tag to be voted upon: > > > v2.11.0-candidate-3 (a1f88ff83f2eba822fdfa3b62b01ed75c6dcb9b3) > > > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-3 > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > Docker images: > > > > > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc3/images/sha256-039de7f6c124adf5ee94b05879b4f833a4b8c1d7395a53f95b3cd8171d9f316b > > > > > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc3/images/sha256-f6f693227a467f7bd3f518224ec99b3c4f23f290fff07449ef18c18e2b0cee6d > > > > > > Please download the source package, and follow the > > > release-candidate-validation doc to build > > > and run the Pulsar standalone service. > > > https://pulsar.apache.org/contribute/validate-release-candidate > > > > > > Since the metadata store is changed from ZK to RocksDB, the verification > > of > > > the `stateful functions` needs to set the parameter "export > > > PULSAR_STANDALONE_USE_ZOOKEEPER=1" > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > >
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
> but it needs to cast the `TopicMessageId` from `MessageId`, which is very > user-unfriendly. Sorry I think my proposal doesn't express it well. In my original thought, no cast is needed, please see the update in https://github.com/apache/pulsar/issues/18616. In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is a `TopicMessageId`. Thanks, Yunze On Wed, Dec 21, 2022 at 11:26 PM 丛搏 wrote: > > > If you mean `msg.getTopicName()`, how can you declare it's better than > > `msgId.getOwnerTopic()`? > > > It can. Because the `TopicMessageIdImpl` already contains the correct > > topic. That's the point. > > ``` > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > ``` > if `msgId.getOwnerTopic()` is the interface of `MessageId`, I have no > problem. but it needs to cast the `TopicMessageId` from `MessageId`, > which is very user-unfriendly. And it doesn't make sense. > > > I don't know what you're thinking about using > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > > topic consumer. If it's accepted, and you want to unify the use case > > of `seek`, the original `seek` API should be deprecated and much > > existing code could be affected. If it's not accepted, users have to > > distinguish if a consumer is a multi-topics consumer. > > the same as `consumer.seek(TopicMessageId topicMessageId)` for a > single topic consumer is also strange. My point is either TopicName > belongs to MessageId or separate the two. it's not a good interface > implementation to couple them together. Very unclear. > > Thanks, > Bo > > Yunze Xu 于2022年12月21日周三 22:46写道： > > > > > If messageID does not contain `TopicName`, the `TopicName` is best get > > > from msg. > > > > If you mean `msg.getTopicName()`, how can you declare it's better than > > `msgId.getOwnerTopic()`? > > > > > but it still cannot avoid `TopicName` for marking this `MessageID` > > > belongs to this topic. > > > > It can. Because the `TopicMessageIdImpl` already contains the correct > > topic. That's the point. > > > > > if using `TopicMessageId` also has the same problem, why we need to use > > > `TopicMessageId` not `MessageId` > > > > Because `TopicMessageId` is constructed by the Pulsar Client library > > itself, which can guarantee `getOwnerTopic()` returns the correct > > topic name. The benefit of passing a `TopicMessageId` rather than the > > combination of a topic name and a `MessageId` is, users won't need to > > care about how to get the correct topic name for a given partition by > > themselves. > > > > The key point is that if there is only one valid value for an > > argument, which relies on the other argument, then the API design is > > bad. Assume you need to use the Pulsar client like: > > > > ``` > > // numberOfMessages must be the same with msgIds.size(), otherwise, an > > exception will be thrown > > consumer.acknowledge(numberOfMessages, msgIds); > > ``` > > > > With the API of this proposal, users don't need to care much about how > > to call `seek` correctly, except the MessageId is returned by > > Producer#send. `consumer.seek(msg.getMessageId())` works for all > > cases. > > > > With the `seek(String, MessageId)` API, you have to write more > > explanations like: > > 1. If the consumer only subscribes to a topic, use > > `consumer.seek(msg.getMessageId())`. > > 2. If the consumer subscribes to multiple topics, use > > `consumer.seek(topic, msg.getMessageId())`. The topic must be what the > > message belongs to, so you have to use the correct topic like > > `consumer.seek(msg.getTopicName(), msg.getMessageId()`. Otherwise, > > seek would fail. > > > > I don't know what you're thinking about using > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > > topic consumer. If it's accepted, and you want to unify the use case > > of `seek`, the original `seek` API should be deprecated and much > > existing code could be affected. If it's not accepted, users have to > > distinguish if a consumer is a multi-topics consumer. > > > > Thanks, > > Yunze > > > > On Wed, Dec 21, 2022 at 8:50 PM 丛搏 wrote: > > > > > > Hi, Yunze: > > > > > > < ```java > > > < var msg = multiTopicsConsumer.receive(); > > > < var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > > > < consumer.seek(msgId.getOwnerTopic(), msgId); > > > < ``` > > > > > > the code can be like this: > > > ```java > > > var msg = anyConsumer.receive(); > > > var msgId = anyConsume.getMessageId(); > > > consumer.seek(msg, msgId); > > > ``` > > > If messageID does not contain `TopicName`, the `TopicName` is best get > > > from msg. > > > > > > < What's different is that the offset in Kafka can represent a position > > > < of ANY partition, while the MessageId in Pulsar can only represent the > > > < position of A SPECIFIC partition. > > > > > > Although MessageId in Pulsar can only represent the position of A > > > SPECIFIC partition, but it still needs a TopicName. `LedgerID` and > > > `EntryID` do not
Re: [DISCUSS] Change the default IO threads and listener threads of Java Client
Hi PengHui I notice that if changing the default value of conf `ioThreads` of pulsar client, it does not just affect the num of io threads, and it also affects these thread pools: - internal executor of the client - scheduled executor of the client - HTTP client io thread pool of pulsar admin client see: https://github.com/apache/pulsar/pull/19020 Should we provide other APIs to configure these three thread pools? Thanks Yubiao On Tue, Dec 20, 2022 at 11:17 AM PengHui Li wrote: > Hi all, > > I noticed the Java Client (I haven't checked other clients) uses 1 IO > thread and 1 listener > thread by default. It will require users to update the thread configuration > if they have > multiple cores and desired high throughput. > > Here is the example that we change to 16 IO threads in > openmessaging benchmark > > https://github.com/openmessaging/benchmark/blob/master/driver-pulsar/pulsar.yaml#L22 > > We can apply the configuration of the threads based on the CPU cores. So > that for the > most common cases, users don't need to touch the thread configuration. > > ``` > private int numIoThreads = Runtime.getRuntime().availableProcessors(); > private int numListenerThreads = > Runtime.getRuntime().availableProcessors(); > ``` > > WDYT? > > Thanks, > Penghui >
RE: Re: [DISCUSS] forbid user to upload `BYTES` schema
Hi Xiangying Meng, For the Pulsar client, even if users set BYTES schema, it won’t store anything in schema registry and we don’t need any special compatibility check. The key point here is, users can use admin api to upload a BYTES schema and it store a NONE type of schema in schema registry, which is unexpected and broke the compatibility checks. Tell users there is no schema just after user upload a BYTES schema is strange too. So I suggest to forbid users to upload BYTES schema. Thanks, Donglai On 2022/12/13 06:36:53 Xiangying Meng wrote: > Hi, Donglai: > Thanks for your discussion. > I think we should think about why we can not add a new schema of BYTES. > The users use the byte schema to send or consume messages. But the schema > registry has never stored the BYTES schema and we always need some special > compatibility checks for the BYTES schema. > I mean it is strange for users and difficult to maintain for developers. > Thanks, > Xiangying >
Re: [DISCUSS] Change the default IO threads and listener threads of Java Client
> I notice that if changing the default value of conf `ioThreads` of pulsar > client, it does not just affect the num of io threads, and it also affects > these thread pools: > - internal executor of the client > - scheduled executor of the client > - HTTP client io thread pool of pulsar admin client I think it's ok. Users can change the IO threads, it will also change the above thread pools. It could be different issues. We can support changing the scheduled executor threads and HTTP client IO threads independently once we need it. >From the performance test experience before. I don't see any obvious negative impact from the scheduled executor and HTTP client IO threads after changing the IO threads configuration. One salient point is if you don't change the IO threads, you will not be able to get an expected message producing and consuming throughput. Thanks, Penghui On Thu, Dec 22, 2022 at 12:35 AM Yubiao Feng wrote: > Hi PengHui > > I notice that if changing the default value of conf `ioThreads` of pulsar > client, it does not just affect the num of io threads, and it also affects > these thread pools: > - internal executor of the client > - scheduled executor of the client > - HTTP client io thread pool of pulsar admin client > > see: https://github.com/apache/pulsar/pull/19020 > > Should we provide other APIs to configure these three thread pools? > > Thanks > Yubiao > > On Tue, Dec 20, 2022 at 11:17 AM PengHui Li wrote: > > > Hi all, > > > > I noticed the Java Client (I haven't checked other clients) uses 1 IO > > thread and 1 listener > > thread by default. It will require users to update the thread > configuration > > if they have > > multiple cores and desired high throughput. > > > > Here is the example that we change to 16 IO threads in > > openmessaging benchmark > > > > > https://github.com/openmessaging/benchmark/blob/master/driver-pulsar/pulsar.yaml#L22 > > > > We can apply the configuration of the threads based on the CPU cores. So > > that for the > > most common cases, users don't need to touch the thread configuration. > > > > ``` > > private int numIoThreads = Runtime.getRuntime().availableProcessors(); > > private int numListenerThreads = > > Runtime.getRuntime().availableProcessors(); > > ``` > > > > WDYT? > > > > Thanks, > > Penghui > > >
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
> In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is a `TopicMessageId`. Does it look like we don't need to add the following new APIs? ``` void seek(TopicMessageId topicMessageId) throws PulsarClientException; CompletableFuture seekAsync(TopicMessageId topicMessageId); ``` Users can use ``` MessageId.fromByteArrayWithTopic() ``` or ``` TopicMessageId.fromByteArray(); TopicMessageId.create(); ``` to construct a TopicMessageId instance. But we can use the existing seek API with the constructed TopicMessageId instance ``` consumer.seek(MessageId messageId); ``` Thanks, Penghui On Thu, Dec 22, 2022 at 12:35 AM Yunze Xu wrote: > > but it needs to cast the `TopicMessageId` from `MessageId`, which is > very user-unfriendly. > > Sorry I think my proposal doesn't express it well. In my original > thought, no cast is needed, please see the update in > https://github.com/apache/pulsar/issues/18616. > > In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is > a `TopicMessageId`. > > Thanks, > Yunze > > On Wed, Dec 21, 2022 at 11:26 PM 丛搏 wrote: > > > > > If you mean `msg.getTopicName()`, how can you declare it's better than > > > `msgId.getOwnerTopic()`? > > > > > It can. Because the `TopicMessageIdImpl` already contains the correct > > > topic. That's the point. > > > > ``` > > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > > ``` > > if `msgId.getOwnerTopic()` is the interface of `MessageId`, I have no > > problem. but it needs to cast the `TopicMessageId` from `MessageId`, > > which is very user-unfriendly. And it doesn't make sense. > > > > > I don't know what you're thinking about using > > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > > > topic consumer. If it's accepted, and you want to unify the use case > > > of `seek`, the original `seek` API should be deprecated and much > > > existing code could be affected. If it's not accepted, users have to > > > distinguish if a consumer is a multi-topics consumer. > > > > the same as `consumer.seek(TopicMessageId topicMessageId)` for a > > single topic consumer is also strange. My point is either TopicName > > belongs to MessageId or separate the two. it's not a good interface > > implementation to couple them together. Very unclear. > > > > Thanks, > > Bo > > > > Yunze Xu 于2022年12月21日周三 22:46写道： > > > > > > > If messageID does not contain `TopicName`, the `TopicName` is best > get from msg. > > > > > > If you mean `msg.getTopicName()`, how can you declare it's better than > > > `msgId.getOwnerTopic()`? > > > > > > > but it still cannot avoid `TopicName` for marking this `MessageID` > belongs to this topic. > > > > > > It can. Because the `TopicMessageIdImpl` already contains the correct > > > topic. That's the point. > > > > > > > if using `TopicMessageId` also has the same problem, why we need to > use `TopicMessageId` not `MessageId` > > > > > > Because `TopicMessageId` is constructed by the Pulsar Client library > > > itself, which can guarantee `getOwnerTopic()` returns the correct > > > topic name. The benefit of passing a `TopicMessageId` rather than the > > > combination of a topic name and a `MessageId` is, users won't need to > > > care about how to get the correct topic name for a given partition by > > > themselves. > > > > > > The key point is that if there is only one valid value for an > > > argument, which relies on the other argument, then the API design is > > > bad. Assume you need to use the Pulsar client like: > > > > > > ``` > > > // numberOfMessages must be the same with msgIds.size(), otherwise, an > > > exception will be thrown > > > consumer.acknowledge(numberOfMessages, msgIds); > > > ``` > > > > > > With the API of this proposal, users don't need to care much about how > > > to call `seek` correctly, except the MessageId is returned by > > > Producer#send. `consumer.seek(msg.getMessageId())` works for all > > > cases. > > > > > > With the `seek(String, MessageId)` API, you have to write more > > > explanations like: > > > 1. If the consumer only subscribes to a topic, use > > > `consumer.seek(msg.getMessageId())`. > > > 2. If the consumer subscribes to multiple topics, use > > > `consumer.seek(topic, msg.getMessageId())`. The topic must be what the > > > message belongs to, so you have to use the correct topic like > > > `consumer.seek(msg.getTopicName(), msg.getMessageId()`. Otherwise, > > > seek would fail. > > > > > > I don't know what you're thinking about using > > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > > > topic consumer. If it's accepted, and you want to unify the use case > > > of `seek`, the original `seek` API should be deprecated and much > > > existing code could be affected. If it's not accepted, users have to > > > distinguish if a consumer is a multi-topics consumer. > > > > > > Thanks, > > > Yunze > > > > > > On Wed, Dec 21, 2022 at 8:50 PM 丛搏 wrote: > > > > > > > > Hi, Yunze: > > > > > > > > < ```j
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
> Because `TopicMessageId` is constructed by the Pulsar Client library itself, which can guarantee `getOwnerTopic()` returns the correct topic name. The benefit of passing a `TopicMessageId` rather than the combination of a topic name and a `MessageId` is, users won't need to care about how to get the correct topic name for a given partition by themselves. This is a good point of why we should not introduce seek(TopicName, MessageId) It's better also to mention this part in the proposal. It will help us to understand why seek(TopicName, MessageId) is not a good way for Pulsar. Thanks, Penghui On Thu, Dec 22, 2022 at 9:48 AM PengHui Li wrote: > > In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is > a `TopicMessageId`. > > Does it look like we don't need to add the following new APIs? > > ``` > void seek(TopicMessageId topicMessageId) throws PulsarClientException; > CompletableFuture seekAsync(TopicMessageId topicMessageId); > ``` > > Users can use > ``` > MessageId.fromByteArrayWithTopic() > ``` > or > ``` > TopicMessageId.fromByteArray(); > TopicMessageId.create(); > ``` > to construct a TopicMessageId instance. > > But we can use the existing seek API with the constructed TopicMessageId > instance > > ``` > consumer.seek(MessageId messageId); > ``` > > Thanks, > Penghui > > On Thu, Dec 22, 2022 at 12:35 AM Yunze Xu > wrote: > >> > but it needs to cast the `TopicMessageId` from `MessageId`, which is >> very user-unfriendly. >> >> Sorry I think my proposal doesn't express it well. In my original >> thought, no cast is needed, please see the update in >> https://github.com/apache/pulsar/issues/18616. >> >> In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is >> a `TopicMessageId`. >> >> Thanks, >> Yunze >> >> On Wed, Dec 21, 2022 at 11:26 PM 丛搏 wrote: >> > >> > > If you mean `msg.getTopicName()`, how can you declare it's better than >> > > `msgId.getOwnerTopic()`? >> > >> > > It can. Because the `TopicMessageIdImpl` already contains the correct >> > > topic. That's the point. >> > >> > ``` >> > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); >> > ``` >> > if `msgId.getOwnerTopic()` is the interface of `MessageId`, I have no >> > problem. but it needs to cast the `TopicMessageId` from `MessageId`, >> > which is very user-unfriendly. And it doesn't make sense. >> > >> > > I don't know what you're thinking about using >> > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single >> > > topic consumer. If it's accepted, and you want to unify the use case >> > > of `seek`, the original `seek` API should be deprecated and much >> > > existing code could be affected. If it's not accepted, users have to >> > > distinguish if a consumer is a multi-topics consumer. >> > >> > the same as `consumer.seek(TopicMessageId topicMessageId)` for a >> > single topic consumer is also strange. My point is either TopicName >> > belongs to MessageId or separate the two. it's not a good interface >> > implementation to couple them together. Very unclear. >> > >> > Thanks, >> > Bo >> > >> > Yunze Xu 于2022年12月21日周三 22:46写道： >> > > >> > > > If messageID does not contain `TopicName`, the `TopicName` is best >> get from msg. >> > > >> > > If you mean `msg.getTopicName()`, how can you declare it's better than >> > > `msgId.getOwnerTopic()`? >> > > >> > > > but it still cannot avoid `TopicName` for marking this `MessageID` >> belongs to this topic. >> > > >> > > It can. Because the `TopicMessageIdImpl` already contains the correct >> > > topic. That's the point. >> > > >> > > > if using `TopicMessageId` also has the same problem, why we need to >> use `TopicMessageId` not `MessageId` >> > > >> > > Because `TopicMessageId` is constructed by the Pulsar Client library >> > > itself, which can guarantee `getOwnerTopic()` returns the correct >> > > topic name. The benefit of passing a `TopicMessageId` rather than the >> > > combination of a topic name and a `MessageId` is, users won't need to >> > > care about how to get the correct topic name for a given partition by >> > > themselves. >> > > >> > > The key point is that if there is only one valid value for an >> > > argument, which relies on the other argument, then the API design is >> > > bad. Assume you need to use the Pulsar client like: >> > > >> > > ``` >> > > // numberOfMessages must be the same with msgIds.size(), otherwise, an >> > > exception will be thrown >> > > consumer.acknowledge(numberOfMessages, msgIds); >> > > ``` >> > > >> > > With the API of this proposal, users don't need to care much about how >> > > to call `seek` correctly, except the MessageId is returned by >> > > Producer#send. `consumer.seek(msg.getMessageId())` works for all >> > > cases. >> > > >> > > With the `seek(String, MessageId)` API, you have to write more >> > > explanations like: >> > > 1. If the consumer only subscribes to a topic, use >> > > `consumer.seek(msg.getMessageId())`. >> > > 2. If the consumer subscribes to multiple
Re: [VOTE] PIP-228: Refactor the information architecture of Pulsar client docs
+1 (binding) Thanks, Penghui On Wed, Dec 21, 2022 at 10:18 AM 丛搏 wrote: > +1 (non-binding) > > Thanks, > Bo > > Yu 于2022年12月21日周三 09:51写道： > > > > +1 > > > > On Tue, Dec 20, 2022 at 3:12 PM Yunze Xu > > wrote: > > > > > +1 (non-binding) > > > > > > Thanks, > > > Yunze > > > > > > On Tue, Dec 20, 2022 at 3:06 PM Zike Yang wrote: > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > On Tue, Dec 13, 2022 at 4:38 PM Jun Ma > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > I'm going to start the vote for PIP-228 [Refactor the information > > > architecture of Pulsar client docs]( > > > https://github.com/apache/pulsar/issues/18822). > > > > > > > > > > And this is the original thread for discussion: > > > https://lists.apache.org/thread/bv6lwnt708dxst173knyzv2bfy4d1ox4. > > > > > > > > > > The vote will be open for at least three days. > > > > > > > > > > > > > > > Thank you. > > > > > Jun > > > >
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
> Users can use > ``` > MessageId.fromByteArrayWithTopic() > ``` > or > ``` > TopicMessageId.fromByteArray(); > TopicMessageId.create(); > ``` I think this is a good point. `TopicName` don't in the `MessageId` means that multiConsumer must know that this topic is a partitioned topic and needs to assign `TopicName` to `MessageId` and doesn't need to add new interfaces. Thanks, Bo PengHui Li 于2022年12月22日周四 09:50写道： > > > Because `TopicMessageId` is constructed by the Pulsar Client library > itself, which can guarantee `getOwnerTopic()` returns the correct > topic name. The benefit of passing a `TopicMessageId` rather than the > combination of a topic name and a `MessageId` is, users won't need to > care about how to get the correct topic name for a given partition by > themselves. > > This is a good point of why we should not introduce seek(TopicName, > MessageId) > It's better also to mention this part in the proposal. It will help us to > understand > why seek(TopicName, MessageId) is not a good way for Pulsar. > > Thanks, > Penghui > > On Thu, Dec 22, 2022 at 9:48 AM PengHui Li wrote: > > > > In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is > > a `TopicMessageId`. > > > > Does it look like we don't need to add the following new APIs? > > > > ``` > > void seek(TopicMessageId topicMessageId) throws PulsarClientException; > > CompletableFuture seekAsync(TopicMessageId topicMessageId); > > ``` > > > > Users can use > > ``` > > MessageId.fromByteArrayWithTopic() > > ``` > > or > > ``` > > TopicMessageId.fromByteArray(); > > TopicMessageId.create(); > > ``` > > to construct a TopicMessageId instance. > > > > But we can use the existing seek API with the constructed TopicMessageId > > instance > > > > ``` > > consumer.seek(MessageId messageId); > > ``` > > > > Thanks, > > Penghui > > > > On Thu, Dec 22, 2022 at 12:35 AM Yunze Xu > > wrote: > > > >> > but it needs to cast the `TopicMessageId` from `MessageId`, which is > >> very user-unfriendly. > >> > >> Sorry I think my proposal doesn't express it well. In my original > >> thought, no cast is needed, please see the update in > >> https://github.com/apache/pulsar/issues/18616. > >> > >> In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is > >> a `TopicMessageId`. > >> > >> Thanks, > >> Yunze > >> > >> On Wed, Dec 21, 2022 at 11:26 PM 丛搏 wrote: > >> > > >> > > If you mean `msg.getTopicName()`, how can you declare it's better than > >> > > `msgId.getOwnerTopic()`? > >> > > >> > > It can. Because the `TopicMessageIdImpl` already contains the correct > >> > > topic. That's the point. > >> > > >> > ``` > >> > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > >> > ``` > >> > if `msgId.getOwnerTopic()` is the interface of `MessageId`, I have no > >> > problem. but it needs to cast the `TopicMessageId` from `MessageId`, > >> > which is very user-unfriendly. And it doesn't make sense. > >> > > >> > > I don't know what you're thinking about using > >> > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > >> > > topic consumer. If it's accepted, and you want to unify the use case > >> > > of `seek`, the original `seek` API should be deprecated and much > >> > > existing code could be affected. If it's not accepted, users have to > >> > > distinguish if a consumer is a multi-topics consumer. > >> > > >> > the same as `consumer.seek(TopicMessageId topicMessageId)` for a > >> > single topic consumer is also strange. My point is either TopicName > >> > belongs to MessageId or separate the two. it's not a good interface > >> > implementation to couple them together. Very unclear. > >> > > >> > Thanks, > >> > Bo > >> > > >> > Yunze Xu 于2022年12月21日周三 22:46写道： > >> > > > >> > > > If messageID does not contain `TopicName`, the `TopicName` is best > >> get from msg. > >> > > > >> > > If you mean `msg.getTopicName()`, how can you declare it's better than > >> > > `msgId.getOwnerTopic()`? > >> > > > >> > > > but it still cannot avoid `TopicName` for marking this `MessageID` > >> belongs to this topic. > >> > > > >> > > It can. Because the `TopicMessageIdImpl` already contains the correct > >> > > topic. That's the point. > >> > > > >> > > > if using `TopicMessageId` also has the same problem, why we need to > >> use `TopicMessageId` not `MessageId` > >> > > > >> > > Because `TopicMessageId` is constructed by the Pulsar Client library > >> > > itself, which can guarantee `getOwnerTopic()` returns the correct > >> > > topic name. The benefit of passing a `TopicMessageId` rather than the > >> > > combination of a topic name and a `MessageId` is, users won't need to > >> > > care about how to get the correct topic name for a given partition by > >> > > themselves. > >> > > > >> > > The key point is that if there is only one valid value for an > >> > > argument, which relies on the other argument, then the API design is > >> > > bad. Assume you need to use the Pulsar client like: > >> > > > >> > > ``` > >
[VOTE][PIP-226] Add JWKS support for AuthenticationProviderToken
Hi all, Voting for https://github.com/apache/pulsar/issues/18798. Thanks, Zixuan
[GitHub] [pulsar] momo-jun added a comment to the discussion: GCP Tiered Storage org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1
GitHub user momo-jun added a comment to the discussion: GCP Tiered Storage org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1 Sorry for my late reply. As discussed with @RobertIndie, we both think the docs are okay by clarifying the attribute (bucket name) and providing an example (pulsar-topic-offload). It would be too much if we add patches to not use the prefix. GitHub link: https://github.com/apache/pulsar/discussions/18868#discussioncomment-4472579 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [DISCUSS] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
Hi Penghui and Bo, I realized this problem as well. No new APIs should be added. I added the new API mainly to express that the semantics of seek will be modified while the acknowledge semantics keeps not changed. Now I removed the new APIs and updated this proposal. Thanks, Yunze On Thu, Dec 22, 2022 at 10:55 AM 丛搏 wrote: > > > Users can use > > ``` > > MessageId.fromByteArrayWithTopic() > > ``` > > or > > ``` > > TopicMessageId.fromByteArray(); > > TopicMessageId.create(); > > ``` > > I think this is a good point. `TopicName` don't in the `MessageId` > means that multiConsumer must know that this topic is a partitioned > topic and needs to assign `TopicName` to `MessageId` and doesn't need > to add new interfaces. > > Thanks, > Bo > > PengHui Li 于2022年12月22日周四 09:50写道： > > > > > Because `TopicMessageId` is constructed by the Pulsar Client library > > itself, which can guarantee `getOwnerTopic()` returns the correct > > topic name. The benefit of passing a `TopicMessageId` rather than the > > combination of a topic name and a `MessageId` is, users won't need to > > care about how to get the correct topic name for a given partition by > > themselves. > > > > This is a good point of why we should not introduce seek(TopicName, > > MessageId) > > It's better also to mention this part in the proposal. It will help us to > > understand > > why seek(TopicName, MessageId) is not a good way for Pulsar. > > > > Thanks, > > Penghui > > > > On Thu, Dec 22, 2022 at 9:48 AM PengHui Li wrote: > > > > > > In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is > > > a `TopicMessageId`. > > > > > > Does it look like we don't need to add the following new APIs? > > > > > > ``` > > > void seek(TopicMessageId topicMessageId) throws PulsarClientException; > > > CompletableFuture seekAsync(TopicMessageId topicMessageId); > > > ``` > > > > > > Users can use > > > ``` > > > MessageId.fromByteArrayWithTopic() > > > ``` > > > or > > > ``` > > > TopicMessageId.fromByteArray(); > > > TopicMessageId.create(); > > > ``` > > > to construct a TopicMessageId instance. > > > > > > But we can use the existing seek API with the constructed TopicMessageId > > > instance > > > > > > ``` > > > consumer.seek(MessageId messageId); > > > ``` > > > > > > Thanks, > > > Penghui > > > > > > On Thu, Dec 22, 2022 at 12:35 AM Yunze Xu > > > wrote: > > > > > >> > but it needs to cast the `TopicMessageId` from `MessageId`, which is > > >> very user-unfriendly. > > >> > > >> Sorry I think my proposal doesn't express it well. In my original > > >> thought, no cast is needed, please see the update in > > >> https://github.com/apache/pulsar/issues/18616. > > >> > > >> In short, `seek(msgId)` will call `seek(TopicMessageId)` if `msgId` is > > >> a `TopicMessageId`. > > >> > > >> Thanks, > > >> Yunze > > >> > > >> On Wed, Dec 21, 2022 at 11:26 PM 丛搏 wrote: > > >> > > > >> > > If you mean `msg.getTopicName()`, how can you declare it's better > > >> > > than > > >> > > `msgId.getOwnerTopic()`? > > >> > > > >> > > It can. Because the `TopicMessageIdImpl` already contains the correct > > >> > > topic. That's the point. > > >> > > > >> > ``` > > >> > var msgId = (TopicMessageId) multiTopicsConsumer.getMessageId(); > > >> > ``` > > >> > if `msgId.getOwnerTopic()` is the interface of `MessageId`, I have no > > >> > problem. but it needs to cast the `TopicMessageId` from `MessageId`, > > >> > which is very user-unfriendly. And it doesn't make sense. > > >> > > > >> > > I don't know what you're thinking about using > > >> > > `consumer.seek(msg.getTopicName(), msg.getMessageId()` for a single > > >> > > topic consumer. If it's accepted, and you want to unify the use case > > >> > > of `seek`, the original `seek` API should be deprecated and much > > >> > > existing code could be affected. If it's not accepted, users have to > > >> > > distinguish if a consumer is a multi-topics consumer. > > >> > > > >> > the same as `consumer.seek(TopicMessageId topicMessageId)` for a > > >> > single topic consumer is also strange. My point is either TopicName > > >> > belongs to MessageId or separate the two. it's not a good interface > > >> > implementation to couple them together. Very unclear. > > >> > > > >> > Thanks, > > >> > Bo > > >> > > > >> > Yunze Xu 于2022年12月21日周三 22:46写道： > > >> > > > > >> > > > If messageID does not contain `TopicName`, the `TopicName` is best > > >> get from msg. > > >> > > > > >> > > If you mean `msg.getTopicName()`, how can you declare it's better > > >> > > than > > >> > > `msgId.getOwnerTopic()`? > > >> > > > > >> > > > but it still cannot avoid `TopicName` for marking this `MessageID` > > >> belongs to this topic. > > >> > > > > >> > > It can. Because the `TopicMessageIdImpl` already contains the correct > > >> > > topic. That's the point. > > >> > > > > >> > > > if using `TopicMessageId` also has the same problem, why we need to > > >> use `TopicMessageId` not `MessageId` > > >> > > > > >> > > Because `TopicMessageId` is
