Re: [VOTE] Pulsar Node.js Client Release 1.8.0 Candidate 2
Hi, Baodi, > I don't think it's a good idea to upload (napi-xx.tar.gz) to npm, these files > are tens of MB in total, and it doesn't solve the problem that the release rc > version needs to change the address. If we add `Pulsar.node` (which currently included in napi-xx.tar.gz) to npm package file, then the npm will not need to download the Pulsar.node file from an external URL. In this way, we no longer need to specify the binary path. Is it possible to be implemented? It seems this should solve the issue of release rc package. Regarding the file size issue, The key point is whether we can make npm use different package files on different platforms, so we don't need to put all the Pulsar.node files of all platforms in one npm package file. > I think we can provide scripts to automatically modify these addresses and > modify the release process documentation to explain the steps of release rc > separately. That's a good idea. And I'm working on this. BR, Zike Yang On Sat, Jan 7, 2023 at 11:21 AM Baodi Shi wrote: > > Hi, Zike: > > > > However, doing so also creates some inconvenience for the release > > process. I was wondering if we could put these files(napi-xxx.tar.gz) > > in the npm package file. > > Do you have any ideas? > > I don't think it's a good idea to upload (napi-xx.tar.gz) to npm, these files > are tens of MB in total, and it doesn't solve the problem that the release rc > version needs to change the address. > > I think we can provide scripts to automatically modify these addresses and > modify the release process documentation to explain the steps of release rc > separately. > > > Thanks, > Baodi Shi
[DISCUSS] Registering Jackson Java 8 support modules by default for all Pulsar components, including client
Hi all, Jackson has a separate Java 8 support modules for adding support for proper serialization and deserialization of new classes that were added in Java 8 (Java 8 was released in 2014). These Jackson Java 8 support modules haven't been used in the Pulsar code base. This is a pity. This causes a lot of pain when using Java Time classes in Pulsar applications or Pulsar Functions. There are ways to get the classes working for applications, but the documentation is missing. It would make things easier if the Java 8 support modules for Jackson would be included and registered by default. I have created a PR to register Jackson Java 8 support modules by default for all Pulsar components. The PR is https://github.com/apache/pulsar/pull/19161 . Please review and provide feedback. Do we need a PIP for this change? -Lari
Re: [DISCUSS] Registering Jackson Java 8 support modules by default for all Pulsar components, including client
Hi, Lari: Will it affect compatibility? If it is just an improved function, I think it can also be added to the pulsar-common module. it adds the dependency, so it needs PIP to discuss. Thanks, Bo Lari Hotari 于2023年1月9日周一 19:06写道： > > Hi all, > > Jackson has a separate Java 8 support modules for adding support for proper > serialization and deserialization of new classes that were added in Java 8 > (Java 8 was released in 2014). > > These Jackson Java 8 support modules haven't been used in the Pulsar code > base. This is a pity. This causes a lot of pain when using Java Time classes > in Pulsar applications or Pulsar Functions. There are ways to get the classes > working for applications, but the documentation is missing. It would make > things easier if the Java 8 support modules for Jackson would be included and > registered by default. > > I have created a PR to register Jackson Java 8 support modules by default for > all Pulsar components. The PR is https://github.com/apache/pulsar/pull/19161 . > > Please review and provide feedback. Do we need a PIP for this change? > > -Lari
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
+0 (binding) - Checked signature - Checked license for pulsar and pulsar-shell - Build from the source code (MacOS Intel, JDK 17, Maven 3.8.6) - Run standalone Pulsar based on Zookeeper and RocksDB metastore - Run Pulsar-perf to produce and consume messages - Use pulsar-shell to get topics stats - Run pulsar-io-lakehouse connector based on Pulsar 2.11.0 When I ran Pulsar standalone based on RocksDB metastore, I found the following issues. - Create a 1 partitions topic failed with timeout using pulsar-admin - Use Pulsar perf to produce topics with 5000 partitions, produce failed with generate ledgerId timeout on the Broker side - Create a 1000 partitions topic succeed using pulsar-admin, but produce messages to the topic failed with timeout However, after I started Pulsar standalone based on Zookeeper metastore, the above issues disappeared, and the topic could produce and consume messages. Currently, the Pulsar standalone mode starts with RocksDB metastore by default since 2.11.0, but it can not run well in some cases. I'm unsure whether it will impact the current user who wants to start the pulsar service in standalone mode. Even though Pulsar standalone is not recommended to run in the production environment, we need to ensure it can perform the best performance on one local node. If the standalone issue is the expected behavior, I will give my +1. Thanks, Hang Enrico Olivelli 于2023年1月9日周一 15:55写道： > > +1 (binding) > > - checked signatures/digests/rat > - built from source > - run smoke tests on the built binaries > > > Thanks > Enrico > > Il giorno lun 9 gen 2023 alle ore 06:18 guo jiwei > ha scritto: > > > > +1 (binding) > > > > - Checked the signature > > - Build from source > > - Checked license(server and shell) > > - Start standalone with zookeeper > > - Checked function > > - Checked Cassandra connector > > - Checked stateful function > > > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Wed, Jan 4, 2023 at 8:25 PM Nicolò Boschi wrote: > > > > > About Pulsar Shell, > > > 1) zip file is distributed as a commodity for Windows users since the > > > .tar.gz archive is more tricky to extract. If this is a problem, we can > > > discuss it in another thread > > > 2) The bug on Windows is not a blocker for the release since it's not a > > > regression. We could fix it in 2.11.1 > > > > > > Here's my vote: > > > > > > +1 non binding > > > > > > - Signature and checksum > > > - LICENSE on server and shell > > > > > > - Rat check passes > > > > > > - Compile from source with JDK 17 on MacOs Intel > > > > > > - Run Pulsar standalone and produce-consume from CLI > > > > > > - Tested K8S installation with Datastax Pulsar helm chart and verified > > > TLS, > > > offloads, transactions and ElasticSearch sink > > > > > > - Pulsar Shell commands > > > > > > > > > Thanks, > > > Nicolò Boschi > > > > > > > > > Il giorno mer 4 gen 2023 alle ore 12:33 丛搏 ha scritto: > > > > > > > +1 (non-binding) > > > > > > > > system: mac os 12.6, Apple M1 > > > > maven: 3.8.5 > > > > java: OpenJDK 17.0.3 > > > > > > > > - Checked the signature > > > > - Checked LICENSE > > > > - Start standalone with zookeeper stream storage > > > > - Publish and consume messages > > > > - Verified Function and State Function > > > > - Verified Cassandra connector > > > > - Build from the source package > > > > - Run a simple transaction performance check > > > > > > > > Thanks, > > > > Bo > > > > > > > > Yunze Xu 于2023年1月4日周三 16:41写道： > > > > > > > > > > Okay, I will give my +1 (binding). > > > > > > > > > > Thanks, > > > > > Yunze > > > > > > > > > > On Wed, Jan 4, 2023 at 4:00 PM PengHui Li wrote: > > > > > > > > > > > > I think it's ok. > > > > > > 2.11.0 is the first release version of pulsar-shell > > > > > > So it's not a regression that was introduced in 2.11.0 and not > > > > > > critical security issues or license issues which will block users > > > move > > > > > > to the new version. > > > > > > > > > > > > Thanks, > > > > > > Penghui > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 4, 2023 at 3:36 PM Yunze Xu > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > +0 (binding) > > > > > > > > > > > > > > - Checked the signature > > > > > > > - Build from source (Java 17, Ubuntu 20.04 WSL2) > > > > > > > - Start standalone with KoP branch-2.11 (ab9d761f) > > > > > > > - Verify Pulsar client 2.11.0, Kafka client 3.3.1 (through KoP) > > > > > > > - Verify pulsar-shell on Ubuntu 20.04 WSL2 and Windows: > > > > > > > 1. Use `client` command to produce and consume messages > > > > > > > 2. Use `admin` command to create, list and delete topics > > > > > > > > > > > > > > I didn't give +1 because I have to modify the > > > > > > > `bin/pulsar-admin-common.cmd` to make `pulsar-shell` work on > > > Windows. > > > > > > > I left a command here: > > > > > > > https://github.com/apache/pulsar/pull/17243#discussion_r1061199501 > > > . > > > > > > > Not sure if it's a blocker so I didn't give +1 or -1 until some
Re: [DISCUSS] Registering Jackson Java 8 support modules by default for all Pulsar components, including client
Any change to Jackson configuration is a potential breaking change. Yes, it will need a PIP. I guess we can continue to discuss the change in this thread before I create an actual PIP which can be voted on. -Lari On 2023/01/09 11:53:02 丛搏 wrote: > Hi, Lari: > > Will it affect compatibility? If it is just an improved function, I > think it can also be added to the pulsar-common module. it adds the > dependency, so it needs PIP to discuss. > > Thanks, > Bo > > Lari Hotari 于2023年1月9日周一 19:06写道： > > > > Hi all, > > > > Jackson has a separate Java 8 support modules for adding support for proper > > serialization and deserialization of new classes that were added in Java 8 > > (Java 8 was released in 2014). > > > > These Jackson Java 8 support modules haven't been used in the Pulsar code > > base. This is a pity. This causes a lot of pain when using Java Time > > classes in Pulsar applications or Pulsar Functions. There are ways to get > > the classes working for applications, but the documentation is missing. It > > would make things easier if the Java 8 support modules for Jackson would be > > included and registered by default. > > > > I have created a PR to register Jackson Java 8 support modules by default > > for all Pulsar components. The PR is > > https://github.com/apache/pulsar/pull/19161 . > > > > Please review and provide feedback. Do we need a PIP for this change? > > > > -Lari >
Re: [DISCUSS] Introduce oshi library to sensory OS resources
Hi - I think oshi is very interesting. I really would like a discussion about how we will use it along with how we can test for any gaps it might have. Testing will need to consider many more types of platforms than laptops and cloud providers. Those of us who use AWS are aware that there may be gaps in features like nic speed. The software does have a test class. One step might be to start trying different VMs in various environments to understand any gaps. Best Regards, Dave Sent from my iPhone > On Jan 8, 2023, at 11:33 PM, Enrico Olivelli wrote: > > Please create a plan about how to leverage it. > > The PR only adds the dependency > > Enrico > >> Il giorno lun 9 gen 2023 alle ore 04:26 Cong Zhao >> ha scritto: >> >> +1 >> >> Thanks, >> Cong >> >>> On 2022/12/19 10:19:07 mattisonc...@gmail.com wrote: >>> >>> Hi, All >>> >>> I would like to introduce a new library oshi[1] to help Apache Pulsar >>> sensory OS resources. It can help us to get away from the complex file >>> manipulation and cross-platform compatibility issues in some operating >>> systems. >>> >>> code example: https://github.com/apache/pulsar/pull/18984 >>> >>> Please feel free to left comments. >>> >>> >>> Best, >>> Mattison >>> >>> >>> [1] https://github.com/oshi/oshi >>>
Re: [DISCUSS] PIP-239: Retry Mechanism per Message
Hello Enrico, For your concern about temporarily increasing of retry. It can be achieved using overriding msg property while calling reconsuming later with custom properties.. About msg immutable as per current design if consumer call reconsumeLater function it creates a new msg in the system adding few properties to it like how many times it get consumed. That also allow other custom properties to be added in newly generated msg.. so if msg needs temporary high retry or change in retry count on msg they can override it using custom properties… Thanks Nitin Goyal On Mon, 9 Jan 2023 at 1:11 PM, Enrico Olivelli wrote: > I don't think that this is a good idea. > > Because IIUC we want to add a property per message that sets the > maximum time of retries. > > Unfortunately in a real system sometimes you have to change the number > of retries temporarily, > for instance in case of major system failure you don't want to lose > all your messages. > > If we allow you to set a property on a message then you won't be able > to change it because the message is immutable. > > TTL (time to live) is a similar concept but it is related to the > concept of "physical time", if you have message that represents > a task to be executed within a given deadline then it makes sense to > state it in the message metadata. > > But the "number of retries" depends on how you deal with the retries: > - how much time do you wait ? > - how often do you have a temporary failure to retry ? > > It would make more sense to have a QOS (quality of service) attribute > on the message, like "important/"non-important"/"foo"/"bar" > and have a way for the brokers and the clients to handle that. I am > pretty sure that with interceptors you can already do something. > > > I am against hard coding the behaviour described in the PIP (and I > voted -1 in the VOTE thread) > > Enrico > > Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang ha > scritto: > > > > Hi, > > > > This looks good to me. > > +1 > > > > I was thinking if we could add a new API for `reconsumeLater` to let > > users set the max retry times easily. But I saw that there are too > > many reconsumeLater API and this will make the consumer more complex. > > > > Thanks, > > Zike Yang > > > > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu wrote: > > > > > > +1 > > > > > > Thanks, > > > Zixuan > > > > > > Nitin Goyal 于2023年1月5日周四 13:50写道： > > > > > > > Hi all, > > > > > > > > I made a PIP to discuss: > https://github.com/apache/pulsar/issues/19136 > > > > > > > > Thanks, > > > > Nitin Goyal > > > > > -- Regards Nitin Goyal
Re: [DISCUSS] Introduce oshi library to sensory OS resources
I will draft a PIP for it. Mattison On Jan 9, 2023, 22:30 +0800, Dave Fisher , wrote: > I think oshi is very interesting. I really would like a discussion about how > we will use it along with how we can test for any gaps it might have. Testing > will need to consider many more types of platforms than laptops and cloud > providers. Those of us who use AWS are aware that there may be gaps in > features like nic speed. The software does have a test class. One step might > be to start trying different VMs in various environments to understand any > gaps. Best Regards, Dave
Re: [DISCUSS] PIP-239: Retry Mechanism per Message
Thanks for submitting this PIP, Nitin Goyal. At the heart of this PIP is an assumption about the relationship between producers and consumers. The PIP assumes a producer knows how many times a consumer should attempt to consume a message before giving up and sending it to the DLQ. Does anyone have strong opinions on the boundaries between producers and consumers in relation to this PIP? This PIP expands the relationship between producer and consumer by letting the producer tell the consumer's pulsar client when to send a message to the DLQ, and as such, we should be very intentional about accepting this PIP. Because a user can easily implement this feature on their own and because it tightly couples producers and consumers, I think we should not move forward with this PIP. I am open to discussion, though. > for instance in case of major system failure you don't want to lose > all your messages. For what it's worth, the retry letter topic feature, which this PIP relies on, sends all messages to the DLQ, so this feature does not introduce conditions for message loss. As an aside, if we move forward with this feature, we need to make sure that the protocol documentation is updated and we should consider putting this field in the `MessageMetadata` protobuf object, not in a properties map. Thanks, Michael On Mon, Jan 9, 2023 at 9:47 AM Nitin Goyal wrote: > > Hello Enrico, > > For your concern about temporarily increasing of retry. It can be achieved > using overriding msg property while calling reconsuming later with custom > properties.. > > About msg immutable as per current design if consumer call reconsumeLater > function it creates a new msg in the system adding few properties to it > like how many times it get consumed. That also allow other custom > properties to be added in newly generated msg.. so if msg needs temporary > high retry or change in retry count on msg they can override it using > custom properties… > > Thanks > Nitin Goyal > > > On Mon, 9 Jan 2023 at 1:11 PM, Enrico Olivelli wrote: > > > I don't think that this is a good idea. > > > > Because IIUC we want to add a property per message that sets the > > maximum time of retries. > > > > Unfortunately in a real system sometimes you have to change the number > > of retries temporarily, > > for instance in case of major system failure you don't want to lose > > all your messages. > > > > If we allow you to set a property on a message then you won't be able > > to change it because the message is immutable. > > > > TTL (time to live) is a similar concept but it is related to the > > concept of "physical time", if you have message that represents > > a task to be executed within a given deadline then it makes sense to > > state it in the message metadata. > > > > But the "number of retries" depends on how you deal with the retries: > > - how much time do you wait ? > > - how often do you have a temporary failure to retry ? > > > > It would make more sense to have a QOS (quality of service) attribute > > on the message, like "important/"non-important"/"foo"/"bar" > > and have a way for the brokers and the clients to handle that. I am > > pretty sure that with interceptors you can already do something. > > > > > > I am against hard coding the behaviour described in the PIP (and I > > voted -1 in the VOTE thread) > > > > Enrico > > > > Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang ha > > scritto: > > > > > > Hi, > > > > > > This looks good to me. > > > +1 > > > > > > I was thinking if we could add a new API for `reconsumeLater` to let > > > users set the max retry times easily. But I saw that there are too > > > many reconsumeLater API and this will make the consumer more complex. > > > > > > Thanks, > > > Zike Yang > > > > > > > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu wrote: > > > > > > > > +1 > > > > > > > > Thanks, > > > > Zixuan > > > > > > > > Nitin Goyal 于2023年1月5日周四 13:50写道： > > > > > > > > > Hi all, > > > > > > > > > > I made a PIP to discuss: > > https://github.com/apache/pulsar/issues/19136 > > > > > > > > > > Thanks, > > > > > Nitin Goyal > > > > > > > > -- > Regards > Nitin Goyal
RE: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar
Congratulations, Yunze! On 2022/12/29 12:42:36 Haiting Jiang wrote: > Hi all, > > The Apache Pulsar Project Management Committee (PMC) has invited Yunze Xu > (https://github.com/BewareMyPower) as a member of the PMC and we are > pleased to announce that he has accepted. > > He is very active in the community in the past few years and made a lot of > great contributions. > > Welcome Yunze to the Apache Pulsar PMC. > > Best Regards, > Haiting Jiang on behalf of the Pulsar PMC >
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
Hi Hang The upgrade does not affect it. It will automatically detect if there is a zookeeper directory and will continue to use zookeeper. If the new user has requirements for standalone performance, just enable the zookeeper option. Also, we will notice this on the release note. Regards Jiwei Guo (Tboy) On Mon, Jan 9, 2023 at 8:40 PM Hang Chen wrote: > +0 (binding) > > - Checked signature > - Checked license for pulsar and pulsar-shell > - Build from the source code (MacOS Intel, JDK 17, Maven 3.8.6) > - Run standalone Pulsar based on Zookeeper and RocksDB metastore > - Run Pulsar-perf to produce and consume messages > - Use pulsar-shell to get topics stats > - Run pulsar-io-lakehouse connector based on Pulsar 2.11.0 > > When I ran Pulsar standalone based on RocksDB metastore, I found the > following issues. > - Create a 1 partitions topic failed with timeout using pulsar-admin > - Use Pulsar perf to produce topics with 5000 partitions, produce > failed with generate ledgerId timeout on the Broker side > - Create a 1000 partitions topic succeed using pulsar-admin, but > produce messages to the topic failed with timeout > > However, after I started Pulsar standalone based on Zookeeper > metastore, the above issues disappeared, and the topic could produce > and consume messages. > > Currently, the Pulsar standalone mode starts with RocksDB metastore by > default since 2.11.0, but it can not run well in some cases. I'm > unsure whether it will impact the current user who wants to start the > pulsar service in standalone mode. Even though Pulsar standalone is > not recommended to run in the production environment, we need to > ensure it can perform the best performance on one local node. > > If the standalone issue is the expected behavior, I will give my +1. > > Thanks, > Hang > > Enrico Olivelli 于2023年1月9日周一 15:55写道： > > > > +1 (binding) > > > > - checked signatures/digests/rat > > - built from source > > - run smoke tests on the built binaries > > > > > > Thanks > > Enrico > > > > Il giorno lun 9 gen 2023 alle ore 06:18 guo jiwei > > ha scritto: > > > > > > +1 (binding) > > > > > > - Checked the signature > > > - Build from source > > > - Checked license(server and shell) > > > - Start standalone with zookeeper > > > - Checked function > > > - Checked Cassandra connector > > > - Checked stateful function > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Wed, Jan 4, 2023 at 8:25 PM Nicolò Boschi > wrote: > > > > > > > About Pulsar Shell, > > > > 1) zip file is distributed as a commodity for Windows users since the > > > > .tar.gz archive is more tricky to extract. If this is a problem, we > can > > > > discuss it in another thread > > > > 2) The bug on Windows is not a blocker for the release since it's > not a > > > > regression. We could fix it in 2.11.1 > > > > > > > > Here's my vote: > > > > > > > > +1 non binding > > > > > > > > - Signature and checksum > > > > - LICENSE on server and shell > > > > > > > > - Rat check passes > > > > > > > > - Compile from source with JDK 17 on MacOs Intel > > > > > > > > - Run Pulsar standalone and produce-consume from CLI > > > > > > > > - Tested K8S installation with Datastax Pulsar helm chart and > verified TLS, > > > > offloads, transactions and ElasticSearch sink > > > > > > > > - Pulsar Shell commands > > > > > > > > > > > > Thanks, > > > > Nicolò Boschi > > > > > > > > > > > > Il giorno mer 4 gen 2023 alle ore 12:33 丛搏 ha > scritto: > > > > > > > > > +1 (non-binding) > > > > > > > > > > system: mac os 12.6, Apple M1 > > > > > maven: 3.8.5 > > > > > java: OpenJDK 17.0.3 > > > > > > > > > > - Checked the signature > > > > > - Checked LICENSE > > > > > - Start standalone with zookeeper stream storage > > > > > - Publish and consume messages > > > > > - Verified Function and State Function > > > > > - Verified Cassandra connector > > > > > - Build from the source package > > > > > - Run a simple transaction performance check > > > > > > > > > > Thanks, > > > > > Bo > > > > > > > > > > Yunze Xu 于2023年1月4日周三 16:41写道： > > > > > > > > > > > > Okay, I will give my +1 (binding). > > > > > > > > > > > > Thanks, > > > > > > Yunze > > > > > > > > > > > > On Wed, Jan 4, 2023 at 4:00 PM PengHui Li > wrote: > > > > > > > > > > > > > > I think it's ok. > > > > > > > 2.11.0 is the first release version of pulsar-shell > > > > > > > So it's not a regression that was introduced in 2.11.0 and not > > > > > > > critical security issues or license issues which will block > users > > > > move > > > > > > > to the new version. > > > > > > > > > > > > > > Thanks, > > > > > > > Penghui > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 4, 2023 at 3:36 PM Yunze Xu > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > +0 (binding) > > > > > > > > > > > > > > > > - Checked the signature > > > > > > > > - Build from source (Java 17, Ubuntu 20.04 WSL2) > > > > > > > > - Start standalone with KoP branch-2.11 (ab9d761f) > > > > >
[DISCUSS] Should we bump up Node.js Client for C++ Client's upgrade
Hi all, Currently Node.js Client is tied to specified version of C++ Client due to the following PR: https://github.com/apache/pulsar-client-node/pull/235 Should we bump up the version of Node.js Client even when there is no update except for C++ Client’s upgrade(at least for critical bug fixies)? Best Regards, Yuto Yuto Furuta Yahoo Japan Corp. E-mail: yfur...@yahoo-corp.jp
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
Thank you all, Close the vote with 5 bindings(PengHui, Enrico, Yunze, Jiwei, Hang), and 1non-bindings(Bo). I will continue the release process. Regards Jiwei Guo (Tboy) On Tue, Jan 10, 2023 at 10:00 AM guo jiwei wrote: > Hi Hang >The upgrade does not affect it. It will automatically detect if there > is a zookeeper directory and will continue to use zookeeper. > If the new user has requirements for standalone performance, just enable > the zookeeper option. >Also, we will notice this on the release note. > > Regards > Jiwei Guo (Tboy) > > > On Mon, Jan 9, 2023 at 8:40 PM Hang Chen wrote: > >> +0 (binding) >> >> - Checked signature >> - Checked license for pulsar and pulsar-shell >> - Build from the source code (MacOS Intel, JDK 17, Maven 3.8.6) >> - Run standalone Pulsar based on Zookeeper and RocksDB metastore >> - Run Pulsar-perf to produce and consume messages >> - Use pulsar-shell to get topics stats >> - Run pulsar-io-lakehouse connector based on Pulsar 2.11.0 >> >> When I ran Pulsar standalone based on RocksDB metastore, I found the >> following issues. >> - Create a 1 partitions topic failed with timeout using pulsar-admin >> - Use Pulsar perf to produce topics with 5000 partitions, produce >> failed with generate ledgerId timeout on the Broker side >> - Create a 1000 partitions topic succeed using pulsar-admin, but >> produce messages to the topic failed with timeout >> >> However, after I started Pulsar standalone based on Zookeeper >> metastore, the above issues disappeared, and the topic could produce >> and consume messages. >> >> Currently, the Pulsar standalone mode starts with RocksDB metastore by >> default since 2.11.0, but it can not run well in some cases. I'm >> unsure whether it will impact the current user who wants to start the >> pulsar service in standalone mode. Even though Pulsar standalone is >> not recommended to run in the production environment, we need to >> ensure it can perform the best performance on one local node. >> >> If the standalone issue is the expected behavior, I will give my +1. >> >> Thanks, >> Hang >> >> Enrico Olivelli 于2023年1月9日周一 15:55写道： >> > >> > +1 (binding) >> > >> > - checked signatures/digests/rat >> > - built from source >> > - run smoke tests on the built binaries >> > >> > >> > Thanks >> > Enrico >> > >> > Il giorno lun 9 gen 2023 alle ore 06:18 guo jiwei >> > ha scritto: >> > > >> > > +1 (binding) >> > > >> > > - Checked the signature >> > > - Build from source >> > > - Checked license(server and shell) >> > > - Start standalone with zookeeper >> > > - Checked function >> > > - Checked Cassandra connector >> > > - Checked stateful function >> > > >> > > >> > > Regards >> > > Jiwei Guo (Tboy) >> > > >> > > >> > > On Wed, Jan 4, 2023 at 8:25 PM Nicolò Boschi >> wrote: >> > > >> > > > About Pulsar Shell, >> > > > 1) zip file is distributed as a commodity for Windows users since >> the >> > > > .tar.gz archive is more tricky to extract. If this is a problem, we >> can >> > > > discuss it in another thread >> > > > 2) The bug on Windows is not a blocker for the release since it's >> not a >> > > > regression. We could fix it in 2.11.1 >> > > > >> > > > Here's my vote: >> > > > >> > > > +1 non binding >> > > > >> > > > - Signature and checksum >> > > > - LICENSE on server and shell >> > > > >> > > > - Rat check passes >> > > > >> > > > - Compile from source with JDK 17 on MacOs Intel >> > > > >> > > > - Run Pulsar standalone and produce-consume from CLI >> > > > >> > > > - Tested K8S installation with Datastax Pulsar helm chart and >> verified TLS, >> > > > offloads, transactions and ElasticSearch sink >> > > > >> > > > - Pulsar Shell commands >> > > > >> > > > >> > > > Thanks, >> > > > Nicolò Boschi >> > > > >> > > > >> > > > Il giorno mer 4 gen 2023 alle ore 12:33 丛搏 ha >> scritto: >> > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > system: mac os 12.6, Apple M1 >> > > > > maven: 3.8.5 >> > > > > java: OpenJDK 17.0.3 >> > > > > >> > > > > - Checked the signature >> > > > > - Checked LICENSE >> > > > > - Start standalone with zookeeper stream storage >> > > > > - Publish and consume messages >> > > > > - Verified Function and State Function >> > > > > - Verified Cassandra connector >> > > > > - Build from the source package >> > > > > - Run a simple transaction performance check >> > > > > >> > > > > Thanks, >> > > > > Bo >> > > > > >> > > > > Yunze Xu 于2023年1月4日周三 16:41写道： >> > > > > > >> > > > > > Okay, I will give my +1 (binding). >> > > > > > >> > > > > > Thanks, >> > > > > > Yunze >> > > > > > >> > > > > > On Wed, Jan 4, 2023 at 4:00 PM PengHui Li >> wrote: >> > > > > > > >> > > > > > > I think it's ok. >> > > > > > > 2.11.0 is the first release version of pulsar-shell >> > > > > > > So it's not a regression that was introduced in 2.11.0 and not >> > > > > > > critical security issues or license issues which will block >> users >> > > > move >> > > > > > > to the new version. >> > > > > > > >> > > > > > > Thanks,
Re: [DISCUSS] Should we bump up Node.js Client for C++ Client's upgrade
Hi, Yuto Furuta, Currently, the latest Nodejs client 1.8.0 is already tied to the latest C++ client 3.1.0. If there is a new C++ client version release, I think it's also better to release a new Nodejs client version that is tied to the latest C++ client version even though there are no new commits in the Nodejs client. In this case, we can make sure the latest Nodejs client always gets the latest updates or bug fixes of the C++ client. BR, Zike Yang On Tue, Jan 10, 2023 at 1:57 PM Yuto Furuta wrote: > > Hi all, > > Currently Node.js Client is tied to specified version of C++ Client due to > the following PR: > https://github.com/apache/pulsar-client-node/pull/235 > > Should we bump up the version of Node.js Client even when there is no update > except for C++ Client’s upgrade(at least for critical bug fixies)? > > > Best Regards, > Yuto > > Yuto Furuta > Yahoo Japan Corp. > E-mail: yfur...@yahoo-corp.jp
Re: [DISCUSS] Should we bump up Node.js Client for C++ Client's upgrade
My answer is yes. Some bug fixes might be included in a newer C++ client. But I think we don't need to start a release in a hurry once a newer C++ client is released. Thanks, Yunze On Tue, Jan 10, 2023 at 3:03 PM Zike Yang wrote: > > Hi, Yuto Furuta, > > Currently, the latest Nodejs client 1.8.0 is already tied to the > latest C++ client 3.1.0. If there is a new C++ client version release, > I think it's also better to release a new Nodejs client version that > is tied to the latest C++ client version even though there are no new > commits in the Nodejs client. In this case, we can make sure the > latest Nodejs client always gets the latest updates or bug fixes of > the C++ client. > > BR, > Zike Yang > > On Tue, Jan 10, 2023 at 1:57 PM Yuto Furuta wrote: > > > > Hi all, > > > > Currently Node.js Client is tied to specified version of C++ Client due to > > the following PR: > > https://github.com/apache/pulsar-client-node/pull/235 > > > > Should we bump up the version of Node.js Client even when there is no > > update except for C++ Client’s upgrade(at least for critical bug fixies)? > > > > > > Best Regards, > > Yuto > > > > Yuto Furuta > > Yahoo Japan Corp. > > E-mail: yfur...@yahoo-corp.jp
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
Il giorno mar 10 gen 2023 alle ore 07:18 guo jiwei ha scritto: > > Thank you all, > > Close the vote with 5 bindings(PengHui, Enrico, Yunze, Jiwei, Hang), and > 1non-bindings(Bo). actually there are 2 non-binding votes: + Nicolò Boschi Enrico > > I will continue the release process. > > Regards > Jiwei Guo (Tboy) > > > On Tue, Jan 10, 2023 at 10:00 AM guo jiwei wrote: > > > Hi Hang > >The upgrade does not affect it. It will automatically detect if there > > is a zookeeper directory and will continue to use zookeeper. > > If the new user has requirements for standalone performance, just enable > > the zookeeper option. > >Also, we will notice this on the release note. > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Mon, Jan 9, 2023 at 8:40 PM Hang Chen wrote: > > > >> +0 (binding) > >> > >> - Checked signature > >> - Checked license for pulsar and pulsar-shell > >> - Build from the source code (MacOS Intel, JDK 17, Maven 3.8.6) > >> - Run standalone Pulsar based on Zookeeper and RocksDB metastore > >> - Run Pulsar-perf to produce and consume messages > >> - Use pulsar-shell to get topics stats > >> - Run pulsar-io-lakehouse connector based on Pulsar 2.11.0 > >> > >> When I ran Pulsar standalone based on RocksDB metastore, I found the > >> following issues. > >> - Create a 1 partitions topic failed with timeout using pulsar-admin > >> - Use Pulsar perf to produce topics with 5000 partitions, produce > >> failed with generate ledgerId timeout on the Broker side > >> - Create a 1000 partitions topic succeed using pulsar-admin, but > >> produce messages to the topic failed with timeout > >> > >> However, after I started Pulsar standalone based on Zookeeper > >> metastore, the above issues disappeared, and the topic could produce > >> and consume messages. > >> > >> Currently, the Pulsar standalone mode starts with RocksDB metastore by > >> default since 2.11.0, but it can not run well in some cases. I'm > >> unsure whether it will impact the current user who wants to start the > >> pulsar service in standalone mode. Even though Pulsar standalone is > >> not recommended to run in the production environment, we need to > >> ensure it can perform the best performance on one local node. > >> > >> If the standalone issue is the expected behavior, I will give my +1. > >> > >> Thanks, > >> Hang > >> > >> Enrico Olivelli 于2023年1月9日周一 15:55写道： > >> > > >> > +1 (binding) > >> > > >> > - checked signatures/digests/rat > >> > - built from source > >> > - run smoke tests on the built binaries > >> > > >> > > >> > Thanks > >> > Enrico > >> > > >> > Il giorno lun 9 gen 2023 alle ore 06:18 guo jiwei > >> > ha scritto: > >> > > > >> > > +1 (binding) > >> > > > >> > > - Checked the signature > >> > > - Build from source > >> > > - Checked license(server and shell) > >> > > - Start standalone with zookeeper > >> > > - Checked function > >> > > - Checked Cassandra connector > >> > > - Checked stateful function > >> > > > >> > > > >> > > Regards > >> > > Jiwei Guo (Tboy) > >> > > > >> > > > >> > > On Wed, Jan 4, 2023 at 8:25 PM Nicolò Boschi > >> wrote: > >> > > > >> > > > About Pulsar Shell, > >> > > > 1) zip file is distributed as a commodity for Windows users since > >> the > >> > > > .tar.gz archive is more tricky to extract. If this is a problem, we > >> can > >> > > > discuss it in another thread > >> > > > 2) The bug on Windows is not a blocker for the release since it's > >> not a > >> > > > regression. We could fix it in 2.11.1 > >> > > > > >> > > > Here's my vote: > >> > > > > >> > > > +1 non binding > >> > > > > >> > > > - Signature and checksum > >> > > > - LICENSE on server and shell > >> > > > > >> > > > - Rat check passes > >> > > > > >> > > > - Compile from source with JDK 17 on MacOs Intel > >> > > > > >> > > > - Run Pulsar standalone and produce-consume from CLI > >> > > > > >> > > > - Tested K8S installation with Datastax Pulsar helm chart and > >> verified TLS, > >> > > > offloads, transactions and ElasticSearch sink > >> > > > > >> > > > - Pulsar Shell commands > >> > > > > >> > > > > >> > > > Thanks, > >> > > > Nicolò Boschi > >> > > > > >> > > > > >> > > > Il giorno mer 4 gen 2023 alle ore 12:33 丛搏 ha > >> scritto: > >> > > > > >> > > > > +1 (non-binding) > >> > > > > > >> > > > > system: mac os 12.6, Apple M1 > >> > > > > maven: 3.8.5 > >> > > > > java: OpenJDK 17.0.3 > >> > > > > > >> > > > > - Checked the signature > >> > > > > - Checked LICENSE > >> > > > > - Start standalone with zookeeper stream storage > >> > > > > - Publish and consume messages > >> > > > > - Verified Function and State Function > >> > > > > - Verified Cassandra connector > >> > > > > - Build from the source package > >> > > > > - Run a simple transaction performance check > >> > > > > > >> > > > > Thanks, > >> > > > > Bo > >> > > > > > >> > > > > Yunze Xu 于2023年1月4日周三 16:41写道： > >> > > > > > > >> > > > > > Okay, I will give my +1 (binding). > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Yunze > >> > > >
