Re: [DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-13 Thread 丛搏
> > > the user only creates one producer to send all Kafka topic data, if > using Pulsar schema, the user needs to create all schema producers in > a map > > It doesn't make sense to me. If the source topic has messages of > multiple schemas, why did you try to sink them into the same topic > with

Re: [DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-13 Thread 丛搏
Yunze Xu 于2022年12月14日周三 12:40写道: > > > the user only creates one producer to send all Kafka topic data, if > using Pulsar schema, the user needs to create all schema producers in > a map > > It doesn't make sense to me. If the source topic has messages of > multiple schemas, why did you try to sin

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread 丛搏
> As more people joined the development of Pulsar and more PIPs are > opened, I found the configurations became very large. At the moment > for commit 9917aac, there are 426 configuration items in broker.conf, > which is too many. I agree with you, the configuration of pulsar is too complicated, we

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread 丛搏
> As more people joined the development of Pulsar and more PIPs are > opened, I found the configurations became very large. At the moment > for commit 9917aac, there are 426 configuration items in broker.conf, > which is too many. I agree with you, the configuration of pulsar is too complicated, we

Re: [DISCUSS] Move the source of swagger files to the site repo

2022-12-13 Thread Michael Marshall
I sent a reply here [0] before realizing there was a thread. My main issue with stopping the generation of these files is that they can be used to generate admin clients. Although, I am not sure how "good" these generated clients are, though. I know of at least one third party project that has sta

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread Michael Marshall
I agree that it is confusing. I don't have any immediate ideas for fixing the problem, but here are my observations: It can also be confusing to new users that the Apache Pulsar docker image has a different bookkeeper.conf than the Apache Bookkeeper project. I found configuring the function worke

Re: [VOTE] Pulsar Client Python Release 3.0.0 Candidate 1

2022-12-13 Thread Yunze Xu
Hi Matteo, Thanks for your reminder, I will cancel this vote and start a new candidate. Thanks, Yunze On Wed, Dec 14, 2022 at 12:47 PM Matteo Merli wrote: > > Hi Yunze, > > I think we need to fix the Py client version "3.0.0a1" --> "3.0.0" in > https://github.com/apache/pulsar-client-python/blo

Re: [VOTE] Pulsar Client Python Release 3.0.0 Candidate 1

2022-12-13 Thread Matteo Merli
Hi Yunze, I think we need to fix the Py client version "3.0.0a1" --> "3.0.0" in https://github.com/apache/pulsar-client-python/blob/main/version.txt -- Matteo Merli On Tue, Dec 13, 2022 at 8:10 PM Yunze Xu wrote: > > This is the first release candidate for Apache Pulsar Client Python, > versi

Re: [DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-13 Thread Yunze Xu
> the user only creates one producer to send all Kafka topic data, if using Pulsar schema, the user needs to create all schema producers in a map It doesn't make sense to me. If the source topic has messages of multiple schemas, why did you try to sink them into the same topic with a schema? The k

[VOTE] Pulsar Client Python Release 3.0.0 Candidate 1

2022-12-13 Thread Yunze Xu
This is the first release candidate for Apache Pulsar Client Python, version 3.0.0. It fixes the following issues: https://github.com/apache/pulsar-client-python/milestone/1?closed=1 *** Please download, test and vote on this release. This vote will stay open for at least 72 hours *** Python whe

Re: [DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-13 Thread 丛搏
Hi, Yunze: Yunze Xu 于2022年12月14日周三 02:26写道: > First, how do you guarantee the schema can be used to encode the raw > bytes whose format is unknown? I think this is what the user needs to ensure that the user knows all the schema from the Kafka topic and the date(bytes[]) that the user can send w

Re: Too many emails - Is there a better way to control or manage emails from GitBox

2022-12-13 Thread tison
> can we set it not to sync It seems the current .asf.yaml and it's underneath tech doesn't support such fine-grained control. I ever talk to the author about configuring in category level but it's still unsupported yet also. The source code, IIRC, is hosted at https://github.com/apache/infrastr

[DISCUSSIONS] Should we use AUTO_PRODUCE schema?

2022-12-13 Thread Yunze Xu
Hi all, Pulsar supports AUTO_PRODUCE schema, but this feature was introduced at an early time [1] when there was no PIP. I have read the documents [2] and found the example scenario. > Suppose that: > - You have a producer processing messages from a Kafka topic K. > - You have a Pulsar topic P, a

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread Yunze Xu
> they have to prepend PULSAR_PREFIX_ to the other options. Good point. It should be taken carefully. Thanks, Yunze On Tue, Dec 13, 2022 at 10:46 PM Enrico Olivelli wrote: > > I agree that we should make it simpler. > There are too many knobs that people shouldn't try to turn if they > don't kn

[DISCUSS] Move the source of swagger files to the site repo

2022-12-13 Thread tison
Hi, This can be regarded as a follow-up to the API docs improving movement[1]. I noticed that we generate swagger-based REST API docs every release: hold the source in the main repo, and then sync to the site repo. However, this isn't necessary. We can, as we do for the other API docs, directly g

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread Enrico Olivelli
I agree that we should make it simpler. There are too many knobs that people shouldn't try to turn if they don't know what they are doing. Il giorno mar 13 dic 2022 alle ore 14:19 Kai Wang ha scritto: > > +1, we can provide a minimal configuration file to users. It only contains > the required c

Re: [DISCUSS] Release Nodejs Client 1.8.0

2022-12-13 Thread Baodi Shi
+1 Thanks, Baodi Shi > 2022年12月13日 21:09,Yunze Xu 写道: > > +1 > > Thanks, > Yunze > > On Tue, Dec 13, 2022 at 9:05 PM Matteo Merli wrote: >> >> +1 >> >> On Tue, Dec 13, 2022 at 2:03 AM Zike Yang wrote: >> >>> Hi everyone, >>> >>> I would like to propose releasing the Pulsar Client Nodejs

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread Kai Wang
+1, we can provide a minimal configuration file to users. It only contains the required config and a few commonly used configs. The full configuration file can be named `broker.full.conf`, and it is used to  provide a reference for users. Thanks, Kai On Dec 13, 2022 at 9:03 PM +0800, Yunze Xu ,

Re: [DISCUSS] Release Pulsar Python Client 3.0.0

2022-12-13 Thread Yunze Xu
FYI, I will start the Python release this week since the C++ 3.1.0 release is out. Thanks, Yunze On Thu, Nov 24, 2022 at 3:57 PM Haiting Jiang wrote: > > +1 > > Haiting > > On Thu, Nov 24, 2022 at 3:47 PM Zike Yang wrote: > > > > +1 > > > > Thanks > > Zike Yang > > > > On Wed, Nov 23, 2022 at 1

Re: [DISCUSS] Release Nodejs Client 1.8.0

2022-12-13 Thread Yunze Xu
+1 Thanks, Yunze On Tue, Dec 13, 2022 at 9:05 PM Matteo Merli wrote: > > +1 > > On Tue, Dec 13, 2022 at 2:03 AM Zike Yang wrote: > > > Hi everyone, > > > > I would like to propose releasing the Pulsar Client Nodejs 1.8.0 > > > > There are several new features and bug fixes in the master branch[

Re: [DISCUSS] Release Nodejs Client 1.8.0

2022-12-13 Thread Matteo Merli
+1 On Tue, Dec 13, 2022 at 2:03 AM Zike Yang wrote: > Hi everyone, > > I would like to propose releasing the Pulsar Client Nodejs 1.8.0 > > There are several new features and bug fixes in the master branch[0]. > And it’s time to release a new version. > The new Nodejs client will depend on the P

Re: [DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread Yunze Xu
For example, when running a standalone (without TLS enabled), only the following configs are required: ```properties brokerServicePort=6650 webServicePort=8080 allowLoopback=true clusterName=standalone managedLedgerDefaultEnsembleSize=1 managedLedgerDefaultWriteQuorum=1 managedLedgerDefaultAckQuor

[DISCUSSION] Any idea about simplify the configuration file?

2022-12-13 Thread Yunze Xu
Hi all, As more people joined the development of Pulsar and more PIPs are opened, I found the configurations became very large. At the moment for commit 9917aac, there are 426 configuration items in broker.conf, which is too many. ```bash $ grep "^[^#]" conf/broker.conf | wc -l 426 ``` For begin

Re: [VOTE][PIP-225] Pulsar Functions fetch parameters from local config file.

2022-12-13 Thread Enrico Olivelli
+1 (binding) We must have a flag to turn this on/off and disable it by default (this will also be needed for the transition to the new method) Enrico Il giorno ven 9 dic 2022 alle ore 05:39 Yufei Zhang ha scritto: > > Hi Enrico, > > Thank you for your feedback! > > I want to add more context to

Re: [Discussion] Breaking change of Pulsar standalone's start

2022-12-13 Thread Yunze Xu
FYI, I pushed a PR: https://github.com/apache/pulsar/pull/18909 Thanks, Yunze On Tue, Dec 13, 2022 at 7:55 PM Yunze Xu wrote: > > In addition to the setup process, Go client assumes the cluster was > not created [1] while C++ and Python clients don't care if it's > created because it checks if t

Re: [VOTE] Pulsar Release 2.11.0 Candidate-2

2022-12-13 Thread Yunze Xu
I found another breaking change. Please take a look at this PR: https://github.com/apache/pulsar/pull/18909 I've also created a discussion at dev mail list, which can be found in the PR description. Thanks, Yunze On Mon, Dec 12, 2022 at 9:22 PM guo jiwei wrote: > > Hi >All the issues have b

Re: [Discussion] Breaking change of Pulsar standalone's start

2022-12-13 Thread Yunze Xu
In addition to the setup process, Go client assumes the cluster was not created [1] while C++ and Python clients don't care if it's created because it checks if the cluster exists before. [1] https://github.com/apache/pulsar-client-go/blob/1fe9b624ca673e3a647780b7c6c4b2567089285a/scripts/pulsar-t

[VOTE] Pulsar Release 2.9.4 Candidate 3

2022-12-13 Thread 丛搏
This is the third release candidate for Apache Pulsar, version 2.9.4. This release contains 319 commits by 69 contributors. https://github.com/apache/pulsar/compare/v2.9.3...v2.9.4-candidate-3 *** Please download, test and vote on this release. This vote will stay open for at least 72 hours ***

[Discussion] Breaking change of Pulsar standalone's start

2022-12-13 Thread Yunze Xu
Hi all, I found a breaking change introduced from #15186 [1] that changed the behavior of Pulsar standalone. Before #15186, it used Pulsar's built-in admin client to create tenant (public) and namespace (public/default). After #15186, it created the metadata nodes directly to the metadata store.

[DISCUSS] Release Nodejs Client 1.8.0

2022-12-13 Thread Zike Yang
Hi everyone, I would like to propose releasing the Pulsar Client Nodejs 1.8.0 There are several new features and bug fixes in the master branch[0]. And it’s time to release a new version. The new Nodejs client will depend on the Pulsar C++ Client 3.1.0. Please remind me if you have any important

[VOTE] PIP-228: Refactor the information architecture of Pulsar client docs

2022-12-13 Thread Jun Ma
Hi all, I'm going to start the vote for PIP-228 [Refactor the information architecture of Pulsar client docs](https://github.com/apache/pulsar/issues/18822). And this is the original thread for discussion: https://lists.apache.org/thread/bv6lwnt708dxst173knyzv2bfy4d1ox4. The vote will be open