Re: [DISCUSS] ClientVersion conflict across multiple language client

2023-02-21 Thread Baodi Shi 
 +1, Good idea.

Thanks,
Baodi Shi


在 2023年2月21日 15:24:45 上，avinash kala  写道：

> +1, sounds good.
>
> On Tue, Feb 21, 2023, 12:39 PM Zixuan Liu  wrote:
>
> +1, good idea!
>
>
> Thanks,
>
> Zixuan
>
>
> Zike Yang  于2023年2月21日周二 11:33写道：
>
>
> > Hi, all
>
> >
>
> > Currently, the Pulsar broker uses the field `client_version` to get
>
> > the version of the client. The client will send the client_version to
>
> > the broker through `CommandConnect` [0] during the connect or
>
> > `CommandAuthResponse ` [1] during the auth challenge. We could get the
>
> > client_version from the admin using `pulsar-admin topics stats xxx`
>
> > [2].
>
> >
>
> > But the `client_version ` only shows the version information. And
>
> > clients in different languages use their own separate version numbers.
>
> > This can lead to conflict. For example, the java client 2.10.2 uses
>
> > the same value `2.10.2` as the C++ client 2.10.2. And C++ client 3.0.0
>
> > may conflict with the future java client 3.0.0. The information of
>
> > `client_version` is incomplete. We could not determine what
>
> > language(or specific library) the client uses. This raises
>
> > inconvenience for debugging.
>
> >
>
> > I would like to propose adding the client language information to the
>
> > `client_version`. Like:
>
> > * `java-2.11.1` for Java client 2.11
>
> > * `cpp-3.1.2` for C++ client 3.1.2
>
> > * `go-0.9.0` for go client 0.9.0
>
> > We can easily do that because the `client_version` is a string type.
>
> >
>
> > In addition, the Nodejs client and Python client all use the
>
> > client_version of C++ client they depend on. So it will use `3.1.2` as
>
> > the client_verion in Nodejs client 1.8.0. This is incorrect, and I
>
> > think we need to fix them. We don't need to print the C++ client
>
> > version because we can get that information if we know the Nodejs or
>
> > Python client version.
>
> >
>
> > What do you think? Please share your thoughts if you have any ideas.
>
> >
>
> > [0]
>
> >
>
>
> https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L269
>
> > [1]
>
> >
>
>
> https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L309
>
> > [2] https://pulsar.apache.org/docs/next/admin-api-topics/#get-stats
>
> >
>
> > Thanks,
>
> > Zike Yang
>
> >
>
>
>

Re: [VOTE][PIP-242] Topic name restriction

2023-02-21 Thread 丛搏 
+1 binding

avinash kala  于2023年2月21日周二 15:21写道：
>
> +1
>
> On Tue, Feb 21, 2023, 12:44 PM Haiting Jiang  wrote:
>
> > +1 binding
> >
> > Haiting
> >
> > On Tue, Feb 21, 2023 at 3:07 PM guo jiwei  wrote:
> > >
> > > +1 (binding)
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >
> > > On Mon, Feb 20, 2023 at 6:06 PM Zike Yang  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Zike Yang
> > > >
> > > >
> > > > On Mon, Feb 20, 2023 at 1:53 PM PengHui Li  wrote:
> > > > >
> > > > > +1(binding)
> > > > >
> > > > > Thanks,
> > > > > Penghui
> > > > >
> > > > > On Mon, Feb 20, 2023 at 11:54 AM Cong Zhao 
> > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Cong
> > > > > >
> > > > > > On 2023/02/18 08:58:26 mattisonc...@gmail.com wrote:
> > > > > > > Hi, All
> > > > > > >
> > > > > > > After a fascinating discussion, I would start the vote of
> > PIP-242.
> > > > > > >
> > > > > > > We have chosen to drop out the `system topic` related
> > improvement to
> > > > > > another PIP. Therefore, the current version is simple enough and
> > it has a
> > > > > > clear boundary.
> > > > > > >
> > > > > > > Please leave +1/-1 in this thread to join the vote. and feel
> > free to
> > > > > > leave any concerns.
> > > > > > >
> > > > > > > Thanks to you guys.
> > > > > > >
> > > > > > > Best,
> > > > > > > Mattison
> > > > > > >
> > > > > > > References:
> > > > > > >
> > > > > > > • PIP https://github.com/apache/pulsar/issues/19239
> > > > > > > • Discussion
> > > > > > https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> >

Re: [PROPOSAL] Roadmap for 3.0 release

2023-02-21 Thread Enrico Olivelli 
+1

Perfect

Enrico

Il giorno mar 21 feb 2023 alle ore 08:11 Haiting Jiang
 ha scritto:
>
> +1 Looks great!
>
>
> Haiting
>
> On Tue, Feb 21, 2023 at 12:11 PM Hang Chen  wrote:
> >
> > +1
> >
> > Thanks,
> > Hang
> >
> > Christophe Bornet  于2023年2月20日周一 21:46写道：
> > >
> > > +1
> > >
> > > Also, I'd like to volunteer as a release manager for this release.
> > >
> > > Christophe
> > >
> > >
> > >
> > >
> > > Le ven. 17 févr. 2023 à 23:44, Matteo Merli  a écrit :
> > > >
> > > > Since the LTS release model has been formally approved, I'm proposing
> > > > the following schedule for the release:
> > > >
> > > >  * Tue - 2023-05-11
> > > >- RC-1
> > > >- Code Freeze -- Only critical fixes will be merged in the 3.0
> > > > release branch. Contributors should plan to have all the changes merged 
> > > > in
> > > > before this date. Exceptions should be extremely rare and strongly
> > > > motivated.
> > > >
> > > >  * Tue - 2023-05-18 - RC-2
> > > >  * Tue - 2023-05-25 - RC-3
> > > >  * Tue - 2023-05-02 - Announce 3.0 Release
> > > >
> > > > These dates will be published on the website to present users with a
> > > > "roadmap" and we should commit to and respect these dates.
> > > >
> > > > I also wanted to propose trying out a model where we have 3 release
> > > > managers for all major releases.
> > > >
> > > > The reasoning behind this is for this small group of people to 
> > > > collaborate
> > > > and divide the tasks for the release: merging patches from the "master"
> > > > branch, preparing RC, and testing.
> > > >
> > > > Since everyone also has other work duties and unexpected tasks that can 
> > > > pop
> > > > up at any time, it will help to have redundancy in the 
> > > > release-management
> > > > "team", so that we can release on the exact dates.
> > > >
> > > > Thanks,
> > > > Matteo
> > > >
> > > > --
> > > > Matteo Merli
> > > >

Re: [DISCUSS] ClientVersion conflict across multiple language client

2023-02-21 Thread Enrico Olivelli 
+1

Enrico

Il giorno mar 21 feb 2023 alle ore 10:23 Baodi Shi 
ha scritto:
>
>  +1, Good idea.
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年2月21日 15:24:45 上，avinash kala  写道：
>
> > +1, sounds good.
> >
> > On Tue, Feb 21, 2023, 12:39 PM Zixuan Liu  wrote:
> >
> > +1, good idea!
> >
> >
> > Thanks,
> >
> > Zixuan
> >
> >
> > Zike Yang  于2023年2月21日周二 11:33写道：
> >
> >
> > > Hi, all
> >
> > >
> >
> > > Currently, the Pulsar broker uses the field `client_version` to get
> >
> > > the version of the client. The client will send the client_version to
> >
> > > the broker through `CommandConnect` [0] during the connect or
> >
> > > `CommandAuthResponse ` [1] during the auth challenge. We could get the
> >
> > > client_version from the admin using `pulsar-admin topics stats xxx`
> >
> > > [2].
> >
> > >
> >
> > > But the `client_version ` only shows the version information. And
> >
> > > clients in different languages use their own separate version numbers.
> >
> > > This can lead to conflict. For example, the java client 2.10.2 uses
> >
> > > the same value `2.10.2` as the C++ client 2.10.2. And C++ client 3.0.0
> >
> > > may conflict with the future java client 3.0.0. The information of
> >
> > > `client_version` is incomplete. We could not determine what
> >
> > > language(or specific library) the client uses. This raises
> >
> > > inconvenience for debugging.
> >
> > >
> >
> > > I would like to propose adding the client language information to the
> >
> > > `client_version`. Like:
> >
> > > * `java-2.11.1` for Java client 2.11
> >
> > > * `cpp-3.1.2` for C++ client 3.1.2
> >
> > > * `go-0.9.0` for go client 0.9.0
> >
> > > We can easily do that because the `client_version` is a string type.
> >
> > >
> >
> > > In addition, the Nodejs client and Python client all use the
> >
> > > client_version of C++ client they depend on. So it will use `3.1.2` as
> >
> > > the client_verion in Nodejs client 1.8.0. This is incorrect, and I
> >
> > > think we need to fix them. We don't need to print the C++ client
> >
> > > version because we can get that information if we know the Nodejs or
> >
> > > Python client version.
> >
> > >
> >
> > > What do you think? Please share your thoughts if you have any ideas.
> >
> > >
> >
> > > [0]
> >
> > >
> >
> >
> > https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L269
> >
> > > [1]
> >
> > >
> >
> >
> > https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L309
> >
> > > [2] https://pulsar.apache.org/docs/next/admin-api-topics/#get-stats
> >
> > >
> >
> > > Thanks,
> >
> > > Zike Yang
> >
> > >
> >
> >
> >

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

2023-02-21 Thread SiNan Liu 
About Proto3 and Proto2:
1. *schema compatibility checking for proto3*
Proto3 canceled the required field, so in the current design, there is no
need to check the required field. We can get the syntax of proto in the
code, and skip the check of the required field if it is proto3. All other
checking rules should apply to proto3.
2. *schema compatibility checking between proto3 and proto2*
For `SchemaCompatibilityStrategy` as BACKWARD, FORWARD and FULL, if the two
proto file syntax is different I think we should just throw an
exception. Schema
incompatibility.
throw new ProtoBufCanReadCheckException("Protobuf syntax have been changed."
);


Kiryl Valkovich  于2023年2月20日周一 22:51写道：

> [Edit] Sorry, it’s documented here:
> https://pulsar.apache.org/docs/2.11.x/schema-understand/#schema-compatibility-check
>
> From: Kiryl Valkovich 
> Date: Monday, February 20, 2023 at 3:48 PM
> To: dev@pulsar.apache.org 
> Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema
> compatibility checks without using avro-protobuf
> Hm. I tend to think that for my Pulsar use case, it would be good to have
> the ability to implement a custom schema compatibility checker.
>
> For example, buf.build (a popular Protobuf linter and build) offers the
> following list of breaking rules. Half of them prefixed with
> FIELD_SAME_XXLANG_PACKAGE_ aren’t relevant.
>
> And I would want to use a subset of them if we are checking a single
> message descriptor. Most likely:
> ENUM_VALUE_NO_DELETE
> FIELD_NO_DELETE
> FIELD_SAME_TYPE
> ONEOF_NO_DELETE
> FIELD_SAME_LABEL
> RESERVED_ENUM_NO_DELETE
> RESERVED_MESSAGE_NO_DELETE
>
> I found out that the Pulsar broker has the following configuration option:
> schemaRegistryCompatibilityCheckers
> [org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,
> org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,
> org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck]
>
> But I can’t find documentation for it.
>
> At first look, it seems that for my need it would be enough to have a such
> custom configurable Protobuf message descriptor checker class that
> implements SchemaCompatibilityCheck.
>
>
> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java#L31
>
> [cid:image002.png@01D94541.991807E0]
>
> [Text  Description automatically generated]
>
> From: SiNan Liu 
> Date: Monday, February 20, 2023 at 1:41 PM
> To: dev@pulsar.apache.org 
> Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema
> compatibility checks without using avro-protobuf
> It seems that we should only warn the user about changes to the field type,
> not define a compatibility check, or throw an exception.
> *Just like this: *
> *log.warn("proto.read.ProtobufSchema.uint64Field field type for uint64, was
> changed into a uint32");*
>
> I will update this in the PIP issue Alternatives and discuss both designs
> with other people.
>
>
> Kiryl Valkovich  于2023年2月20日周一 18:14写道：
>
> > 1. Sure, I didn’t mean that it's only about the required fields.
> > 2. I found the page you are referring to.
> > https://protobuf.dev/programming-guides/proto3/#updating
> >
> > QUOTE START
> > If a number is parsed from the wire which doesn’t fit in the
> corresponding
> > type, you will get the same effect as if you had cast the number to that
> > type in C++ (for example, if a 64-bit number is read as an int32, it will
> > be truncated to 32 bits).
> > QUOTE END
> >
> > It’s discussable. Maybe I’m just missing something.
> >
> > I personally wouldn’t rely on such compatibility guarantees in a real
> > application.
> > If my check amount (> int32 lol) would be truncated because someone
> > changed the field type in a schema, I would be quite upset.
> >
> > From: SiNan Liu 
> > Date: Monday, February 20, 2023 at 2:30 AM
> > To: dev@pulsar.apache.org 
> > Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema
> > compatibility checks without using avro-protobuf
> > Thank you for your suggestions and questions.
> > 1. Your first question. It's not just a matter of the required field.
> There
> > should be many differences between proto3 and proto2. I will later test
> the
> > current code for compatibility checks in proto3, and also look at
> > compatibility between different protobuf versions.
> > 2. On your second question. This is the compatibility rule for field type
> > changes I found on the official website. You mean that this rule should
> not
> > pass the compatibility check. Or should an exception be thrown whenever
> the
> > field type changes?
> >
> >
> > Kiryl Valkovich  于 2023年2月20日周一 上午12:31写道：
> >
> > > First, thank you for looking into it!
> > >
> > > There is a thing caught my eye:
> > >
> > > > The writtenSchema cannot add required fields, …
> > >
> > > As far as I know, the required fields were removed in Protocol Buffers
> > v3.

Re: [PROPOSAL] Roadmap for 3.0 release

2023-02-21 Thread tison 
+1

Thanks for driving this effort!

Also, I created https://github.com/apache/pulsar/issues/19570 to
correspondingly update our website. Feel free to leave a comment. I'm going
to send a draft in days.

Best,
tison.


Enrico Olivelli  于2023年2月21日周二 19:20写道：

> +1
>
> Perfect
>
> Enrico
>
> Il giorno mar 21 feb 2023 alle ore 08:11 Haiting Jiang
>  ha scritto:
> >
> > +1 Looks great!
> >
> >
> > Haiting
> >
> > On Tue, Feb 21, 2023 at 12:11 PM Hang Chen  wrote:
> > >
> > > +1
> > >
> > > Thanks,
> > > Hang
> > >
> > > Christophe Bornet  于2023年2月20日周一 21:46写道：
> > > >
> > > > +1
> > > >
> > > > Also, I'd like to volunteer as a release manager for this release.
> > > >
> > > > Christophe
> > > >
> > > >
> > > >
> > > >
> > > > Le ven. 17 févr. 2023 à 23:44, Matteo Merli  a
> écrit :
> > > > >
> > > > > Since the LTS release model has been formally approved, I'm
> proposing
> > > > > the following schedule for the release:
> > > > >
> > > > >  * Tue - 2023-05-11
> > > > >- RC-1
> > > > >- Code Freeze -- Only critical fixes will be merged in the
> 3.0
> > > > > release branch. Contributors should plan to have all the changes
> merged in
> > > > > before this date. Exceptions should be extremely rare and strongly
> > > > > motivated.
> > > > >
> > > > >  * Tue - 2023-05-18 - RC-2
> > > > >  * Tue - 2023-05-25 - RC-3
> > > > >  * Tue - 2023-05-02 - Announce 3.0 Release
> > > > >
> > > > > These dates will be published on the website to present users with
> a
> > > > > "roadmap" and we should commit to and respect these dates.
> > > > >
> > > > > I also wanted to propose trying out a model where we have 3 release
> > > > > managers for all major releases.
> > > > >
> > > > > The reasoning behind this is for this small group of people to
> collaborate
> > > > > and divide the tasks for the release: merging patches from the
> "master"
> > > > > branch, preparing RC, and testing.
> > > > >
> > > > > Since everyone also has other work duties and unexpected tasks
> that can pop
> > > > > up at any time, it will help to have redundancy in the
> release-management
> > > > > "team", so that we can release on the exact dates.
> > > > >
> > > > > Thanks,
> > > > > Matteo
> > > > >
> > > > > --
> > > > > Matteo Merli
> > > > > 
>

Re: [DISCUSS] ClientVersion conflict across multiple language client

2023-02-21 Thread Yunze Xu 
+1

But I'm wondering if it's possible to support configuring the client
version string by users? For example, if users forked their own
client, they might want to differ the client version with the official
client. The disadvantage could be that users can configure any version
string as they want.

Thanks,
Yunze

On Tue, Feb 21, 2023 at 7:23 PM Enrico Olivelli  wrote:
>
> +1
>
> Enrico
>
> Il giorno mar 21 feb 2023 alle ore 10:23 Baodi Shi 
> ha scritto:
> >
> >  +1, Good idea.
> >
> > Thanks,
> > Baodi Shi
> >
> >
> > 在 2023年2月21日 15:24:45 上，avinash kala  写道：
> >
> > > +1, sounds good.
> > >
> > > On Tue, Feb 21, 2023, 12:39 PM Zixuan Liu  wrote:
> > >
> > > +1, good idea!
> > >
> > >
> > > Thanks,
> > >
> > > Zixuan
> > >
> > >
> > > Zike Yang  于2023年2月21日周二 11:33写道：
> > >
> > >
> > > > Hi, all
> > >
> > > >
> > >
> > > > Currently, the Pulsar broker uses the field `client_version` to get
> > >
> > > > the version of the client. The client will send the client_version to
> > >
> > > > the broker through `CommandConnect` [0] during the connect or
> > >
> > > > `CommandAuthResponse ` [1] during the auth challenge. We could get the
> > >
> > > > client_version from the admin using `pulsar-admin topics stats xxx`
> > >
> > > > [2].
> > >
> > > >
> > >
> > > > But the `client_version ` only shows the version information. And
> > >
> > > > clients in different languages use their own separate version numbers.
> > >
> > > > This can lead to conflict. For example, the java client 2.10.2 uses
> > >
> > > > the same value `2.10.2` as the C++ client 2.10.2. And C++ client 3.0.0
> > >
> > > > may conflict with the future java client 3.0.0. The information of
> > >
> > > > `client_version` is incomplete. We could not determine what
> > >
> > > > language(or specific library) the client uses. This raises
> > >
> > > > inconvenience for debugging.
> > >
> > > >
> > >
> > > > I would like to propose adding the client language information to the
> > >
> > > > `client_version`. Like:
> > >
> > > > * `java-2.11.1` for Java client 2.11
> > >
> > > > * `cpp-3.1.2` for C++ client 3.1.2
> > >
> > > > * `go-0.9.0` for go client 0.9.0
> > >
> > > > We can easily do that because the `client_version` is a string type.
> > >
> > > >
> > >
> > > > In addition, the Nodejs client and Python client all use the
> > >
> > > > client_version of C++ client they depend on. So it will use `3.1.2` as
> > >
> > > > the client_verion in Nodejs client 1.8.0. This is incorrect, and I
> > >
> > > > think we need to fix them. We don't need to print the C++ client
> > >
> > > > version because we can get that information if we know the Nodejs or
> > >
> > > > Python client version.
> > >
> > > >
> > >
> > > > What do you think? Please share your thoughts if you have any ideas.
> > >
> > > >
> > >
> > > > [0]
> > >
> > > >
> > >
> > >
> > > https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L269
> > >
> > > > [1]
> > >
> > > >
> > >
> > >
> > > https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L309
> > >
> > > > [2] https://pulsar.apache.org/docs/next/admin-api-topics/#get-stats
> > >
> > > >
> > >
> > > > Thanks,
> > >
> > > > Zike Yang
> > >
> > > >
> > >
> > >
> > >

Re: Does anyone build UI for Pulsar?

2023-02-21 Thread Devin Bost 
Kiryl,

> Just make a side-by-side comparison of Pulsar Manager with any of the
following options:
> - Conduktor (Commercial).
> - Kafka UI by Provectus (Apache License 2.0).
> - Redpanda Console (ex-Kowl) (Mixed license).
>
> I see a significant gap in user experience between any of them and the
Pulsar Manager.

Thanks for bringing up this subject. You're making good points that we have
a gap in this area.

In your perspective, where do you think the biggest gaps are in terms of
functionality?

--
Devin Bost
Sent from mobile
Cell: 801-400-4602

On Mon, Feb 20, 2023, 8:54 AM Enrico Olivelli  wrote:

> Il giorno lun 20 feb 2023 alle ore 14:41 Kiryl Valkovich
>  ha scritto:
> >
> > Enrico, it’s easy.
> >
> > When I tried it, the basic functionality just didn’t work.
> >
> > Just make a side-by-side comparison of Pulsar Manager with any of the
> following options:
> > - Conduktor (Commercial).
> > - Kafka UI by Provectus (Apache License 2.0).
> > - Redpanda Console (ex-Kowl) (Mixed license).
> >
> > I see a significant gap in user experience between any of them and the
> Pulsar Manager.
> > Also, I just don’t see much contribution activity.
> >
> > Redpanda Console and Kafka UI, each have about x10 times more commits in
> a year, than Pulsar Manager has in the last 3 years. Therefore I can’t
> conclude that the project is alive.
>
> (Unfortunately) I 100% agree with you.
>
> >
> > Regarding the “Why from scratch?” question:
> > - I found it easier for me personally because of the different tech
> stack I use.
> > - Starting from scratch is a good opportunity to learn Pulsar itself.
> > - I’d want to make a product around that.
> >
> > I'm not claiming that my result will be 100% better, but why not try?
>
> I understand your points.
> I hope you will share your project with OSS license, this way it will
> be easier for Pulsar users to try it out and contribute.
> >
> > If someone is already doing or plans to do similar work, let’s
> collaborate!
>
> Thanks for sharing
>
> Enrico
>
> >
> > From: Enrico Olivelli 
> > Date: Monday, February 20, 2023 at 2:04 PM
> > To: dev@pulsar.apache.org 
> > Subject: Re: Does anyone build UI for Pulsar?
> > Kiryl,
> >
> > Il giorno lun 20 feb 2023 alle ore 12:18 Kiryl Valkovich
> >  ha scritto:
> > >
> > > Enrico, it seems you read only the mail message title.
> >
> > Sorry about that,
> > I have re-read the message, and I have just realised that I skipped
> > the very first line :-)
> >
> > I have one question.
> > IIUC you started from scratch a new UI, could you please explain why
> > Pulsar Manager doesn't work for you?
> >
> >
> > Cheers
> > Enrico
> >
> > >
> > > From: Enrico Olivelli 
> > > Date: Monday, February 20, 2023 at 11:59 AM
> > > To: dev@pulsar.apache.org 
> > > Subject: Re: Does anyone build UI for Pulsar?
> > > Kiryl,
> > >
> > > You can use the official Apache Pulsar Manager the is maintained by
> > > this community
> > > https://github.com/apache/pulsar-manager
> > >
> > > At DataStax we also maintain this other UI that is also 100% opensource
> > > https://github.com/datastax/pulsar-admin-console
> > >
> > > For the BookKeeper part there is BKVM (BookKeeper Visual Manager)
> > > https://github.com/diennea/bookkeeper-visual-manager
> > > This is also bundled with Apache Pulsar Manager
> > >
> > >
> > > I hope that helps
> > >
> > > Enrico
> > >
> > >
> > >
> > > Il giorno lun 20 feb 2023 alle ore 10:51 Kiryl Valkovich
> > >  ha scritto:
> > > >
> > > > Hi everyone!
> > > >
> > > > Does anyone personally or some company work on UI for Pulsar other
> than pulsar-manager or pulsar-admin-console?
> > > >
> > > > I understand that StreamNative and DataStax have managed solutions
> and obviously work on their UI.
> > > >
> > > > I rather looking for an open-source or commercial tool that can be
> used in pair with any Pulsar deployment.
> > > >
> > > > I spent some time implementing UI for Apache Pulsar. It’s not ready
> to release yet. As usual, the most difficult 20% of the work remained.
> > > >
> > > > I’m asking that to avoid situations when several people or teams are
> doing the same thing.
> > > > Recently I proposed the site redesign and some teams are already
> doing that as it’s figured out.
> > > >
> > > > See Asaf’s comment:
> > > >
> https://github.com/apache/pulsar-site/pull/426#issuecomment-1436589392
> > > >
> > > > Please at least DM me if it’s not public information yet.
> > > >
> > > > Also DM me if you think that we can try to cooperate.
> > > >
> > > > Thank you.
>

Re: Does anyone build UI for Pulsar?

2023-02-21 Thread Kiryl Valkovich 
Devin,

The most desired feature I need as an event-driven application developer is the 
ability to read and filter messages in topics.
With the NonDurable subscription type, it shouldn’t affect the topic’s data.

The ability to easily produce a message or generate a bunch of them, based on 
some rules, allows me to quickly check how the application behaves in different 
scenarios.

It’s also useful for the QA audience.

From: Devin Bost 
Date: Tuesday, February 21, 2023 at 4:58 PM
To: Dev 
Subject: Re: Does anyone build UI for Pulsar?
Kiryl,

> Just make a side-by-side comparison of Pulsar Manager with any of the
following options:
> - Conduktor (Commercial).
> - Kafka UI by Provectus (Apache License 2.0).
> - Redpanda Console (ex-Kowl) (Mixed license).
>
> I see a significant gap in user experience between any of them and the
Pulsar Manager.

Thanks for bringing up this subject. You're making good points that we have
a gap in this area.

In your perspective, where do you think the biggest gaps are in terms of
functionality?

--
Devin Bost
Sent from mobile
Cell: 801-400-4602

On Mon, Feb 20, 2023, 8:54 AM Enrico Olivelli  wrote:

> Il giorno lun 20 feb 2023 alle ore 14:41 Kiryl Valkovich
>  ha scritto:
> >
> > Enrico, it’s easy.
> >
> > When I tried it, the basic functionality just didn’t work.
> >
> > Just make a side-by-side comparison of Pulsar Manager with any of the
> following options:
> > - Conduktor (Commercial).
> > - Kafka UI by Provectus (Apache License 2.0).
> > - Redpanda Console (ex-Kowl) (Mixed license).
> >
> > I see a significant gap in user experience between any of them and the
> Pulsar Manager.
> > Also, I just don’t see much contribution activity.
> >
> > Redpanda Console and Kafka UI, each have about x10 times more commits in
> a year, than Pulsar Manager has in the last 3 years. Therefore I can’t
> conclude that the project is alive.
>
> (Unfortunately) I 100% agree with you.
>
> >
> > Regarding the “Why from scratch?” question:
> > - I found it easier for me personally because of the different tech
> stack I use.
> > - Starting from scratch is a good opportunity to learn Pulsar itself.
> > - I’d want to make a product around that.
> >
> > I'm not claiming that my result will be 100% better, but why not try?
>
> I understand your points.
> I hope you will share your project with OSS license, this way it will
> be easier for Pulsar users to try it out and contribute.
> >
> > If someone is already doing or plans to do similar work, let’s
> collaborate!
>
> Thanks for sharing
>
> Enrico
>
> >
> > From: Enrico Olivelli 
> > Date: Monday, February 20, 2023 at 2:04 PM
> > To: dev@pulsar.apache.org 
> > Subject: Re: Does anyone build UI for Pulsar?
> > Kiryl,
> >
> > Il giorno lun 20 feb 2023 alle ore 12:18 Kiryl Valkovich
> >  ha scritto:
> > >
> > > Enrico, it seems you read only the mail message title.
> >
> > Sorry about that,
> > I have re-read the message, and I have just realised that I skipped
> > the very first line :-)
> >
> > I have one question.
> > IIUC you started from scratch a new UI, could you please explain why
> > Pulsar Manager doesn't work for you?
> >
> >
> > Cheers
> > Enrico
> >
> > >
> > > From: Enrico Olivelli 
> > > Date: Monday, February 20, 2023 at 11:59 AM
> > > To: dev@pulsar.apache.org 
> > > Subject: Re: Does anyone build UI for Pulsar?
> > > Kiryl,
> > >
> > > You can use the official Apache Pulsar Manager the is maintained by
> > > this community
> > > https://github.com/apache/pulsar-manager
> > >
> > > At DataStax we also maintain this other UI that is also 100% opensource
> > > https://github.com/datastax/pulsar-admin-console
> > >
> > > For the BookKeeper part there is BKVM (BookKeeper Visual Manager)
> > > https://github.com/diennea/bookkeeper-visual-manager
> > > This is also bundled with Apache Pulsar Manager
> > >
> > >
> > > I hope that helps
> > >
> > > Enrico
> > >
> > >
> > >
> > > Il giorno lun 20 feb 2023 alle ore 10:51 Kiryl Valkovich
> > >  ha scritto:
> > > >
> > > > Hi everyone!
> > > >
> > > > Does anyone personally or some company work on UI for Pulsar other
> than pulsar-manager or pulsar-admin-console?
> > > >
> > > > I understand that StreamNative and DataStax have managed solutions
> and obviously work on their UI.
> > > >
> > > > I rather looking for an open-source or commercial tool that can be
> used in pair with any Pulsar deployment.
> > > >
> > > > I spent some time implementing UI for Apache Pulsar. It’s not ready
> to release yet. As usual, the most difficult 20% of the work remained.
> > > >
> > > > I’m asking that to avoid situations when several people or teams are
> doing the same thing.
> > > > Recently I proposed the site redesign and some teams are already
> doing that as it’s figured out.
> > > >
> > > > See Asaf’s comment:
> > > >
> https://github.com/apache/pulsar-site/pull/426#issuecomment-1436589392
> > > >
> > > > Please at least DM me if it’s not public information yet.
> > > >
> > > > Also DM me if you think

Re: [DISCUSS] ClientVersion conflict across multiple language client

2023-02-21 Thread Michael Marshall 
+1 - I think this is a helpful change. We already do this in the Java
Admin http client when we set the user agent [0].

> But I'm wondering if it's possible to support configuring the client
> version string by users?

This is always the case because we support third party clients. In my
view, this field is primarily helpful for troubleshooting and for
getting weakly trusted stats on usage in a given cluster. Operators
could even use this information to determine and alert if any clients
are connecting with known vulnerabilities.

Thanks,
Michael

[0] 
https://github.com/apache/pulsar/blob/d8569cd4ec6da14f8b2b9338db1ed2f6a3eacf0a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java#L105

On Tue, Feb 21, 2023 at 9:50 AM Yunze Xu  wrote:
>
> +1
>
> But I'm wondering if it's possible to support configuring the client
> version string by users? For example, if users forked their own
> client, they might want to differ the client version with the official
> client. The disadvantage could be that users can configure any version
> string as they want.
>
> Thanks,
> Yunze
>
> On Tue, Feb 21, 2023 at 7:23 PM Enrico Olivelli  wrote:
> >
> > +1
> >
> > Enrico
> >
> > Il giorno mar 21 feb 2023 alle ore 10:23 Baodi Shi 
> > ha scritto:
> > >
> > >  +1, Good idea.
> > >
> > > Thanks,
> > > Baodi Shi
> > >
> > >
> > > 在 2023年2月21日 15:24:45 上，avinash kala  写道：
> > >
> > > > +1, sounds good.
> > > >
> > > > On Tue, Feb 21, 2023, 12:39 PM Zixuan Liu  wrote:
> > > >
> > > > +1, good idea!
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Zixuan
> > > >
> > > >
> > > > Zike Yang  于2023年2月21日周二 11:33写道：
> > > >
> > > >
> > > > > Hi, all
> > > >
> > > > >
> > > >
> > > > > Currently, the Pulsar broker uses the field `client_version` to get
> > > >
> > > > > the version of the client. The client will send the client_version to
> > > >
> > > > > the broker through `CommandConnect` [0] during the connect or
> > > >
> > > > > `CommandAuthResponse ` [1] during the auth challenge. We could get the
> > > >
> > > > > client_version from the admin using `pulsar-admin topics stats xxx`
> > > >
> > > > > [2].
> > > >
> > > > >
> > > >
> > > > > But the `client_version ` only shows the version information. And
> > > >
> > > > > clients in different languages use their own separate version numbers.
> > > >
> > > > > This can lead to conflict. For example, the java client 2.10.2 uses
> > > >
> > > > > the same value `2.10.2` as the C++ client 2.10.2. And C++ client 3.0.0
> > > >
> > > > > may conflict with the future java client 3.0.0. The information of
> > > >
> > > > > `client_version` is incomplete. We could not determine what
> > > >
> > > > > language(or specific library) the client uses. This raises
> > > >
> > > > > inconvenience for debugging.
> > > >
> > > > >
> > > >
> > > > > I would like to propose adding the client language information to the
> > > >
> > > > > `client_version`. Like:
> > > >
> > > > > * `java-2.11.1` for Java client 2.11
> > > >
> > > > > * `cpp-3.1.2` for C++ client 3.1.2
> > > >
> > > > > * `go-0.9.0` for go client 0.9.0
> > > >
> > > > > We can easily do that because the `client_version` is a string type.
> > > >
> > > > >
> > > >
> > > > > In addition, the Nodejs client and Python client all use the
> > > >
> > > > > client_version of C++ client they depend on. So it will use `3.1.2` as
> > > >
> > > > > the client_verion in Nodejs client 1.8.0. This is incorrect, and I
> > > >
> > > > > think we need to fix them. We don't need to print the C++ client
> > > >
> > > > > version because we can get that information if we know the Nodejs or
> > > >
> > > > > Python client version.
> > > >
> > > > >
> > > >
> > > > > What do you think? Please share your thoughts if you have any ideas.
> > > >
> > > > >
> > > >
> > > > > [0]
> > > >
> > > > >
> > > >
> > > >
> > > > https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L269
> > > >
> > > > > [1]
> > > >
> > > > >
> > > >
> > > >
> > > > https://github.com/apache/pulsar/blob/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e/pulsar-common/src/main/proto/PulsarApi.proto#L309
> > > >
> > > > > [2] https://pulsar.apache.org/docs/next/admin-api-topics/#get-stats
> > > >
> > > > >
> > > >
> > > > > Thanks,
> > > >
> > > > > Zike Yang
> > > >
> > > > >
> > > >
> > > >
> > > >

Re: [VOTE][PIP-242] Topic name restriction

2023-02-21 Thread mattisonchao 
This PIP has passed voting. 5 (binding) and 4 (non-binding)

+1 (binding)

• Yunze Xu
• Penghui Li
• Guo jiwei
• Haiting Jiang
• Cong Bo


+1 (non-binding)

• Asaf Mesika
• Cong Zhao
• Zike Yang
• avinash kala



Thanks to you all, Let's meet at the next PIP.

Best,
Mattison
On Feb 18, 2023, 16:58 +0800, mattisonc...@gmail.com, wrote:
> Hi, All
>
> After a fascinating discussion, I would start the vote of PIP-242.
>
> We have chosen to drop out the `system topic` related improvement to another 
> PIP. Therefore, the current version is simple enough and it has a clear 
> boundary.
>
> Please leave +1/-1 in this thread to join the vote. and feel free to leave 
> any concerns.
>
> Thanks to you guys.
>
> Best,
> Mattison
>
> References:
>
> • PIP https://github.com/apache/pulsar/issues/19239
> • Discussion https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
>
>

Re: [VOTE][PIP-242] Topic name restriction

2023-02-21 Thread Michael Marshall 
+1 (sorry for the late vote, and thank you for the extra discussion)

- Michael

On Tue, Feb 21, 2023 at 9:04 PM  wrote:
>
> This PIP has passed voting. 5 (binding) and 4 (non-binding)
>
> +1 (binding)
>
> • Yunze Xu
> • Penghui Li
> • Guo jiwei
> • Haiting Jiang
> • Cong Bo
>
>
> +1 (non-binding)
>
> • Asaf Mesika
> • Cong Zhao
> • Zike Yang
> • avinash kala
>
>
>
> Thanks to you all, Let's meet at the next PIP.
>
> Best,
> Mattison
> On Feb 18, 2023, 16:58 +0800, mattisonc...@gmail.com, wrote:
> > Hi, All
> >
> > After a fascinating discussion, I would start the vote of PIP-242.
> >
> > We have chosen to drop out the `system topic` related improvement to 
> > another PIP. Therefore, the current version is simple enough and it has a 
> > clear boundary.
> >
> > Please leave +1/-1 in this thread to join the vote. and feel free to leave 
> > any concerns.
> >
> > Thanks to you guys.
> >
> > Best,
> > Mattison
> >
> > References:
> >
> > • PIP https://github.com/apache/pulsar/issues/19239
> > • Discussion 
> > https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
> >
> >

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

2023-02-21 Thread Yunze Xu 
+1 (binding)
* Verified checksum and signature
* Build from source
* Install from npm on Ubuntu 20.04
* Run an end-to-end test with custom `tlsTrustCertsFilePath` config on
StreamNative cloud with OAuth2 authentication

BTW, from the discussion here [1], it would be better to use
https://downloads.apache.org/pulsar/KEYS as the KEYS,

[1] https://lists.apache.org/thread/f9w430oqpm0g72b1htwbtc8y3mfqf8r6

Thanks,
Yunze

On Mon, Feb 20, 2023 at 5:36 PM Nozomi Kurihara  wrote:
>
> +1 (binding)
>
> * checked license headers
> * verified checksum and signature
> * install from npm and run producer/consumer
>
> Thanks,
> Nozomi
>
> 2023年2月17日(金) 19:12 Baodi Shi :
>
> > Hi everyone,
> >
> > This is the first release candidate for Apache Pulsar Node.js client,
> > version 1.8.1.
> >
> > It fixes the following
> > issues:
> > https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.1+is%3Aclosed
> >
> > Please download the source files and review this release candidate:
> > - Download the source package, verify shasum and asc
> > - Follow the README.md to build and run the Pulsar Node.js client.
> >
> > The release candidate package has been published to the npm
> > registry:https://www.npmjs.com/package/pulsar-client/v/1.8.1-rc.1
> > You can install it by `npm i pulsar-client@1.8.1-rc.1
> > --pulsar_binary_host_mirror=
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
> > 
> > and verify the package.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Source files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.1-rc.1/
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the
> > release:https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > SHA-512 checksum:
> >
> > ed89b4ad467d3cb75ed37096b35d91b872cd93d36cd953512fc7edcb75dbac5162592f6f51b5ab08f26b3dca1c57a3d3fe7a5e4f109551c66943a5b09392d51a
> >  apache-pulsar-client-node-1.8.1.tar.gz
> > The tag to be voted upon:
> > v1.8.1-rc.1(3e843f0)
> > https://github.com/apache/pulsar-client-node/releases/tag/v1.8.1-rc.1
> >
> > Please review and vote on the release candidate #1 for the version
> > 1.8.1, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> >
> >
> > Thanks,
> > Baodi Shi
> >

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

2023-02-21 Thread Nicolò Boschi 
Hi,

I'm having issues while validating the fix related to the hostname
verification: https://github.com/apache/pulsar-client-cpp/pull/126
My usecase is with a valid TLS certificate signed by a CA (not a
self-signed one).

My code is very simple (see below): it creates a client with token auth +
TLS and sends some messages.

It works well with node client 1.7.0 with cpp client 3.1.2
It fails with node client 1.8.0 (as expected)
It still fails with the rc: 1.8.1-rc.1

(I'm installing the dependency with "npm i pulsar-client@1.8.1-rc.1
--pulsar_binary_host_mirror=
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/";

The error I'm seeing is this one:

[INFO][ClientConnection:388] Connected to broker
[ERROR][ClientConnection:488] Handshake failed: certificate verify failed
(SSL routines, tls_process_server_certificate)
[INFO][ClientConnection:1600] Connection closed with ConnectError

Note that setting `tlsValidateHostname: true` "resolves" the problem,
however it's not acceptable as you know.

I'm pretty sure that it's related to the cpp client dependency, however I'm
not very familiar with it and how it's bundled in the node client >= 1.8.0
Is there a way to verify if the bundled cpp client is actually the
expected one?



This is the code snippet:
```
const tokenStr = asToken;
  const pulsarUri = pulsarUrl;
  const topicName = asTopic;

  const auth = new Pulsar.AuthenticationToken({ token: tokenStr });
  const client = new Pulsar.Client({
serviceUrl: pulsarUri,
authentication: auth,
operationTimeoutSeconds: 30,
tlsCertificateFilePath: "",
tlsValidateHostname: false
  });
  Pulsar.Client.setLogHandler((level, file, line, message) => {
console.log('[%s][%s:%d] %s', Pulsar.LogLevel.toString(level), file,
line, message);
  });

  const producer = await client.createProducer({
topic: topicName,
  })

  for (let i = 0; i < 10; i += 1) {
await producer.send({
  data: Buffer.from("nodejs-message-" + i),
});
console.log("send message " + i);
  }
  await producer.flush();
  await producer.close();
  await client.close();

```

Thanks,
Nicolò Boschi


Il giorno mer 22 feb 2023 alle ore 08:02 Yunze Xu
 ha scritto:

> +1 (binding)
> * Verified checksum and signature
> * Build from source
> * Install from npm on Ubuntu 20.04
> * Run an end-to-end test with custom `tlsTrustCertsFilePath` config on
> StreamNative cloud with OAuth2 authentication
>
> BTW, from the discussion here [1], it would be better to use
> https://downloads.apache.org/pulsar/KEYS as the KEYS,
>
> [1] https://lists.apache.org/thread/f9w430oqpm0g72b1htwbtc8y3mfqf8r6
>
> Thanks,
> Yunze
>
> On Mon, Feb 20, 2023 at 5:36 PM Nozomi Kurihara 
> wrote:
> >
> > +1 (binding)
> >
> > * checked license headers
> > * verified checksum and signature
> > * install from npm and run producer/consumer
> >
> > Thanks,
> > Nozomi
> >
> > 2023年2月17日(金) 19:12 Baodi Shi :
> >
> > > Hi everyone,
> > >
> > > This is the first release candidate for Apache Pulsar Node.js client,
> > > version 1.8.1.
> > >
> > > It fixes the following
> > > issues:
> > >
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.1+is%3Aclosed
> > >
> > > Please download the source files and review this release candidate:
> > > - Download the source package, verify shasum and asc
> > > - Follow the README.md to build and run the Pulsar Node.js client.
> > >
> > > The release candidate package has been published to the npm
> > > registry:https://www.npmjs.com/package/pulsar-client/v/1.8.1-rc.1
> > > You can install it by `npm i pulsar-client@1.8.1-rc.1
> > > --pulsar_binary_host_mirror=
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
> 
> > > 
> > > and verify the package.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Source files:
> > >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.1-rc.1/
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the
> > > release:https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > SHA-512 checksum:
> > >
> > >
> ed89b4ad467d3cb75ed37096b35d91b872cd93d36cd953512fc7edcb75dbac5162592f6f51b5ab08f26b3dca1c57a3d3fe7a5e4f109551c66943a5b09392d51a
> > >  apache-pulsar-client-node-1.8.1.tar.gz
> > > The tag to be voted upon:
> > > v1.8.1-rc.1(3e843f0)
> > > https://github.com/apache/pulsar-client-node/releases/tag/v1.8.1-rc.1
> > >
> > > Please review and vote on the release candidate #1 for the version
> > > 1.8.1, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > >
> > >
> > > Thanks,
> > > Baodi Shi
> > >
>