Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Asaf Mesika]

I would have the bot open a Thread for the message, *suggesting* the user
to click to convert this question into a GitHub Discussion question. This
way you can have the actual GitHub user asking the question and not a bot
one.

On Fri, Feb 17, 2023 at 10:59 PM Kiryl Valkovich 
wrote:

> What about such wording?
>
> ---
> Your question was moved here:
> https://github.com/apache/pulsar/discussions/123
>
> Please consider asking new questions here:
>
>   *   At StackOverflow using apache-pulsar tag.
>   *   In the Q&A category at GitHub Discussions.
>   *   Apache Pulsar User Mailing List.
>
>
> It will make it searchable by others. Also, this way we can collect a
> knowledge base outside of Slack over time.
>
> I can’t see how the words “please consider” force the user to do something.
>
> Users who have an account on StackOverflow or GitHub can use these
> platforms next time.
> Others can send their question via the mailing list.
>
> From: Dave Fisher 
> Date: Friday, February 17, 2023 at 9:28 PM
> To: dev@pulsar.apache.org 
> Subject: Re: Force redirect questions from Slack to GitHub Discussions or
> StackOverflow?
> My concern is that users should have a choice on where to post their
> questions. They might have concerns about GitHub’s terms and conditions. We
> can pin a message to slack pointing to GitHub discussions and stackoverflow.
>
> Best,
> Dave
>
> Sent from my iPhone
>
> > On Feb 17, 2023, at 9:22 AM, Kiryl Valkovich 
> wrote:
> >
> > I’m the owner of this account.
> > The goal is to test drive duplicating Slack questions to the GitHub
> discussions.
> > With the current level of activity in Slack it’s not so hard to do it
> manually.
> >
> > I’m in CET now. I can share the account credentials with people who can
> post questions to GitHub Discussions on behalf of this account in other
> time zones.
> > Or I can do it once a day.
> >
> > If someone doesn’t find it useful or has ideas on how to do it in a
> better way, just say it directly.
> >
> > From: Enrico Olivelli 
> > Date: Friday, February 17, 2023 at 3:43 PM
> > To: dev@pulsar.apache.org 
> > Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> > Hello,
> > I see that some "Pulsar Community Bot" appeared in Slack
> >
> > it is connected to this email address "pulsar.community@gmail.com"
> >
> > While I find this thing "amazing"I wonder if I missed something,
> > who is the owner of this "bot" ?
> >
> >
> > Enrico
> >
> >> Il giorno gio 16 feb 2023 alle ore 16:03 Kiryl Valkovich
> >>  ha scritto:
> >>
> >> Played with Slack and StackOverflow APIs a bit.
> >>
> >> The Slack API works as expected. After clicking the message button, it
> sends a message descriptor to my app where I can do anything with its
> content.
> >>
> >> It’s possible to post messages via the StackOverflow API, but it’s
> unlikely that any Slack message can be converted to a StackOverflow
> question.
> >>
> >> I encountered several types of validation errors:
> >>
> >> -  This question body does not meet our quality standards.
> Please make sure that it completely describes your problem - including what
> you have already tried - and is written using proper grammar.
> >>
> >>  *   Something similar to “This message looks like a duplicate of
> another message”.
> >>
> >> I believe, GitHub Discussions don’t have such kind of “quality
> standards” validation.
> >>
> >> From: Kiryl Valkovich 
> >> Date: Thursday, February 16, 2023 at 1:33 PM
> >> To: dev@pulsar.apache.org 
> >> Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> >> If there are no hidden obstacles, we can implement it.
> >> StackOverflow supports question creation using REST API:
> https://api.stackexchange.com/docs/create-question
> >>
> >> From: Zike Yang 
> >> Date: Thursday, February 16, 2023 at 11:41 AM
> >> To: dev@pulsar.apache.org 
> >> Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> >> +1, That sounds great to me.
> >>
> >>> 2. You click the three dots button on his message -> “Convert to a
> GitHub discussion”.
> >>
> >> Is it a similar workflow for converting to a StackOverflow question?
> >>
> >> BR,
> >> Zike Yang
> >>
> >>> On Thu, Feb 16, 2023 at 12:14 PM  wrote:
> >>>
> >>> +1
> >>>
> >>> Since web pages are more easily and public.
> >>>
> >>> Best
> >>> Mattison
> >>> On Feb 16, 2023, 07:58 +0800, Christophe Bornet <
> bornet.ch...@gmail.com>, wrote:
>  +100
>  Also note that for good or bad reasons, the number of questions on
>  StaOverflow is often used as a metric for the popularity of a project.
> 
>  Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich
> 
>  a écrit :
> 
> > Hi everyone! Enrico Olivelli asked me to repost it to the mailing
> list.
> >
> > --- Me
> > I’m very worried that good answers from David Kjerrumgaard and others
> > won’t be googleable because it’s in Slack. To make any search 

Re: [VOTE][PIP-242] Topic name restriction

[Asaf Mesika]

+1 (non-binding)

On Sat, Feb 18, 2023 at 10:58 AM  wrote:

> Hi, All
>
> After a fascinating discussion, I would start the vote of PIP-242.
>
> We have chosen to drop out the `system topic` related improvement to
> another PIP. Therefore, the current version is simple enough and it has a
> clear boundary.
>
> Please leave +1/-1 in this thread to join the vote. and feel free to leave
> any concerns.
>
> Thanks to you guys.
>
> Best,
> Mattison
>
> References:
>
> • PIP https://github.com/apache/pulsar/issues/19239
> • Discussion
> https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
>
>
>

Re: [DISCUSS] PIP-242 Topic name restrictions

[Asaf Mesika]

Is  https://github.com/apache/pulsar/pull/19235 somehow related?

On Sat, Feb 18, 2023 at 10:38 AM  wrote:

> Hi, All
>
> After discussing with Enrico and Michael offline.
> I will split the discussed topic into two PIP.
>
> 1. Topic name restrictions
> a. `-partition-` keyword.
> b. enable topic name character pattern.
> 2. System topic
> a. System topic name pattern.
> b. System topic authorisation.
> c. ...
>
> In this approach, we will get a clear boundary and avoid going off the
> initial scope.
>
> Since we don't have any question about the first scope. I will start vote
> next week.
>
> Thanks to all participant.
>
> Best,
> Mattison
>
>
>
> On Feb 18, 2023, 14:24 +0800, Michael Marshall ,
> wrote:
> > I support breaking this into two PIPs. It was my fault the two PIPs
> > were merged in the first place. I am sorry if I created any confusion.
> > My intention was only to point out that names are a meaningful way to
> > simplify logic, and we should reserve certain names for Pulsar's own
> > usage with a well defined pattern so that we can simplify lifecycle
> > operations.
> >
> > Thanks,
> > Michael
> >
> > On Fri, Feb 17, 2023 at 1:55 AM Enrico Olivelli 
> wrote:
> > >
> > > Mattison,
> > >
> > > Il giorno gio 16 feb 2023 alle ore 00:27  ha
> scritto:
> > > > >
> > > > > > > I am sorry but I am not sure that this is enough to
> preventreads/writes from unallowed clients.
> > > > > IMO, We can consider the authorisation part in another PIP because
> We are just focusing on adding the topic name constraint of topic creation.
> > > > >
> > > > > Maybe we can use another PIP to clearify all of system topic's
> behaviour, like authorisation something.
> > > > > e.g. we just allow superusers to read/write the data to that
> system topic.
> > > > > > > We should elaborate more on this topic on the PIP
> > > > > I will add the internal system topic creation logic in the PIP.
> > > Why do you think that this is enough ?
> > >
> > > I think that we are going off the initial scope of the PIP.
> > > The initial problem is about preventing clients from creating topics
> > > that contain the "-partition-" keyword.
> > >
> > > I totally agree that there must be a clear way to distinguish topics
> > > that are not meant to be accessed by "regular clients".
> > >
> > > The answer is in Micheal's words: only super users are allowed to
> > > access topics that are not meant to be accessed by clients.
> > > Broker to Broker communications are always running with a "super user"
> > > role, so it is not a problem.
> > >
> > > BTW I wonder if it is better to narrow down the scope of the PIP and
> > > go back to "-partition-"
> > >
> > >
> > > Enrico
> > >
> > >
> > > > >
> > > > > Best,
> > > > > Mattison
> > > > > On Feb 16, 2023, 00:41 +0800, Enrico Olivelli ,
> wrote:
> > > > > > > Il giorno mer 15 feb 2023 alle ore 17:07 <
> mattisonc...@gmail.com> ha scritto:
> > > > > > > > >
> > > > > > > > > Hi Enrico
> > > > > > > > >
> > > > > > > > > I think it's a good question. We can introduce a new
> method in the BrokerService to help brokers create the topic internally
> first(maybe just metadata is enough), and then to use a pulsar client to
> connect to it.
> > > > > > >
> > > > > > > I am sorry but I am not sure that this is enough to prevent
> > > > > > > reads/writes from unallowed clients.
> > > > > > > We should elaborate more on this topic on the PIP
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Mattison
> > > > > > > > > On Feb 16, 2023, 00:01 +0800, Enrico Olivelli <
> eolive...@gmail.com>, wrote:
> > > > > > > > > > > > > I have one question (apologies for the top
> posting).
> > > > > > > > > > > > >
> > > > > > > > > > > > > The Broker (and the other Pulsar components) use
> the regular Pulsar
> > > > > > > > > > > > > client to connect to "system topics"
> > > > > > > > > > > > > and in general they use the Pulsar wire protocol.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The question is "how do you distinguish an
> internal component from a
> > > > > > > > > > > > > user component ?"
> > > > > > > > > > > > > How can you say that the broker is able to connect
> to a system topic
> > > > > > > > > > > > > and any other client cannot do it ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Enrico
> > > > > > > > > > > > >
> > > > > > > > > > > > > Il giorno mer 15 feb 2023 alle ore 15:38 <
> mattisonc...@gmail.com> ha scritto:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Asaf
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There is a link to introduce the dynamic
> configuration.
> > > > > > > > > > > > > > > > >
> https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Matti

[DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[SiNan Liu]

Hi all,

I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.

We can talk about the current design here. Especially for the field type
change check rules, please give your valuable advice.

Thanks,
Sinan

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[Kiryl Valkovich]

First, thank you for looking into it!

There is a thing caught my eye:

> The writtenSchema cannot add required fields, …

As far as I know, the required fields were removed in Protocol Buffers v3.

I see proto3 usage in existing schema registry tests:
https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/test/proto/ProtobufSchemaTest.proto#L19

https://github.com/apache/pulsar/blob/1ea4ad814f5f30b8c371db2a86626cd568ace553/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto#L19

I see proto2 usage in the proposed changes:

https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19

https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19

Also, I’m not sure about this:

> int32, uint32, int64, uint64, and bool are all compatible – this means you 
> can change a field from one of these types to another without breaking 
> forwards- or backwards-compatibility.

It leads to JS/PHP-like implicit conversions if I understand it right.

From: SiNan Liu 
Date: Sunday, February 19, 2023 at 1:59 PM
To: dev@pulsar.apache.org 
Subject: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility 
checks without using avro-protobuf
Hi all,

I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.

We can talk about the current design here. Especially for the field type
change check rules, please give your valuable advice.

Thanks,
Sinan

Modern look for the website

[Kiryl Valkovich]

Hi everyone!



I don't think anyone will argue, it’s important to have a fresh-looking website 
to attract new users.



I decided to try to refresh it a little bit.



Over the weekend I managed to make a home page. I think that I can finish other 
pages in 1-2 weeks, depending on my workload.



Current version: https://pulsar.apache.org/

New version: https://pulsar-site-three.vercel.app



Draft PR: https://github.com/apache/pulsar-site/pull/426



After the redesign, I plan to add a few more pages.



Should I continue?

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[SiNan Liu]

Thank you for your suggestions and questions.
1. Your first question. It's not just a matter of the required field. There
should be many differences between proto3 and proto2. I will later test the
current code for compatibility checks in proto3, and also look at
compatibility between different protobuf versions.
2. On your second question. This is the compatibility rule for field type
changes I found on the official website. You mean that this rule should not
pass the compatibility check. Or should an exception be thrown whenever the
field type changes?


Kiryl Valkovich  于 2023年2月20日周一 上午12:31写道：

> First, thank you for looking into it!
>
> There is a thing caught my eye:
>
> > The writtenSchema cannot add required fields, …
>
> As far as I know, the required fields were removed in Protocol Buffers v3.
>
> I see proto3 usage in existing schema registry tests:
>
> https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/test/proto/ProtobufSchemaTest.proto#L19
>
>
> https://github.com/apache/pulsar/blob/1ea4ad814f5f30b8c371db2a86626cd568ace553/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto#L19
>
> I see proto2 usage in the proposed changes:
>
>
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
>
>
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
>
> Also, I’m not sure about this:
>
> > int32, uint32, int64, uint64, and bool are all compatible – this means
> you can change a field from one of these types to another without breaking
> forwards- or backwards-compatibility.
>
> It leads to JS/PHP-like implicit conversions if I understand it right.
>
> From: SiNan Liu 
> Date: Sunday, February 19, 2023 at 1:59 PM
> To: dev@pulsar.apache.org 
> Subject: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility
> checks without using avro-protobuf
> Hi all,
>
> I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.
>
> We can talk about the current design here. Especially for the field type
> change check rules, please give your valuable advice.
>
> Thanks,
> Sinan
>

Re: [VOTE][PIP-242] Topic name restriction

[Yunze Xu]

+1 (binding)

Thanks,
Yunze

On Sun, Feb 19, 2023 at 6:58 PM Asaf Mesika  wrote:
>
> +1 (non-binding)
>
> On Sat, Feb 18, 2023 at 10:58 AM  wrote:
>
> > Hi, All
> >
> > After a fascinating discussion, I would start the vote of PIP-242.
> >
> > We have chosen to drop out the `system topic` related improvement to
> > another PIP. Therefore, the current version is simple enough and it has a
> > clear boundary.
> >
> > Please leave +1/-1 in this thread to join the vote. and feel free to leave
> > any concerns.
> >
> > Thanks to you guys.
> >
> > Best,
> > Mattison
> >
> > References:
> >
> > • PIP https://github.com/apache/pulsar/issues/19239
> > • Discussion
> > https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
> >
> >
> >

Re: Modern look for the website

[Yu]

Hi @visortelle thank you very much for your awsome job! This is a great
addition to the Pulsar website!

I've checked the design and left my comments at
https://github.com/apache/pulsar-site/pull/426#issuecomment-1436220520.

Also request more reviews from relevant stakeholders.

On Mon, Feb 20, 2023 at 6:38 AM Kiryl Valkovich 
wrote:

> Hi everyone!
>
>
>
> I don't think anyone will argue, it’s important to have a fresh-looking
> website to attract new users.
>
>
>
> I decided to try to refresh it a little bit.
>
>
>
> Over the weekend I managed to make a home page. I think that I can finish
> other pages in 1-2 weeks, depending on my workload.
>
>
>
> Current version: https://pulsar.apache.org/
>
> New version: https://pulsar-site-three.vercel.app
>
>
>
> Draft PR: https://github.com/apache/pulsar-site/pull/426
>
>
>
> After the redesign, I plan to add a few more pages.
>
>
>
> Should I continue?
>
>

Re: Modern look for the website

[PengHui Li]

It looks super cool!

I just found some links that don't work.

The button "Learn more" and "QuickStart" in the index page will forward to
404 page
[image: image.png]

And the case study page looks not coordinated well.

https://pulsar-site-three.vercel.app/case-studies

Thanks for the great job.


Penghui

On Mon, Feb 20, 2023 at 6:38 AM Kiryl Valkovich 
wrote:

> Hi everyone!
>
>
>
> I don't think anyone will argue, it’s important to have a fresh-looking
> website to attract new users.
>
>
>
> I decided to try to refresh it a little bit.
>
>
>
> Over the weekend I managed to make a home page. I think that I can finish
> other pages in 1-2 weeks, depending on my workload.
>
>
>
> Current version: https://pulsar.apache.org/
>
> New version: https://pulsar-site-three.vercel.app
>
>
>
> Draft PR: https://github.com/apache/pulsar-site/pull/426
>
>
>
> After the redesign, I plan to add a few more pages.
>
>
>
> Should I continue?
>
>

Re: [VOTE][PIP-242] Topic name restriction

[Cong Zhao]

+1 (non-binding)

Thanks,
Cong

On 2023/02/18 08:58:26 mattisonc...@gmail.com wrote:
> Hi, All
> 
> After a fascinating discussion, I would start the vote of PIP-242.
> 
> We have chosen to drop out the `system topic` related improvement to another 
> PIP. Therefore, the current version is simple enough and it has a clear 
> boundary.
> 
> Please leave +1/-1 in this thread to join the vote. and feel free to leave 
> any concerns.
> 
> Thanks to you guys.
> 
> Best,
> Mattison
> 
> References:
> 
> • PIP https://github.com/apache/pulsar/issues/19239
> • Discussion https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
> 
> 
> 

[DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.

[Xiangying Meng]

Hi, Community,
We plan to add an internal class to `TransactionBufferStats` to record the
snapshot status uniformly.
As we all know, the current transaction buffer(TB) filters the messages
sent using the aborted transaction by storing the aborted ID in TB.
Then TB will periodically store these aborted txn IDs in a bookie entry in
the form of snapshots so that TB can recover faster when recovering.
But as more and more people use transactions, we found that in some extreme
cases, a bookie entry may not be able to store all aborted transaction IDs.
So in PIP196 , we
implemented the multiple-snapshot function.
As the transaction buffer snapshot mechanism becomes increasingly complex,
the only information related to the transaction snapshot is
`lastSnapshotTimestamps`; That is not enough, we need to add more info to
record the snapshot stats.
So I suggest adding an internal class SnapshotStats to
TransactionBufferStats to record the snapshot status uniformly.

The modification could be :
```java
public class TransactionBufferStats {
...
public long lastSnapshotTimestamps;
...
}
```
```java
public class TransactionBufferStats {
...
//public long lastSnapshotTimestamps;
...
public SnapshotStats snapshotStats;

public static class SnapshotStats {
public long segmentsSize;

public long unsealedAbortTxnIDs;


public long lastSnapshotTimestamps;
}
}

```
Thanks.
Xiangying

[DISCUSS] Release 2.9.5

[mattisonchao]

Hello, Pulsar community:

I'd like to propose releasing Apache Pulsar 2.9.5. It's been about
two months since 2.9.4 was released.

There are 54 PRs [0] needed to cherry-pick in branch-2.9. I will
cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
directly into branch-2.9.

There are 21 PRs [1] opened. I'll follow up on each of those PRs to
see if they will be completed soon or will need to be pushed to 2.9.6

If you have any important fixes or any questions, please reply to this
email, and we will evaluate whether to include them in 2.9.5


Best,
Mattison

[0] 
https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.5+-label%3Acherry-picked%2Fbranch-2.9+is%3Aclosed+
[1] 
https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.9.5

Re: Modern look for the website

[Kiryl Valkovich]

I asked only about the home/index/root (/) page look.

Other pages just aren't re-implemented.

I don't know the history behind the previous redesign. I met many situations 
when people didn't want to consider changes for some reason.

Therefore, I must be sure that the proposed changes most likely will be 
accepted before investing my time.


From: PengHui Li 
Date: Monday, February 20, 2023 at 4:51 AM
To: dev@pulsar.apache.org 
Subject: Re: Modern look for the website
It looks super cool!

I just found some links that don't work.

The button "Learn more" and "QuickStart" in the index page will forward to 404 
page
[cid:ii_leca41v80]

And the case study page looks not coordinated well.

https://pulsar-site-three.vercel.app/case-studies

Thanks for the great job.


Penghui

On Mon, Feb 20, 2023 at 6:38 AM Kiryl Valkovich  
wrote:
Hi everyone!



I don't think anyone will argue, it’s important to have a fresh-looking website 
to attract new users.



I decided to try to refresh it a little bit.



Over the weekend I managed to make a home page. I think that I can finish other 
pages in 1-2 weeks, depending on my workload.



Current version: https://pulsar.apache.org/

New version: https://pulsar-site-three.vercel.app



Draft PR: https://github.com/apache/pulsar-site/pull/426



After the redesign, I plan to add a few more pages.



Should I continue?

Re: [VOTE][PIP-242] Topic name restriction

[PengHui Li]

+1(binding)

Thanks,
Penghui

On Mon, Feb 20, 2023 at 11:54 AM Cong Zhao  wrote:

> +1 (non-binding)
>
> Thanks,
> Cong
>
> On 2023/02/18 08:58:26 mattisonc...@gmail.com wrote:
> > Hi, All
> >
> > After a fascinating discussion, I would start the vote of PIP-242.
> >
> > We have chosen to drop out the `system topic` related improvement to
> another PIP. Therefore, the current version is simple enough and it has a
> clear boundary.
> >
> > Please leave +1/-1 in this thread to join the vote. and feel free to
> leave any concerns.
> >
> > Thanks to you guys.
> >
> > Best,
> > Mattison
> >
> > References:
> >
> > • PIP https://github.com/apache/pulsar/issues/19239
> > • Discussion
> https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
> >
> >
> >
>