[RFR] Issue triaging contribution guide
Hi, I squashed over 1000 stale issues and PRs in Dec. 2022. During the work, I notice that the community may lack some written guides for issue triaging, and even if a contributor is willing to help, he/she doesn't know what to do but just leave it as is. I wrote a draft for this case[1]. It encourages contributors to do basic classification and encourages committers to close issues as not planned when it's the case. Moving forward a valid issue requires knowledge and time, but by sharing & aligning issue triage patterns, we can reduce a few engineering loss. Best, tison. [1] https://github.com/apache/pulsar-site/pull/485
[PROPOSAL] Create a Chinese (Q&A) category in GitHub Discussions forum
Hi, I see a continuously increasing trend of users asking questions on our GitHub Discussions forum[1]. Since a significant portion of our users come from China, and they are more expressive when discussing in Chinese, I suggest we create a Chinese category for Q&A only, which is expected to work as user-zh@pulsar.a.o but on the GitHub Discussions forum for being user-friendly. What do you think? Best, tison. [1] https://github.com/apache/pulsar/discussions
[DISCUSS] Cherry-pick #15121 into branch-2.10 to solve the issue sasl authentication failure
Hi community
### Summary
The Admin client (`pulsar-admin`) and Java Client (PulsarAdmin) will throw
Unauthorized Ex in both scenarios:
- If there have more than one broker in a cluster( see issue 1 below ).
- If authentication is enabled for both Pulsar-Proxy and Pulsar-Broker( see
issue 2 below),
```
bin/pulsar-admin topics stats persistent://public/default/tp1
2023-03-28T07:30:58,453+ [main] INFO
org.apache.pulsar.client.impl.auth.AuthenticationSasl - JAAS loginContext
is: PulsarAdmin.
2023-03-28T07:30:58,583+ [main] INFO
org.apache.pulsar.common.sasl.JAASCredentialsContainer - successfully
logged in.
2023-03-28T07:30:58,587+ [pulsar-tgt-refresh-thread] INFO
org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh thread started.
2023-03-28T07:30:58,612+ [pulsar-tgt-refresh-thread] INFO
org.apache.pulsar.common.sasl.TGTRefreshThread - Client principal is "
pulsar-ad...@sn.io".
2023-03-28T07:30:58,613+ [pulsar-tgt-refresh-thread] INFO
org.apache.pulsar.common.sasl.TGTRefreshThread - Server principal is
"krbtgt/sn...@sn.io".
2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
org.apache.pulsar.common.sasl.TGTRefreshThread - TGT valid starting at:
Tue Mar 28 07:30:58 UTC 2023
2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
org.apache.pulsar.common.sasl.TGTRefreshThread - TGT expires:
Wed Mar 29 07:30:58 UTC 2023
2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh sleeping
until: Wed Mar 29 03:12:29 UTC 2023
2023-03-28T07:30:59,861+ [main] INFO
org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using
JAAS/SASL/GSSAPI auth to connect to server Principal broker/pulsar03,
HTTP 401 Unauthorized
Reason: HTTP 401 Unauthorized
```
And I want to cherry-pick https://github.com/apache/pulsar/pull/15121 into
branch-2.10 to fix it.
### Background
When using Kerberos for authentication, Pulsar works like this:
- client: init ticket
- request to broker
- broker identifies the client (Broker can confirm the ticket is valid by
Kerberos)
- sends a token(we call it sasl_role_token) to the client ( at this moment,
the session is successfully created )
- then the client will be authenticated through sasl_role_token, do not use
Kerberos anymore.
The `sasl_role_token` is generated by this logic: `Sha512(saslRoleName,
${secret})`, we call the `secret` sasl_sign_secret.
In version `2.10.x`, the variable `secret` is a random string initialized
when the broker starts.
### Issue 1
If a cluster includes two brokers, and a topic `public/default/tp1` is
owned by broker-0. We will get an error when we call `pulsar-admin topics
stats public/default/tp1` to broker-1.
The whole process goes like this:
- client succeeds in authentication and gets a token from broker-1
- broker-1 tells the client to redirect to broker-0
- client request to broker-0 carries the sasl_role_token generated by
broker-1
- broker-0 can not decode the sasl_role_token, because it has differ secret
of broker-1, and responses 401
### Issue 2
After authentication is enabled for both Pulsar-Proxy and Pulsar-Broker,
the error occurs as follows
- client succeeds in authentication and gets a token from Pulsar Proxy
- proxy forwards the request to broker
- the broker can not decode the `sasl_role_token`, because it has differed
secret of Pulsar Proxy, and responses 401
### solutions
There have two solutions to solve this issue:
Solution 1
- The client saves different tokens for different servers(e.g. ["broker-0",
"broker-1", "pulsar-proxy"]) so servers will receive the tokens issued by
each other, then we can fix Issue 1.
- Proxy and Broker do not enable authentication simultaneously, then we can
fix Issue 2.
Solution 2
- Make `sasl_sign_secret` configurable. Users can configure this variable
to the same value, then multi servers can decode every
`sasl_role_token.` PR #15121 does this.
I'd prefer Solution 2 because it is already in the master branch, so I want
to cherry-pick #15121 into branch-2.10.
### Forward Compatibility
In PR #15121, the config `sasl_sign_secret` is a new item in config files.
Since it is required, users will get a system error if does not set it. To
ensure forward compatibility, we can make this variable optional in
branch-2.10
Thanks
Yubiao Feng
Re: [DISCUSS] Cherry-pick #15121 into branch-2.10 to solve the issue sasl authentication failure
I agree with cherry-picking PR 15121 to branch-2.10 and keep compatibility.
Best,
Mattison
On Mar 28, 2023, 19:52 +0800, Yubiao Feng
, wrote:
> Hi community
>
> ### Summary
> The Admin client (`pulsar-admin`) and Java Client (PulsarAdmin) will throw
> Unauthorized Ex in both scenarios:
> - If there have more than one broker in a cluster( see issue 1 below ).
> - If authentication is enabled for both Pulsar-Proxy and Pulsar-Broker( see
> issue 2 below),
>
> ```
> bin/pulsar-admin topics stats persistent://public/default/tp1
> 2023-03-28T07:30:58,453+ [main] INFO
> org.apache.pulsar.client.impl.auth.AuthenticationSasl - JAAS loginContext
> is: PulsarAdmin.
> 2023-03-28T07:30:58,583+ [main] INFO
> org.apache.pulsar.common.sasl.JAASCredentialsContainer - successfully
> logged in.
> 2023-03-28T07:30:58,587+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh thread started.
> 2023-03-28T07:30:58,612+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - Client principal is "
> pulsar-ad...@sn.io".
> 2023-03-28T07:30:58,613+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - Server principal is
> "krbtgt/sn...@sn.io".
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT valid starting at:
> Tue Mar 28 07:30:58 UTC 2023
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT expires:
> Wed Mar 29 07:30:58 UTC 2023
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh sleeping
> until: Wed Mar 29 03:12:29 UTC 2023
> 2023-03-28T07:30:59,861+ [main] INFO
> org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using
> JAAS/SASL/GSSAPI auth to connect to server Principal broker/pulsar03,
> HTTP 401 Unauthorized
> Reason: HTTP 401 Unauthorized
> ```
>
> And I want to cherry-pick https://github.com/apache/pulsar/pull/15121 into
> branch-2.10 to fix it.
>
> ### Background
> When using Kerberos for authentication, Pulsar works like this:
> - client: init ticket
> - request to broker
> - broker identifies the client (Broker can confirm the ticket is valid by
> Kerberos)
> - sends a token(we call it sasl_role_token) to the client ( at this moment,
> the session is successfully created )
> - then the client will be authenticated through sasl_role_token, do not use
> Kerberos anymore.
>
> The `sasl_role_token` is generated by this logic: `Sha512(saslRoleName,
> ${secret})`, we call the `secret` sasl_sign_secret.
> In version `2.10.x`, the variable `secret` is a random string initialized
> when the broker starts.
>
> ### Issue 1
> If a cluster includes two brokers, and a topic `public/default/tp1` is
> owned by broker-0. We will get an error when we call `pulsar-admin topics
> stats public/default/tp1` to broker-1.
>
> The whole process goes like this:
> - client succeeds in authentication and gets a token from broker-1
> - broker-1 tells the client to redirect to broker-0
> - client request to broker-0 carries the sasl_role_token generated by
> broker-1
> - broker-0 can not decode the sasl_role_token, because it has differ secret
> of broker-1, and responses 401
>
> ### Issue 2
> After authentication is enabled for both Pulsar-Proxy and Pulsar-Broker,
> the error occurs as follows
> - client succeeds in authentication and gets a token from Pulsar Proxy
> - proxy forwards the request to broker
> - the broker can not decode the `sasl_role_token`, because it has differed
> secret of Pulsar Proxy, and responses 401
>
> ### solutions
> There have two solutions to solve this issue:
>
> Solution 1
> - The client saves different tokens for different servers(e.g. ["broker-0",
> "broker-1", "pulsar-proxy"]) so servers will receive the tokens issued by
> each other, then we can fix Issue 1.
> - Proxy and Broker do not enable authentication simultaneously, then we can
> fix Issue 2.
>
> Solution 2
> - Make `sasl_sign_secret` configurable. Users can configure this variable
> to the same value, then multi servers can decode every
> `sasl_role_token.` PR #15121 does this.
>
> I'd prefer Solution 2 because it is already in the master branch, so I want
> to cherry-pick #15121 into branch-2.10.
>
> ### Forward Compatibility
> In PR #15121, the config `sasl_sign_secret` is a new item in config files.
> Since it is required, users will get a system error if does not set it. To
> ensure forward compatibility, we can make this variable optional in
> branch-2.10
>
>
> Thanks
> Yubiao Feng
Re: [DISCUSS] Cherry-pick #15121 into branch-2.10 to solve the issue sasl authentication failure
I agree with cherry-picking PR 15121 to branch-2.10 and keep compatibility.
Best,
Dezhi
On 2023/03/28 11:52:06 Yubiao Feng wrote:
> Hi community
>
> ### Summary
> The Admin client (`pulsar-admin`) and Java Client (PulsarAdmin) will throw
> Unauthorized Ex in both scenarios:
> - If there have more than one broker in a cluster( see issue 1 below ).
> - If authentication is enabled for both Pulsar-Proxy and Pulsar-Broker( see
> issue 2 below),
>
> ```
> bin/pulsar-admin topics stats persistent://public/default/tp1
> 2023-03-28T07:30:58,453+ [main] INFO
> org.apache.pulsar.client.impl.auth.AuthenticationSasl - JAAS loginContext
> is: PulsarAdmin.
> 2023-03-28T07:30:58,583+ [main] INFO
> org.apache.pulsar.common.sasl.JAASCredentialsContainer - successfully
> logged in.
> 2023-03-28T07:30:58,587+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh thread started.
> 2023-03-28T07:30:58,612+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - Client principal is "
> pulsar-ad...@sn.io".
> 2023-03-28T07:30:58,613+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - Server principal is
> "krbtgt/sn...@sn.io".
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT valid starting at:
> Tue Mar 28 07:30:58 UTC 2023
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT expires:
> Wed Mar 29 07:30:58 UTC 2023
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh sleeping
> until: Wed Mar 29 03:12:29 UTC 2023
> 2023-03-28T07:30:59,861+ [main] INFO
> org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using
> JAAS/SASL/GSSAPI auth to connect to server Principal broker/pulsar03,
> HTTP 401 Unauthorized
> Reason: HTTP 401 Unauthorized
> ```
>
> And I want to cherry-pick https://github.com/apache/pulsar/pull/15121 into
> branch-2.10 to fix it.
>
> ### Background
> When using Kerberos for authentication, Pulsar works like this:
> - client: init ticket
> - request to broker
> - broker identifies the client (Broker can confirm the ticket is valid by
> Kerberos)
> - sends a token(we call it sasl_role_token) to the client ( at this moment,
> the session is successfully created )
> - then the client will be authenticated through sasl_role_token, do not use
> Kerberos anymore.
>
> The `sasl_role_token` is generated by this logic: `Sha512(saslRoleName,
> ${secret})`, we call the `secret` sasl_sign_secret.
> In version `2.10.x`, the variable `secret` is a random string initialized
> when the broker starts.
>
> ### Issue 1
> If a cluster includes two brokers, and a topic `public/default/tp1` is
> owned by broker-0. We will get an error when we call `pulsar-admin topics
> stats public/default/tp1` to broker-1.
>
> The whole process goes like this:
> - client succeeds in authentication and gets a token from broker-1
> - broker-1 tells the client to redirect to broker-0
> - client request to broker-0 carries the sasl_role_token generated by
> broker-1
> - broker-0 can not decode the sasl_role_token, because it has differ secret
> of broker-1, and responses 401
>
> ### Issue 2
> After authentication is enabled for both Pulsar-Proxy and Pulsar-Broker,
> the error occurs as follows
> - client succeeds in authentication and gets a token from Pulsar Proxy
> - proxy forwards the request to broker
> - the broker can not decode the `sasl_role_token`, because it has differed
> secret of Pulsar Proxy, and responses 401
>
> ### solutions
> There have two solutions to solve this issue:
>
> Solution 1
> - The client saves different tokens for different servers(e.g. ["broker-0",
> "broker-1", "pulsar-proxy"]) so servers will receive the tokens issued by
> each other, then we can fix Issue 1.
> - Proxy and Broker do not enable authentication simultaneously, then we can
> fix Issue 2.
>
> Solution 2
> - Make `sasl_sign_secret` configurable. Users can configure this variable
> to the same value, then multi servers can decode every
> `sasl_role_token.` PR #15121 does this.
>
> I'd prefer Solution 2 because it is already in the master branch, so I want
> to cherry-pick #15121 into branch-2.10.
>
> ### Forward Compatibility
> In PR #15121, the config `sasl_sign_secret` is a new item in config files.
> Since it is required, users will get a system error if does not set it. To
> ensure forward compatibility, we can make this variable optional in
> branch-2.10
>
>
> Thanks
> Yubiao Feng
>
Re: [VOTE] Pulsar Release 2.9.5 Candidate 2
+1 (binding) - Checked the signature - Start standalone - Publish and consume messages - Verified Function and State Function - Verified Cassandra connector - Build from the source package Regards Jiwei Guo (Tboy) On Tue, Mar 28, 2023 at 2:34 PM 丛搏 wrote: > > +1 (binding) > > os: mac 12.3.1, Intel > java: OpenJDK 17.0.1 > > - Checked the signature > - Checked LICENSE > - Start standalone > - Publish and consume messages > - Verified Function and State Function > - Verified Cassandra connector > - Build from the source package (maven 3.8.6, openJDK 17.0.1) > - Run a simple transaction check > > Thanks, > Bo > > Cong Zhao 于2023年3月27日周一 23:09写道： > > > > This is the third release candidate for Apache Pulsar, version 2.9.5. > > > > This release contains 103 commits by 30 contributors. > > https://github.com/apache/pulsar/compare/v2.9.4...v2.9.5-candidate-2 > > > > *** Please download, test, and vote on this release. This vote will stay > > open > > for at least 72 hours *** > > > > Note that we are voting upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.5-candidate-2/ > > > > SHA-512 checksums: > > 5e1d0c1b38441cdcb36a2f4e59ab9755b39a5c4a0136e078e91ab9bc2169016f195268692cafd6f13a45248dba2e97959b41f3cfbc8659e3cbd0bade0c954998 > > apache-pulsar-2.9.5-bin.tar.gz > > > > 72c9f47005636c6e629dd5117b15fdc13bfd9c7efe107be77a9d55b7dfcdda2f941003eb120ea8beeffe44c41bc41c385a2e5a9cb6540d2fe83a6d04ea53a7389d > > apache-pulsar-2.9.5-src.tar.gz > > > > 73d286af64e189cf91c0511d360d98371b7ade1eec67bc6acf6ff766784e9e40388d3da8ae99a206369feaf398b254fff36e2206077041c37b8055ee7edde86eea > > apache-pulsar-offloaders-2.9.5-bin.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1222/ > > > > The tag to be voted upon: > > v2.9.5-candidate-2 (c75c811ee48f51cf74f399f5b364bc1527186b34) > > https://github.com/apache/pulsar/releases/tag/v2.9.5-candidate-2 > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > https://downloads.apache.org/pulsar/KEYS > > > > Docker images: > > > > > > https://hub.docker.com/layers/czcoder/pulsar/2.9.5/images/sha256-c6d3435d5699cb3697ee2ddc4f8a45e0ac5e35d8aefd557e280b7cf91366b981?context=explore > > > > > > https://hub.docker.com/layers/czcoder/pulsar-all/2.9.5/images/sha256-a09a8e177ca7856c29dc8b9828cd293c6d44b473add20d1877bb3137b94a20c5?context=explore > > > > > > https://hub.docker.com/layers/czcoder/pulsar-grafana/2.9.5/images/sha256-c43a489c65cf6c407d6c3be6fc7a001227805b1aaa9413115cf55ba11a1e329f?context=explore > > > > Please download the source package, and follow the README to build > > and run the Pulsar standalone service. > > > > Thanks > > Cong Zhao
Re: [VOTE] PIP-257: Add Open ID Connect Support to Server Components
+1 Devin G. Bost On Tue, Mar 28, 2023 at 1:57 AM Enrico Olivelli wrote: > +1 (binding) > > Enrico > > Il giorno mar 28 mar 2023 alle ore 04:59 Dave Fisher > ha scritto: > > > > +1 (binding) > > > > Sent from my iPhone > > > > > On Mar 27, 2023, at 12:59 PM, Lari Hotari wrote: > > > > > > +1, binding > > > > > > -Lari > > > > > >> On Mon, 27 Mar 2023, 22.37 Michael Marshall, > wrote: > > >> > > >> Hi Pulsar Community, > > >> > > >> This thread is to start the vote for PIP 257. > > >> > > >> Discussion: > > >> https://lists.apache.org/thread/ttdh74t6v5nznmtw436otk8qdymjvrfk > > >> Issue: https://github.com/apache/pulsar/issues/19771 > > >> Work in Progress Implementations: > > >> - https://github.com/apache/pulsar/pull/19849 > > >> - https://github.com/apache/pulsar/pull/19888 > > >> > > >> The implementation still has some minor details to iron out, but I > > >> don't think those are a large enough issue to delay the vote any > > >> longer. If you feel otherwise, please let me know and continue the > > >> discussion on the discussion thread. > > >> > > >> Thanks, > > >> Michael > > >> > > >
[VOTE] Pulsar Release 2.9.5 Candidate 2
This is the third release candidate for Apache Pulsar, version 2.9.5. This release contains 103 commits by 30 contributors. https://github.com/apache/pulsar/compare/v2.9.4...v2.9.5-candidate-2 *** Please download, test, and vote on this release. This vote will stay open for at least 72 hours *** Note that we are voting upon the source (tag), binaries are provided for convenience. Source and binary files: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.5-candidate-2/ SHA-512 checksums: 5e1d0c1b38441cdcb36a2f4e59ab9755b39a5c4a0136e078e91ab9bc2169016f195268692cafd6f13a45248dba2e97959b41f3cfbc8659e3cbd0bade0c954998 apache-pulsar-2.9.5-bin.tar.gz 72c9f47005636c6e629dd5117b15fdc13bfd9c7efe107be77a9d55b7dfcdda2f941003eb120ea8beeffe44c41bc41c385a2e5a9cb6540d2fe83a6d04ea53a7389d apache-pulsar-2.9.5-src.tar.gz 73d286af64e189cf91c0511d360d98371b7ade1eec67bc6acf6ff766784e9e40388d3da8ae99a206369feaf398b254fff36e2206077041c37b8055ee7edde86eea apache-pulsar-offloaders-2.9.5-bin.tar.gz Maven staging repo: https://repository.apache.org/content/repositories/orgapachepulsar-1222/ The tag to be voted upon: v2.9.5-candidate-2 (c75c811ee48f51cf74f399f5b364bc1527186b34) https://github.com/apache/pulsar/releases/tag/v2.9.5-candidate-2 Pulsar's KEYS file containing PGP keys you use to sign the release: https://downloads.apache.org/pulsar/KEYS Docker images: https://hub.docker.com/layers/czcoder/pulsar/2.9.5/images/sha256-c6d3435d5699cb3697ee2ddc4f8a45e0ac5e35d8aefd557e280b7cf91366b981?context=explore https://hub.docker.com/layers/czcoder/pulsar-all/2.9.5/images/sha256-a09a8e177ca7856c29dc8b9828cd293c6d44b473add20d1877bb3137b94a20c5?context=explore https://hub.docker.com/layers/czcoder/pulsar-grafana/2.9.5/images/sha256-c43a489c65cf6c407d6c3be6fc7a001227805b1aaa9413115cf55ba11a1e329f?context=explore Please download the source package, and follow the README to build and run the Pulsar standalone service. Thanks Cong Zhao
Re: [DISCUSS] Cherry-pick #15121 into branch-2.10 to solve the issue sasl authentication failure
Looks good to me to make it optional in branch-2.10 since we don't want to
introduce any break behaviors in the subsequent patch releases.
Thanks,
Penghui
On Tue, Mar 28, 2023 at 9:39 PM Dezhi Liu wrote:
> I agree with cherry-picking PR 15121 to branch-2.10 and keep compatibility.
>
>
> Best,
> Dezhi
>
> On 2023/03/28 11:52:06 Yubiao Feng wrote:
> > Hi community
> >
> > ### Summary
> > The Admin client (`pulsar-admin`) and Java Client (PulsarAdmin) will
> throw
> > Unauthorized Ex in both scenarios:
> > - If there have more than one broker in a cluster( see issue 1 below ).
> > - If authentication is enabled for both Pulsar-Proxy and Pulsar-Broker(
> see
> > issue 2 below),
> >
> > ```
> > bin/pulsar-admin topics stats persistent://public/default/tp1
> > 2023-03-28T07:30:58,453+ [main] INFO
> > org.apache.pulsar.client.impl.auth.AuthenticationSasl - JAAS loginContext
> > is: PulsarAdmin.
> > 2023-03-28T07:30:58,583+ [main] INFO
> > org.apache.pulsar.common.sasl.JAASCredentialsContainer - successfully
> > logged in.
> > 2023-03-28T07:30:58,587+ [pulsar-tgt-refresh-thread] INFO
> > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh thread
> started.
> > 2023-03-28T07:30:58,612+ [pulsar-tgt-refresh-thread] INFO
> > org.apache.pulsar.common.sasl.TGTRefreshThread - Client principal is "
> > pulsar-ad...@sn.io".
> > 2023-03-28T07:30:58,613+ [pulsar-tgt-refresh-thread] INFO
> > org.apache.pulsar.common.sasl.TGTRefreshThread - Server principal is
> > "krbtgt/sn...@sn.io".
> > 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT valid starting at:
> > Tue Mar 28 07:30:58 UTC 2023
> > 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT expires:
> > Wed Mar 29 07:30:58 UTC 2023
> > 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh sleeping
> > until: Wed Mar 29 03:12:29 UTC 2023
> > 2023-03-28T07:30:59,861+ [main] INFO
> > org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using
> > JAAS/SASL/GSSAPI auth to connect to server Principal broker/pulsar03,
> > HTTP 401 Unauthorized
> > Reason: HTTP 401 Unauthorized
> > ```
> >
> > And I want to cherry-pick https://github.com/apache/pulsar/pull/15121
> into
> > branch-2.10 to fix it.
> >
> > ### Background
> > When using Kerberos for authentication, Pulsar works like this:
> > - client: init ticket
> > - request to broker
> > - broker identifies the client (Broker can confirm the ticket is valid by
> > Kerberos)
> > - sends a token(we call it sasl_role_token) to the client ( at this
> moment,
> > the session is successfully created )
> > - then the client will be authenticated through sasl_role_token, do not
> use
> > Kerberos anymore.
> >
> > The `sasl_role_token` is generated by this logic: `Sha512(saslRoleName,
> > ${secret})`, we call the `secret` sasl_sign_secret.
> > In version `2.10.x`, the variable `secret` is a random string initialized
> > when the broker starts.
> >
> > ### Issue 1
> > If a cluster includes two brokers, and a topic `public/default/tp1` is
> > owned by broker-0. We will get an error when we call `pulsar-admin topics
> > stats public/default/tp1` to broker-1.
> >
> > The whole process goes like this:
> > - client succeeds in authentication and gets a token from broker-1
> > - broker-1 tells the client to redirect to broker-0
> > - client request to broker-0 carries the sasl_role_token generated by
> > broker-1
> > - broker-0 can not decode the sasl_role_token, because it has differ
> secret
> > of broker-1, and responses 401
> >
> > ### Issue 2
> > After authentication is enabled for both Pulsar-Proxy and Pulsar-Broker,
> > the error occurs as follows
> > - client succeeds in authentication and gets a token from Pulsar Proxy
> > - proxy forwards the request to broker
> > - the broker can not decode the `sasl_role_token`, because it has
> differed
> > secret of Pulsar Proxy, and responses 401
> >
> > ### solutions
> > There have two solutions to solve this issue:
> >
> > Solution 1
> > - The client saves different tokens for different servers(e.g.
> ["broker-0",
> > "broker-1", "pulsar-proxy"]) so servers will receive the tokens issued by
> > each other, then we can fix Issue 1.
> > - Proxy and Broker do not enable authentication simultaneously, then we
> can
> > fix Issue 2.
> >
> > Solution 2
> > - Make `sasl_sign_secret` configurable. Users can configure this variable
> > to the same value, then multi servers can decode every
> > `sasl_role_token.` PR #15121 does this.
> >
> > I'd prefer Solution 2 because it is already in the master branch, so I
> want
> > to cherry-pick #15121 into branch-2.10.
> >
> > ### Forward Compatibility
> > In PR #15121, the config `sasl_sign_secret` is a new item in config
> files.
> > Since it is required, users will get a system error if does not set it.
> To
>
Re: [DISCUSS] Cherry-pick #15121 into branch-2.10 to solve the issue sasl authentication failure
+1 (Solution 2)
Thanks,
Bo
PengHui Li 于2023年3月29日周三 09:47写道：
>
> Looks good to me to make it optional in branch-2.10 since we don't want to
> introduce any break behaviors in the subsequent patch releases.
>
> Thanks,
> Penghui
>
> On Tue, Mar 28, 2023 at 9:39 PM Dezhi Liu wrote:
>
> > I agree with cherry-picking PR 15121 to branch-2.10 and keep compatibility.
> >
> >
> > Best,
> > Dezhi
> >
> > On 2023/03/28 11:52:06 Yubiao Feng wrote:
> > > Hi community
> > >
> > > ### Summary
> > > The Admin client (`pulsar-admin`) and Java Client (PulsarAdmin) will
> > throw
> > > Unauthorized Ex in both scenarios:
> > > - If there have more than one broker in a cluster( see issue 1 below ).
> > > - If authentication is enabled for both Pulsar-Proxy and Pulsar-Broker(
> > see
> > > issue 2 below),
> > >
> > > ```
> > > bin/pulsar-admin topics stats persistent://public/default/tp1
> > > 2023-03-28T07:30:58,453+ [main] INFO
> > > org.apache.pulsar.client.impl.auth.AuthenticationSasl - JAAS loginContext
> > > is: PulsarAdmin.
> > > 2023-03-28T07:30:58,583+ [main] INFO
> > > org.apache.pulsar.common.sasl.JAASCredentialsContainer - successfully
> > > logged in.
> > > 2023-03-28T07:30:58,587+ [pulsar-tgt-refresh-thread] INFO
> > > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh thread
> > started.
> > > 2023-03-28T07:30:58,612+ [pulsar-tgt-refresh-thread] INFO
> > > org.apache.pulsar.common.sasl.TGTRefreshThread - Client principal is "
> > > pulsar-ad...@sn.io".
> > > 2023-03-28T07:30:58,613+ [pulsar-tgt-refresh-thread] INFO
> > > org.apache.pulsar.common.sasl.TGTRefreshThread - Server principal is
> > > "krbtgt/sn...@sn.io".
> > > 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> > > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT valid starting at:
> > > Tue Mar 28 07:30:58 UTC 2023
> > > 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> > > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT expires:
> > > Wed Mar 29 07:30:58 UTC 2023
> > > 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> > > org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh sleeping
> > > until: Wed Mar 29 03:12:29 UTC 2023
> > > 2023-03-28T07:30:59,861+ [main] INFO
> > > org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using
> > > JAAS/SASL/GSSAPI auth to connect to server Principal broker/pulsar03,
> > > HTTP 401 Unauthorized
> > > Reason: HTTP 401 Unauthorized
> > > ```
> > >
> > > And I want to cherry-pick https://github.com/apache/pulsar/pull/15121
> > into
> > > branch-2.10 to fix it.
> > >
> > > ### Background
> > > When using Kerberos for authentication, Pulsar works like this:
> > > - client: init ticket
> > > - request to broker
> > > - broker identifies the client (Broker can confirm the ticket is valid by
> > > Kerberos)
> > > - sends a token(we call it sasl_role_token) to the client ( at this
> > moment,
> > > the session is successfully created )
> > > - then the client will be authenticated through sasl_role_token, do not
> > use
> > > Kerberos anymore.
> > >
> > > The `sasl_role_token` is generated by this logic: `Sha512(saslRoleName,
> > > ${secret})`, we call the `secret` sasl_sign_secret.
> > > In version `2.10.x`, the variable `secret` is a random string initialized
> > > when the broker starts.
> > >
> > > ### Issue 1
> > > If a cluster includes two brokers, and a topic `public/default/tp1` is
> > > owned by broker-0. We will get an error when we call `pulsar-admin topics
> > > stats public/default/tp1` to broker-1.
> > >
> > > The whole process goes like this:
> > > - client succeeds in authentication and gets a token from broker-1
> > > - broker-1 tells the client to redirect to broker-0
> > > - client request to broker-0 carries the sasl_role_token generated by
> > > broker-1
> > > - broker-0 can not decode the sasl_role_token, because it has differ
> > secret
> > > of broker-1, and responses 401
> > >
> > > ### Issue 2
> > > After authentication is enabled for both Pulsar-Proxy and Pulsar-Broker,
> > > the error occurs as follows
> > > - client succeeds in authentication and gets a token from Pulsar Proxy
> > > - proxy forwards the request to broker
> > > - the broker can not decode the `sasl_role_token`, because it has
> > differed
> > > secret of Pulsar Proxy, and responses 401
> > >
> > > ### solutions
> > > There have two solutions to solve this issue:
> > >
> > > Solution 1
> > > - The client saves different tokens for different servers(e.g.
> > ["broker-0",
> > > "broker-1", "pulsar-proxy"]) so servers will receive the tokens issued by
> > > each other, then we can fix Issue 1.
> > > - Proxy and Broker do not enable authentication simultaneously, then we
> > can
> > > fix Issue 2.
> > >
> > > Solution 2
> > > - Make `sasl_sign_secret` configurable. Users can configure this variable
> > > to the same value, then multi servers can decode every
> > > `sasl_role_token.` PR #15121 does this.
> > >
> > > I'd prefer S
[ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Dear Community, We are thrilled to announce that Qiang Zhao (https://github.com/mattisonchao) has been invited and has accepted the role of member of the Apache Pulsar Project Management Committee (PMC). Qiang has been a vital asset to our community, consistently demonstrating his dedication and active participation through significant contributions. In addition to his technical contributions, Qiang also plays an important role in reviewing pull requests and ensuring the overall quality of our project. We look forward to his continued contributions. On behalf of the Pulsar PMC, we extend a warm welcome and congratulations to Qiang Zhao. Best regards Jiwei
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congrats! Best, tison. guo jiwei 于2023年3月29日周三 11:22写道： > Dear Community, > > We are thrilled to announce that Qiang Zhao > (https://github.com/mattisonchao) has been invited and has accepted the > role of member of the Apache Pulsar Project Management Committee (PMC). > > Qiang has been a vital asset to our community, consistently > demonstrating his dedication and active participation through > significant contributions. In addition to his technical contributions, > Qiang also plays an important role in reviewing pull requests and > ensuring the overall quality of our project. We look forward to his > continued contributions. > > On behalf of the Pulsar PMC, we extend a warm welcome and > congratulations to Qiang Zhao. > > Best regards > Jiwei >
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congrats! > 2023年3月29日 11:24，tison 写道： > > Congrats!
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congrats! Qiang. Thanks, Baodi Shi 在 2023年3月29日 11:27:58 上，Horizon <1060026...@qq.com.invalid> 写道： > Congrats! > > 2023年3月29日 11:24，tison 写道： > > > Congrats! > > >
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congrats! Thanks, Bo Baodi Shi 于2023年3月29日周三 11:29写道： > > Congrats! Qiang. > > > Thanks, > Baodi Shi > > > 在 2023年3月29日 11:27:58 上，Horizon <1060026...@qq.com.invalid> 写道： > > > Congrats! > > > > 2023年3月29日 11:24，tison 写道： > > > > > > Congrats! > > > > > >
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congratulations! Best, Dave Sent from my iPhone > On Mar 28, 2023, at 8:29 PM, Baodi Shi wrote: > > Congrats! Qiang. > > > Thanks, > Baodi Shi > > >> 在 2023年3月29日 11:27:58 上，Horizon <1060026...@qq.com.invalid> 写道： >> >> Congrats! >> >> 2023年3月29日 11:24，tison 写道： >> >> >> Congrats! >> >> >>
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congrats! Qiang. Thanks Yubiao On Wed, Mar 29, 2023 at 11:22 AM guo jiwei wrote: > Dear Community, > > We are thrilled to announce that Qiang Zhao > (https://github.com/mattisonchao) has been invited and has accepted the > role of member of the Apache Pulsar Project Management Committee (PMC). > > Qiang has been a vital asset to our community, consistently > demonstrating his dedication and active participation through > significant contributions. In addition to his technical contributions, > Qiang also plays an important role in reviewing pull requests and > ensuring the overall quality of our project. We look forward to his > continued contributions. > > On behalf of the Pulsar PMC, we extend a warm welcome and > congratulations to Qiang Zhao. > > Best regards > Jiwei >
Re: [DISCUSS] PIP-260: Client consumer filter received messages
Hi, all :
Thanks to everyone who discussed it.
Our current care points include the following aspects:
1. The filtering efficiency of the client consumer is not as
good as doing something directly in startMessageId
2. Does not support reset cursor
Because my previous PIP description is to add configuration
in consumerBuilder. The definition of this configuration is not
clear, and it will cause great trouble to users.
We can add a separate configuration that is only used for
acks with transactions. Simple example:
```
ConsumerBuilder transactionConfiguration(ConsumerTransactionConfiguration);
@Builder
@Data
@NoArgsConstructor
@AllArgsConstructor
@InterfaceAudience.Public
@InterfaceStability.Stable
public class ConsumerTransactionConfiguration {
boolean isFilterReceivedMessagesEnabled = false;
}
```
if the design of startMessageId can provide the feature,
we can remove the configuration, or currently has a startMessageId
closed loop solution, I agree to use startMessageId.
As for the reset cursor, I think it is another problem,
not related to this PIP.
Thanks,
Bo
丛搏 于2023年3月24日周五 18:53写道：
>
> Hi, Michael:
>
> I thought about it carefully, and using 'startMessageId'
> is indeed a good idea. But it is more complicated, we
> need to ensure its absolute correctness, and take
> performance into consideration. If you can come up
> with a closed-loop solution based on 'startMessageId',
> I support you. If it can't take into account performance
> and correctness, I think we will make a combination of
> our two solutions. You are responsible for ensuring that
> a certain degree of messages are not re-delivered, which
> reduces the overhead caused by the repeated delivery
> of many messages. My design is responsible for
> the final consistency.
>
> Thanks,
> Bo
>
> Michael Marshall 于2023年3月22日周三 14:22写道：
> >
> > Because we already send the `startMessageId`, there is a chance where
> > we might not even need to update the protocol for the
> > CommandSubscribe. In light of that, I quickly put together a PR
> > showing how that field might be used to inform the broker where to
> > start the read position for the cursor.
> >
> > https://github.com/apache/pulsar/pull/19892
> >
> > The PR is not complete, but it does convey the general idea. I wrote
> > additional details in the draft's description.
> >
> > Thanks,
> > Michael
> >
> > On Tue, Mar 21, 2023 at 11:31 PM Michael Marshall
> > wrote:
> > >
> > > I am not following your objections to the protocol solution. It might
> > > be more productive if I provided a draft PR with a sample
> > > implementation. I'm not sure that I'll have time, but I'll try to put
> > > something together this week.
> > >
> > > > At least it will simplify the process of using cumulative ack with the
> > > > transaction.
> > >
> > > Is this the underlying motivation for the PIP?
> > >
> > > From my perspective, the PIP is seeking to decrease duplicate messages
> > > experienced due to disconnections from the broker.
> > >
> > > > The problem of the resetting cursor can be optimized in the future
> > >
> > > Why should we push off solving this problem? It seems fundamental to
> > > this PIP and should not be ignored. At the very least, I think we need
> > > to have an idea of what the future solution would be before we defer
> > > its implementation.
> > >
> > > Thanks,
> > > Michael
> > >
> > >
> > > On Tue, Mar 21, 2023 at 10:52 PM 丛搏 wrote:
> > > >
> > > > Hi, Michael
> > > > > In this case, the consumer does not have the source of truth for the
> > > > > readPosition. It would leave the new protocol field for `readPosition`
> > > > > empty and the broker would use its source of truth for the read
> > > > > position.
> > > > application has received all the messages by application thread. we
> > > > also need a
> > > > correct `startPosition`, right? but in your way, we will think about
> > > > the consumer
> > > > hasn't received any messages.
> > > >
> > > > >
> > > > > > why do we need to invoke `BlockingQueue.take` and `synchronized` in
> > > > > > the
> > > > > > same logic? it's a bad code.
> > > > >
> > > > > We don't need to synchronize this code here because the logic will
> > > > > come after the consumer has been disconnected from broker a and before
> > > > > it is connected to broker b.
> > > > The application takes a message from the queue then reconnect,
> > > > the SubCommond can use the right startPostion? example:
> > > > 1. application receives one message with `MessageId = 1`
> > > > 2. consumer reconnect discovers the queue is empty, and the
> > > > lastDequeMessageId doesn't change.
> > > > 3. consumer sends a subcommand with MessageId.earliest, the `MessageId
> > > > = 1`
> > > > will redeliver from broker to client consumer, right?
> > > >
> > > > As we can see in the example, the application also can receive
> > > > `MessageId = 1`, right?
> > > > > We would not need to lock here because we do not enqueue new messages
> > > > > after we've be
Re: [RFR] Issue triaging contribution guide
Hi tison, Thanks for raising this up! Just my $2: ==Guide Structure== Suggest optimizing the guide structure with the EPPO and 5W2H methods to clarify info and improve readability. For example, ## What is issue triaging? ... ## Why is issue triaging beneficial? ... ## How to triage issues? (a step-by-step flow) ... ## References ### Label instructions ### Issue triage checklist ... ## Related topics ... ==Issue Labels Instructions== To instruct users how to use issue labels, does it make sense to add explanations for them? Currently, we only have explanations for partial labels. [1]. The meaning of some labels is clear at 1st glance, while some are blur. For example, - `component/ML` What does this mean? - `lifecycle/stale`, `Stale` What are the differences between them? Now the issues labeled with them only show they haven't been updated for some time, but no more action is taken next step. Can we define it? - `help wanted` To help newbies find proper issues in minimal time, some communities use "good-first-issuse" while others use "help wanted". We can clarify this explicitly. ==New Issue Labels== Does it make sense to create new labels like "priority" or "severity"? For example, p0, p1, p2; s0, s1, s2. The pros (help identify and solve urgent or severe issues quickly) outweigh the cons (increase complexity a little). ==Lean Toward Closing== For the cases that can close issues, can we add more applicable scenarios to reduce triagers "guilties"? (since we cannot satisfy all users and need to keep the project maintainable) ==Re-Evaluating Closed Issues== Does it make sense to add some instructions for this? This happens at intervals because decisions might be adjusted based on additional info that surfaces later. ==Visuals== Suggest creating illustrations [2] to give users a holistic and bird-eye view of the whole workflow. A picture is worth a thousand words. == Yu [1] https://pulsar.apache.org/contribute/develop-labels/ [2] https://equiptowerhamlets.nhs.uk/wp-content/uploads/2020/06/Triage-process-overview.jpg On Tue, Mar 28, 2023 at 5:20 PM tison wrote: > Hi, > > I squashed over 1000 stale issues and PRs in Dec. 2022. During the work, I > notice that the community may lack some written guides for issue triaging, > and even if a contributor is willing to help, he/she doesn't know what to > do but just leave it as is. > > I wrote a draft for this case[1]. It encourages contributors to do basic > classification and encourages committers to close issues as not planned > when it's the case. > > Moving forward a valid issue requires knowledge and time, but by sharing & > aligning issue triage patterns, we can reduce a few engineering loss. > > Best, > tison. > > [1] https://github.com/apache/pulsar-site/pull/485 >
Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar
Congrats! Qiang. Sincerely, Xiangying On Wed, Mar 29, 2023 at 11:51 AM Yubiao Feng wrote: > Congrats! Qiang. > > Thanks > Yubiao > > On Wed, Mar 29, 2023 at 11:22 AM guo jiwei wrote: > > > Dear Community, > > > > We are thrilled to announce that Qiang Zhao > > (https://github.com/mattisonchao) has been invited and has accepted the > > role of member of the Apache Pulsar Project Management Committee (PMC). > > > > Qiang has been a vital asset to our community, consistently > > demonstrating his dedication and active participation through > > significant contributions. In addition to his technical contributions, > > Qiang also plays an important role in reviewing pull requests and > > ensuring the overall quality of our project. We look forward to his > > continued contributions. > > > > On behalf of the Pulsar PMC, we extend a warm welcome and > > congratulations to Qiang Zhao. > > > > Best regards > > Jiwei > > >
