Re: [VOTE] PIP-130: Apply redelivery backoff policy for ack timeout

[Ran Gao]

+1

Best,
Ran Gao

On 2022/01/04 03:34:33 PengHui Li wrote:
> This is the voting thread for PIP-130. It will stay open for at least 48
> hours.
> 
> https://github.com/apache/pulsar/issues/13528
> 
> Pasted below for quoting convenience.
> 
> -
> 
> PIP 130: Apply redelivery backoff policy for ack timeout
> 
> ## Motivation
> 
> PIP 106
> https://github.com/apache/pulsar/wiki/PIP-106%3A-Negative-acknowledgment-backoff
> introduced negative acknowledgment message redelivery backoff which allows
> users to achieve
> more flexible message redelivery delay time control. But the redelivery
> backoff policy only
> apply to the negative acknowledgment API, for users who use ack timeout to
> trigger the message
> redelivery, not the negative acknowledgment API, they can't use the new
> features introduced by
> PIP 106.
> 
> So the proposal is to apply the message redelivery policy for the ack
> timeout mechanism.
> Users can specify an ack timeout redelivery backoff, for example, apply an
> exponential backoff
> with 10 seconds ack timeout:
> 
> ```java
> client.newConsumer()
> .ackTimeout(10, TimeUnit.SECOND)
> .ackTimeoutRedeliveryBackoff(
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6).build());
> .subscribe();
> ```
> 
> The message redelivery behavior should be:
> 
> |  Redelivery count   | Redelivery delay  |
> |    |   |
> | 1 | 10 + 1 seconds |
> | 2 | 10 + 2 seconds |
> | 3 | 10 + 4 seconds |
> | 4 | 10 + 8 seconds |
> | 5 | 10 + 16 seconds |
> | 6 | 10 + 32 seconds |
> | 7 | 10 + 60 seconds |
> | 8 | 10 + 60 seconds |
> 
> ## Goal
> 
> Add an API to the Java Client to provide the ability to specify the ack
> timeout message redelivery
> backoff and the message redelivery behavior should abide by the redelivery
> backoff policy.
> 
> 
> ## API Changes
> 
> 1. Change `NegativeAckRedeliveryBackoff` to `RedeliveryBackoff`, so that we
> can use the
> MessageRedeliveryBackoff for both negative acknowledgment API and ack
> timeout message redelivery.
> 
> 2. Change `NegativeAckRedeliveryExponentialBackoff` to
> `ExponentialRedeliveryBackoff`, and add `multiplier`
> for `RedeliveryExponentialBackoff` with default value 2.
> 
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6)
> .multiplier(5)
> .build()
> 
> 3. Add `ackTimeoutRedeliveryBackoff` method for the `ConsumerBuilder`:
> 
> ```java
> client.newConsumer()
> .ackTimeout(10, TimeUnit.SECOND)
> .ackTimeoutRedeliveryBackoff(
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6).build());
> .subscribe();
> ```
> 
> ## Compatibility and migration plan
> 
> Since the negative acknowledgment message, redelivery backoff has not been
> released yet,
> so we can modify the API directly.
> 
> ## Tests plan
> 
> - Verify the introduced `multiplier` of ExponentialRedeliveryBackoff
> - Verify the ack timeout message redelivery work as expected
> 
> Regards,
> Penghui
> 

[DISCUSS] Apache Pulsar 2.9.2 release

[Ran Gao]

Hello, Pulsar community:

I'd like to propose that we release Apache Pulsar 2.9.2.

Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
to [0]), they contain the log4j security patch and many important fixes.

I am happy to volunteer to be the release manager.

I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
labeled release/2.9.2, I'll follow them to make sure important fix could be
merged in 2.9.2.


Best,
Ran Gao


[0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9

[VOTE] Pulsar Release 2.9.2 Candidate 1

[Ran Gao]

This is the first release candidate for Apache Pulsar, version 2.9.2.

It fixes the many issues and the log4j security problem.

*** Please download, test, and vote on this release. This vote will stay
open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-1/

SHA-512 checksums:

ccecec6ed53e9aea8882be0ee8d3e2dd353cde48033dad329d907580e2e472ac4b63e5467ccc5ec218ed7095a275526d78f7b9efb0dc467af4113848c2462673
 ./apache-pulsar-2.9.2-bin.tar.gz
ebc4422c7a293aaa2aa32cb1dbd0e7a96e1848ae0a6ad70fe14a672359704085e98a40d9f97af472306f4eba798615836485782f8975738a6771087f9dfb3cbe
 ./apache-pulsar-2.9.2-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1132/

The tag to be voted upon:
v2.9.2-candidate-1 (8a5d2253b888b3b865a2aedf635d672821c7)
https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-1

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [DISCUSS] Apache Pulsar 2.9.2 release

[Ran Gao]

Thanks for your reminder.

Currently, the new added PR will be added the label `branch-x.x`, the label 
indicate the PR should be added to which branch. After the PR is merged, the PR 
will be cherry-picked to the specific branch. After cherry-pick, a label 
`cherry-picked/branch-x.x` will be added.

I think this is a good way because it's hard to check all PRs before releasing 
a new version. Maybe we have a better choice, we could start a discussion.

On 2022/01/05 17:43:17 Enrico Olivelli wrote:
> BTW +1
> 
> I hope we will review carefully what we are including in the release
> 
> And that we will let branch-2.9 stabilise by not adding any other commits
> that are not strictly required:
> - security related issues
> - data loss/data corruption cases
> 
> 
> Enrico
> 
> 
> Thank you Ran Gao for starting this thread.
> 
> 
> 
> Enrico
> 
> Il Mer 5 Gen 2022, 17:29 Enrico Olivelli  ha scritto:
> 
> > 171 commits?
> >
> > Why?
> >
> > This is too much in my opinion for a point release.
> >
> > I am pretty sure that we don't need all that changes.
> >
> > Every change we cherry pick has a good chance to break the stability
> >
> > Enrico
> >
> > Il Mer 5 Gen 2022, 16:30 mattison chao  ha
> > scritto:
> >
> >> +1
> >>
> >> regards,
> >> Mattisonchao
> >>
> >> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
> >>
> >> > +1,Thanks for your work.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Best Regards,
> >> > Lan Liang
> >> > On 1/5/2022 23:01，Hang Chen wrote：
> >> > +1
> >> >
> >> > Best,
> >> > Hang
> >> >
> >> > 陳智弘  于2022年1月5日周三 21:30写道：
> >> >
> >> > +1
> >> >
> >> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
> >> >
> >> > +1
> >> >
> >> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> >> >
> >> > Hello, Pulsar community:
> >> >
> >> > I'd like to propose that we release Apache Pulsar 2.9.2.
> >> >
> >> > Currently, compared to 2.9.1, branch-2.9 already merged 171
> >> commits(refer
> >> > to [0]), they contain the log4j security patch and many important fixes.
> >> >
> >> > I am happy to volunteer to be the release manager.
> >> >
> >> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> >> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> >> > labeled release/2.9.2, I'll follow them to make sure important fix could
> >> > be
> >> > merged in 2.9.2.
> >> >
> >> >
> >> > Best,
> >> > Ran Gao
> >> >
> >> >
> >> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> >> >
> >> >
> >> >
> >>
> >
> 

Re: [DISCUSS] Apache Pulsar 2.9.2 release

[Ran Gao]

Sorry, late response, please take a look at the previous reply. If you have 
good ideas, we could start a discussion.

On 2022/01/05 18:31:38 Michael Marshall wrote:
> +1 - thanks for starting this thread.
> 
> >171 commits?
> > Why?
> > This is too much in my opinion for a point release.
> 
> I agree that this is a lot of commits, and especially since we only
> just released 2.9.1. Note though that this is not a new phenomenon:
> 2.8.2 had 251 commits on top of 2.8.1. [0]
> 
> > Every change we cherry pick has a good chance to break the stability
> 
> This is one of the challenges to our current cherry-pick based git
> workflow. It's hard to decide which commits to cherry-pick, and it
> requires a high level of effort to later verify the correctness of
> what was cherry-picked. This is one of the reasons I am interested in
> exploring a merge based git workflow.
> 
> Our project has grown a ton and will continue to grow. Our git process
> should prioritize the stability of our existing release branches.
> 
> > And that we will let branch-2.9 stabilise by not adding any other commits
> > that are not strictly required:
> > - security related issues
> > - data loss/data corruption cases
> 
> If we don't change our git workflow, we should at least add a
> cherry-picking guide to the wiki. We have added many new committers in
> the past year, but we don't provide them with much documented guidance
> on how to exercise their new rights.
> 
> Thanks,
> Michael
> 
> [0] https://github.com/apache/pulsar/compare/v2.8.1...v2.8.2
> 
> 
> On Wed, Jan 5, 2022 at 11:44 AM Enrico Olivelli  wrote:
> >
> > BTW +1
> >
> > I hope we will review carefully what we are including in the release
> >
> > And that we will let branch-2.9 stabilise by not adding any other commits
> > that are not strictly required:
> > - security related issues
> > - data loss/data corruption cases
> >
> >
> > Enrico
> >
> >
> > Thank you Ran Gao for starting this thread.
> >
> >
> >
> > Enrico
> >
> > Il Mer 5 Gen 2022, 17:29 Enrico Olivelli  ha scritto:
> >
> > > 171 commits?
> > >
> > > Why?
> > >
> > > This is too much in my opinion for a point release.
> > >
> > > I am pretty sure that we don't need all that changes.
> > >
> > > Every change we cherry pick has a good chance to break the stability
> > >
> > > Enrico
> > >
> > > Il Mer 5 Gen 2022, 16:30 mattison chao  ha
> > > scritto:
> > >
> > >> +1
> > >>
> > >> regards,
> > >> Mattisonchao
> > >>
> > >> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
> > >>
> > >> > +1,Thanks for your work.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Best Regards,
> > >> > Lan Liang
> > >> > On 1/5/2022 23:01，Hang Chen wrote：
> > >> > +1
> > >> >
> > >> > Best,
> > >> > Hang
> > >> >
> > >> > 陳智弘  于2022年1月5日周三 21:30写道：
> > >> >
> > >> > +1
> > >> >
> > >> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
> > >> >
> > >> > +1
> > >> >
> > >> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> > >> >
> > >> > Hello, Pulsar community:
> > >> >
> > >> > I'd like to propose that we release Apache Pulsar 2.9.2.
> > >> >
> > >> > Currently, compared to 2.9.1, branch-2.9 already merged 171
> > >> commits(refer
> > >> > to [0]), they contain the log4j security patch and many important 
> > >> > fixes.
> > >> >
> > >> > I am happy to volunteer to be the release manager.
> > >> >
> > >> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> > >> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> > >> > labeled release/2.9.2, I'll follow them to make sure important fix 
> > >> > could
> > >> > be
> > >> > merged in 2.9.2.
> > >> >
> > >> >
> > >> > Best,
> > >> > Ran Gao
> > >> >
> > >> >
> > >> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> > >> >
> > >> >
> > >> >
> > >>
> > >
> 

Re: [VOTE] Pulsar Release 2.9.2 Candidate 1

[Ran Gao]

Sorry, the package has the wrong sign certificate. I'll close the rc1 for 2.9.2 
and open start the rc2.

On 2022/01/24 09:33:32 PengHui Li wrote:
> The Signature failed to verify
> 
> ```
> ~/Downloads/release_2.9.2 » gpg --verify apache-pulsar-2.9.2-bin.tar.gz.asc
> gpg: assuming signed data in 'apache-pulsar-2.9.2-bin.tar.gz'
> gpg: Signature made Sat Jan 22 21:28:07 2022 CST
> gpg:using RSA key 9F6FE6F28CC92CCB54B4E6F6C54B95E1C9106DA3
> gpg: Can't check signature: No public key
> ```
> 
> Looks you are using a different key from the uploaded one.
> 
> Regards,
> Penghui
> 
> On Sun, Jan 23, 2022 at 9:25 PM Ran Gao  wrote:
> 
> > This is the first release candidate for Apache Pulsar, version 2.9.2.
> >
> > It fixes the many issues and the log4j security problem.
> >
> > *** Please download, test, and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-1/
> >
> > SHA-512 checksums:
> >
> >
> > ccecec6ed53e9aea8882be0ee8d3e2dd353cde48033dad329d907580e2e472ac4b63e5467ccc5ec218ed7095a275526d78f7b9efb0dc467af4113848c2462673
> >  ./apache-pulsar-2.9.2-bin.tar.gz
> >
> > ebc4422c7a293aaa2aa32cb1dbd0e7a96e1848ae0a6ad70fe14a672359704085e98a40d9f97af472306f4eba798615836485782f8975738a6771087f9dfb3cbe
> >  ./apache-pulsar-2.9.2-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1132/
> >
> > The tag to be voted upon:
> > v2.9.2-candidate-1 (8a5d2253b888b3b865a2aedf635d672821c7)
> > https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> 

[VOTE] Pulsar Release 2.9.2 Candidate 2

[Ran Gao]

Sorry, the 2.9.2 release candidate-1 has a wrong sign certificate.

This is the second release candidate for Apache Pulsar, version 2.9.2.

*** Please download, test, and vote on this release. This vote will stay
open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-2/

SHA-512 checksums:

563f65582c5307b4ef1e0322958ed19d7c181fb8bb8d7b8cab06ab0a6adb5520f7d18b6f97960b93c3318815529a8b8721e00e9cc9484532a2e5ed3221450094
 ./apache-pulsar-2.9.2-bin.tar.gz
60d1049611b938b0ddc769132124d43820728afc8a06813a5ec9efc095c5497c59d9bbcaaf7df5b0c0e97e051d66f59c1f8ee08885d05ca2c635773e0283770a
 ./apache-pulsar-2.9.2-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1136

The tag to be voted upon:
v2.9.2-candidate-2 (8a5d2253b888b3b865a2aedf635d672821c7)
https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Ran Gao]

Yes, all necessary PRs are cherry-picked to branch-2.9, and the test pass, 
refer to https://github.com/apache/pulsar/commits/branch-2.9, I'm cutting the 
new RC from the current branch-2.9.

On 2022/02/15 14:57:25 Enrico Olivelli wrote:
> Ran,
> Are we cutting the new RC from the current branch-2.9?
> 
> Enrico
> 
> Il Mar 15 Feb 2022, 14:57 PengHui Li  ha scritto:
> 
> > Hi Ran,
> >
> > I think all the PRs that block the 2.9.2 release are merged.
> > Could you please help cherry-pick the PRs and start a new RC?
> >
> > Thanks,
> > Penghui
> >
> > On Mon, Feb 14, 2022 at 8:25 PM PengHui Li  wrote:
> >
> > > Hi Lari,
> > >
> > > We have another issue that needs to confirm if it will introduce break
> > > changes in 2.9.2,
> > > Expected to have a result tomorrow, it related to
> > > https://github.com/apache/pulsar/pull/13383,
> > > We're doing more testing to make sure it doesn't introduce unexpected
> > > behavior.
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Mon, Feb 14, 2022 at 8:10 PM Lari Hotari  wrote:
> > >
> > >> When is 2.9.2 Candidate 3 planned?
> > >> What changes will it include? All current changes in branch-2.9 ?
> > >> The version has already been set to 2.9.3-SNAPSHOT in branch-2.9 with
> > >> https://github.com/apache/pulsar/pull/14089 . If we do 2.9.2 with all
> > >> current changes from branch-2.9, the commit for PR 14089 would have to
> > be
> > >> reverted before the next release.
> > >> Another possibility is to skip 2.9.2 completely and proceed directly
> > with
> > >> 2.9.3 release.
> > >>
> > >> -Lari
> > >>
> > >> On 2022/02/11 08:28:58 PengHui Li wrote:
> > >> > Now, there is a regression introduced in 2.9.2
> > >> >
> > >> > I have pushed out the fix https://github.com/apache/pulsar/pull/14231
> > ,
> > >> PTAL.
> > >> >
> > >> > -1 from my side
> > >> >
> > >> > Need to get the fix merged and roll out the new RC3 @Ran
> > >> >
> > >> > Regards,
> > >> > Penghui
> > >> >
> > >> > On Thu, Feb 10, 2022 at 9:54 PM Nicolò Boschi 
> > >> wrote:
> > >> >
> > >> > > Penghui,
> > >> > >
> > >> > >
> > >> > > I didn't know that there were so many known bugs around transactions
> > >> > > scheduled for 2.9.3, my bad.
> > >> > >
> > >> > > However, as Enrico pointed out, the issue impacts Pulsar clients
> > that
> > >> are
> > >> > > not using the transactions, so we can't just say - ok, just another
> > >> bug
> > >> > > about transactions, it's not critical since they're not production
> > >> ready
> > >> > > (btw, where we state that they aren't production ready on the
> > >> > > documentation?).
> > >> > >
> > >> > >
> > >> > > The workaround you mentioned is not always viable, since you can
> > have
> > >> > > clients of different tenants/customers that are not using
> > transactions
> > >> > > while, at the same time, a little portion that are experiencing with
> > >> them.
> > >> > >
> > >> > > I agree that it is uncommon to have only one message produced. On
> > the
> > >> other
> > >> > > hand, it's a very common case where other projects using Pulsar have
> > >> > > unit/integration tests that write only one message and expect to be
> > >> > > consumed (that's because they test the application logic and not
> > >> Pulsar).
> > >> > >
> > >> > >
> > >> > > Given that, it's fair to say that 2.9.2 is not worse than 2.9.1, so,
> > >> > > finally, we can go ahead.
> > >> > >
> > >> > > Looking forward to see 2.9.3 soon
> > >> > >
> > >> > >
> > >> > > I tested the artifacts, so I'll put my vote here:
> > >> > >
> > >> > >
> > >> > > +1 (non binding)
> > >> > >
> > >> > >
> > >> > > Checks:
> > >> > >
> > >> > > - Checksum and signatures
> > >> > >
> > >> > > - Apache Rat check passes
> > >> > >
> > >> > > - Compile from source w JDK11
> > >> > >
> > >> > > - Build docker image from source
> > >> > >
> > >> > > - Run Pulsar standalone and produce-consume from CLI
> > >> > >
> > >> > >
> > >> > > BR,
> > >> > >
> > >> > > Nicolò
> > >> > >
> > >> > > Il giorno gio 10 feb 2022 alle ore 13:39 PengHui Li <
> > >> peng...@apache.org>
> > >> > > ha
> > >> > > scritto:
> > >> > >
> > >> > > > > Please go ahead with the release, I won't VOTE on this thread.
> > >> > > > But I hope we can follow up soon with a new release, otherwise due
> > >> to
> > >> > > that
> > >> > > > bug
> > >> > > > you cannot enable transactions on your Pulsar cluster if you have
> > to
> > >> > > > support Pulsar client that do not enable transactions
> > >> > > >
> > >> > > >
> > >> > > > Yes, agree. We will follow up the 2.9.3 soon. There are other
> > >> > > > ongoing transaction fixes
> > >> > > > we will complete them ASAP and provide a version with certain
> > >> guarantees
> > >> > > > for transaction stability.
> > >> > > > We are doing lots of tests these days, 2.9.3 should be a good
> > >> version for
> > >> > > > transactions.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Penghui
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Feb 10, 2022 at 7:37 PM Lin Lin 
> > wrote:
> > >> > > >
> > >> > > > >
> 

[VOTE] Pulsar Release 2.9.2 Candidate 3

[Ran Gao]

This is the third release candidate for Apache Pulsar, version 2.9.2.

*** Please download, test, and vote on this release. This vote will stay
open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-3/

SHA-512 checksums:

21ab21df057dc181d02806e67a8d85b861fa60a8413a8e4058ffcb738d9ef81d8797c4e7249d50cf2a23c7437cb2880f4240d010a983eda33a2071f5fddf6b01
 apache-pulsar-2.9.2-bin.tar.gz
f927d77aba32d1112cd009a1d91a8fca27a49fcb20ea215a5792dc16e2b04bd822238c3f29dd81a7853e20062a0454e2abe608626b24e4e25ff06915b7a04785
 apache-pulsar-2.9.2-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1141/

The tag to be voted upon:
v2.9.2-candidate-3 (5615809ffd68599ae7fec1be0679ee83171f42dc)
https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-3

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build and run
the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.8.3 Candidate 2

[Ran Gao]

+1

- Check src build
- Check run standalone mode by binary package
- Check Basic publish and consume partitioned topic
- Check Pulsar SQL query bytes data

On 2022/02/16 04:34:55 Michael Marshall wrote:
> This is the second release candidate for Apache Pulsar, version 2.8.3.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/compare/v2.8.2...v2.8.3-candidate-2
> 
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
> 
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.3-candidate-2/
> 
> SHA-512 checksums:
> 
> aa9fb934260e158fe6a30208324459dd747d32fbb52ee61a28a322c6161f3e21d2e97f61118ee0e82488720f7c7787233949ef9eb80567b83896f08b12c54090
>  ./apache-pulsar-2.8.3-bin.tar.gz
> 47747e2cdc323c00fcdd08c537c77f355a94fbfaee77789718cb70e52726b7084522a842505e7eff1c1be26fddc850f2134d400a803854286ac8ea2f7cada121
>  ./apache-pulsar-2.8.3-src.tar.gz
> 
> Unofficial Docker images:
> michaelmarshall/pulsar:2.8.3-rc2
> michaelmarshall/pulsar-all:2.8.3-rc2
> michaelmarshall/pulsar-standalone:2.8.3-rc2
> michaelmarshall/pulsar-grafana:2.8.3-rc2
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1140/
> 
> The tag to be voted upon:
> v2.8.3-candidate-2 (83a522f3a17d41eb3727ffee67cdf035e8ea471b)
> https://github.com/apache/pulsar/releases/tag/v2.8.3-candidate-2
> 
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
> 

Re: [VOTE] Pulsar Release 2.9.2 Candidate 3

[Ran Gao]

Thanks, WangJialing, Penghui Li.

After the PR 14409 merged, I'll start the Pulsar 2.9.2 candidate 4.

On 2022/02/23 04:05:30 PengHui Li wrote:
> Looks it will affect the 2.8.3 and 2.10.0 releases,
> 
> > We could add a new interface with `applied` parameter, but this also
> introduce inevitable break change.
> And I suggest to do this with PIP in the next major version release.
> For version 2.9.2, we need merge 14409 to avoid return null value.
> 
> https://github.com/apache/pulsar/pull/13831 also is a related PR to fix the
> breaking change, and it has merged.
> I agree with avoiding the API break change in 2.8.3, 2.9.2, and 2.10.0.
> To support get the applied policy of a namespace should start with a PIP,
> it might introduce different behavior with
> existing API.
> 
> Thanks,
> Penghui
> 
> On Wed, Feb 23, 2022 at 10:44 AM WangJialing <18896728...@139.com> wrote:
> 
> > I found a breaking change in version 2.9.2
> > https://github.com/apache/pulsar/pull/14409
> >
> > The change is introduced by https://github.com/apache/pulsar/pull/12786
> > Policies.is_allow_auto_update_schema changed from boolean to Boolean, this
> > may cause user's code get NPE.
> >
> > I'd like to start a discusstion about the solution.
> >
> > We could add a new interface with `applied` parameter, but this also
> > introduce inevitable break change.
> > And I suggest to do this with PIP in the next major version release.
> > For version 2.9.2, we need merge 14409 to avoid return null value.
> >
> >
> >
> > Wangjialing
> >
> > From: Ran Gao
> > Date: 2022-02-17 02:26
> > To: dev
> > Subject: [VOTE] Pulsar Release 2.9.2 Candidate 3
> > This is the third release candidate for Apache Pulsar, version 2.9.2.
> >
> > *** Please download, test, and vote on this release. This vote will stay
> > open for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-3/
> >
> > SHA-512 checksums:
> >
> >
> > 21ab21df057dc181d02806e67a8d85b861fa60a8413a8e4058ffcb738d9ef81d8797c4e7249d50cf2a23c7437cb2880f4240d010a983eda33a2071f5fddf6b01
> > apache-pulsar-2.9.2-bin.tar.gz
> >
> > f927d77aba32d1112cd009a1d91a8fca27a49fcb20ea215a5792dc16e2b04bd822238c3f29dd81a7853e20062a0454e2abe608626b24e4e25ff06915b7a04785
> > apache-pulsar-2.9.2-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1141/
> >
> > The tag to be voted upon:
> > v2.9.2-candidate-3 (5615809ffd68599ae7fec1be0679ee83171f42dc)
> > https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-3
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build and run
> > the Pulsar standalone service.
> >
> 

[VOTE] Pulsar Release 2.9.2 Candidate 4

[Ran Gao]

This is the fourth release candidate for Apache Pulsar, version 2.9.2.

*** Please download, test, and vote on this release. This vote will stay
open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-4/

SHA-512 checksums:

84ee5c809c30cfcc301f26ffe23dc6986715e919395932e33a10a974852dd5861a08ddcd7de6071296c57f31c1c6e9f926c790612881bfe96002b22e1387f26a
apache-pulsar-2.9.2-bin.tar.gz
2081ba80f258c1bf5f535207fdadacc0f7e1a30359cb0b58add8ee91e6947a2576fd84c483dc0b3913fc5fbb28055b08e94bcae661485bcbcb27f8e8b520f4ee
apache-pulsar-2.9.2-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1152

The tag to be voted upon:
v2.9.2-candidate-4 (a3f52891593e093c27b583094a1fbfd09bbbae1a)
https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-4

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build and run
the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.9.2 Candidate 4

[Ran Gao]

Thanks for voting, there are 3 bindings, I'll create the final tag `2.9.2` and 
finish the subsequent release process.

On 2022/03/23 04:31:17 Hang Chen wrote:
> +1 (binding)
> 
> - Checked checksum and license
> - Setup standalone  cluster and run pulsar perf to produce and consume
> - Check the prometheus metrics
> - Run open messaging to check produce and consume throughput and latency
> - Check source and sink connector
> 
> Thanks,
> Hang
> 
> Lin Lin  于2022年3月23日周三 12:12写道：
> >
> > - Checked the signature
> > - Checked the SHA-512 checksums
> > - Start standalone
> > - Create tenant and namespace
> > - Publish messages and consume messages
> > - Check Function
> >
> > Thanks,
> >  Lin Lin
> 

Re: [ANNOUNCE] New Committer: Zike Yang

[Ran Gao]

Hi, zike, congratulations!

Ran Gao

Best

On 2022/04/13 09:34:23 PengHui Li wrote:
> The Apache Pulsar Project Management Committee (PMC) has invited Zike Yang
> https://github.com/RobertIndie to become a committer and we are pleased to
> announce that he has accepted.
> 
> Welcome and Congratulations, Zike Yang!
> 
> Please join us in congratulating and welcoming Zike Yang onboard!
> 
> Best Regards,
> Penghui Li on behalf of the Pulsar PMC
> 

Re: [ANNOUNCE] New Committer: Ruguo Yu

[Ran Gao]

Congratulations!

Ran

Best

On 2022/04/15 10:06:44 Yu wrote:
> The Apache Pulsar Project Management Committee (PMC) has invited Ruguo Yu (
> https://github.com/yuruguo) to become a committer and we are pleased to
> announce that he has accepted.
> 
> Welcome and Congratulations, Ruguo Yu!
> 
> Please join us in congratulating and welcoming Ruguo Yu onboard!
> 
> Best Regards,
> 
> Yu on behalf of the Pulsar PMC
> 

Re: [DISCUSS] PIP-155: Drop support for Python2

[Ran Gao]

+1

Ran Gao

Best

On 2022/04/15 16:06:36 Matteo Merli wrote:
> https://github.com/apache/pulsar/issues/15185
> 
> -
> 
> ## Motivation
> 
> Python 2.x has been deprecated for many years now and it was
> officially end-of-lifed 2.5 years ago
> (https://www.python.org/doc/sunset-python-2/).
> 
> We have well reached the point by which we need to drop Python 2.7
> compatibility for Pulsar client and for Pulsar functions.
> 
> ## Goal
> 
> Support only Python 3.5+ for Pulsar client and for Pulsar functions.
> 
> ## API Changes
> 
> No changes at this time, though Pulsar Python client library will be
> now free to use Python3 specific syntaxes and libraries.
> 
> ## Changes
> 
> 1. Switch the CI build to run Python client lib tests with Python3
> 2. Switch integration tests to use Python3
> 3. Stop building and distributing wheel files for Python 2.7
> 
> 
> 
> 
> --
> Matteo Merli
> 
> 

Re: [VOTE] PIP-149: Making the REST Admin API fully async

[Ran Gao]

+1

Ran Gao

On 2022/03/25 02:13:26 mattison chao wrote:
> This is the voting thread for PIP-149. It will stay open for at least 48
> hours.
> 
> https://github.com/apache/pulsar/issues/14365
> 
> Pasted below for quoting convenience.
> 
> -
> 
> Motivation
> 
> The Rest API was originally designed to be implemented asynchronously, but
> with the iteration of functions, some synchronous implementations were
> added, resulting in many asynchronous methods called synchronous
> implementations. Also, many synchronous calls do not add timeouts. This
> greatly reduces concurrency, user operations, and experience.
> In order to prevent more problems, and improve code readability and
> maintainability, we intend to refactor these synchronous calls and
> standardize the implementation of the API.
> 
> Related discussion:
> https://lists.apache.org/thread/pkkz2jgwtzpksp6d4rdm1pyxzb3z6vmg
> 
> Goals
> 
> Try to avoid synchronous method calls in asynchronous methods.
> Async variable (AsyncResponse) is placed in the first parameter position.
> Async variable (AsyncResponse) cannot be substituted into method
> implementations.
> Add more tests and increase the coverage.
> Modification
> Avoid synchronous method calls in asynchronous methods.
> 
> protected void internalDeleteNamespace(boolean authoritative) {
>validateTenantOperation(namespaceName.getTenant(),
> TenantOperation.DELETE_NAMESPACE);
>validatePoliciesReadOnlyAccess();
> }
> Suggest to do like this:
> 
> protected CompletableFuture internalDeleteNamespace(boolean
> authoritative) {
> return validateTenantOperationAsync(namespaceName.getTenant(),
> TenantOperation.DELETE_NAMESPACE)
>.thenCompose(__ -> validatePoliciesReadOnlyAccessAsync());
> }
> Async variable (AsyncResponse) is placed in the first parameter position
> 
> public void deleteNamespace(@Suspended final AsyncResponse asyncResponse,
>  @PathParam("tenant") String tenant,
>  @PathParam("namespace") String namespace,
>  @QueryParam("force") @DefaultValue("false") boolean force,
>  @QueryParam("authoritative") @DefaultValue("false") boolean
> authoritative) {
> 
> Async variable (AsyncResponse) cannot be substituted into method
> implementations
> 
> internalCreateNonPartitionedTopicAsync(asyncResponse, authoritative,
> properties);
> Suggest to do like this:
> 
> internalCreateNonPartitionedTopicAsync(authoritative, properties)
> .thenAccept(__ ->
> asyncResponse.resume(Response.noContent().build()))
> .exceptionally(ex -> {
> resumeAsyncResponseExceptionally(asyncResponse, ex.getCause());
> return null;
> });
> 
> Task tracking
> In order to unify the modification and track the modified part, it's better
> to open an issue to track, like #14353, #14013, #13854.
> 
> ---
> Best,
> Mattison
> 

Re: [ANNOUNCE] New Committer: Zhangmin Gao

[Ran Gao]

+1

Best,
Ran Gao

On 2022/04/13 09:35:57 PengHui Li wrote:
> The Apache Pulsar Project Management Committee (PMC) has invited Zhangmin
> Gao
> https://github.com/gaozhangmin to become a committer and we are pleased to
> announce that he has accepted.
> 
> Welcome and Congratulations, Zhangmin Gao!
> 
> Please join us in congratulating and welcoming Zhangmin Gao onboard!
> 
> Best Regards,
> Penghui Li on behalf of the Pulsar PMC
> 

Re: [DISCUSS] [PIP-154] Max active transaction limitation for transaction coordinator

[Ran Gao]

Bo, Great work!

+1

best,
Ran Gao

On 2022/04/12 12:44:36 丛搏 wrote:
> pipHi Pulsar community,
> 
> I open a pip to discuss Max active transaction limitation for
> transaction coordinator.
> link: https://github.com/apache/pulsar/issues/15133
> 
> 
> Thanks,
> Bo
> 

[ANNOUNCE] Apache Pulsar 2.9.2 released

[Ran Gao]

The Apache Pulsar team is proud to announce Apache Pulsar version 2.9.2.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, it also provides transaction
feature since 2.8.0, guaranteed exactly-once semantics for producing and
consuming process over topics, automatic cursor management for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
https://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Re: [VOTE] [PIP-154] Max active transaction limitation for transaction coordinator

[Ran Gao]

I think this is a valuable improvement, it could make the Pulsar transaction 
more stable, great work.

+1

best,
Ran Gao

On 2022/04/24 02:00:39 丛搏 wrote:
> Hi Pulsar community,
> 
> This is the voting thread for PIP-154. It will stay open for at least 48 
> hours.
> 
> The proposal can be found: https://github.com/apache/pulsar/issues/15133
> 
> Discuss thread: 
> https://lists.apache.org/thread/h8gzgs78bm4pzy2rjtgvvmo1jbzttcx8
> 
> Thanks,
> Bo
> 

Re: Call for projects and mentors for OSPP 2022

[Ran Gao]

Thanks for Penghui and Dianjin,

Project Name: Query compacted data of topic by Pulsar SQL

Project Description:

The Pulsar SQL is a SQL query engine for Pulsar topics data analytics based on 
the Trino, users could get data from Pulsar topics by SQL scripts. The topic 
compaction is used to compress messages which has a key, after compaction, the 
topic will only maintain the latest value of the key.

Currently, the Pulsar didn't support querying compacted data, it's a useful 
improvement for Pulsar SQL. Some key points for implementing this feature are 
understanding the topic compaction process, how to split topic data(compacted 
part and un-compacted part) to generate query tasks, and reading data of the 
compacted ledger.

Difficulty Level:
- [ ] Basic
- [x] Advanced

Project Validation Items:
Item 1: Understand and try to use topic compaction and the Pulsar SQL component.
Item 2: Prepare a design for this feature.
Item 3: Start coding.
Item 4: Add unit and integration tests.
Item 5: Add doc for this feature.

Project Mentor:
Your Name: Ran Gao
Your Email: r...@apache.org
Your Apache ID: rgao


On 2022/04/24 13:23:18 PengHui Li wrote:
> Hi all,
> 
> Open Source Promotion Plan, OSPP[1] is open for project registration,
> like Google Summer of Code. This Plan aims to encourage college students
> to participate in open-source projects and make contributions.
> 
> The project idea is open from 4/21, 2022 to 5/9, 2022(UTC+8).
> Apache Pulsar ended up with 6 projects seats. If you have great ideas,
> please copy the following template and fill in the fields.
> Once you have completed your project ideas, you can reply to this email.
> We will help you register for the program. Note that One mentor can
> mentor one project at a time and both mentors and students can
> receive financial awards for completed projects. Only Pulsar committers
> can be the mentors as asked by the OSPP.
> 
> The project can include feature catchup, ecological tool construction,
> new features of the pulsar and so on.
> 
> Welcome to join the Pulsar
> slack channel: #ospp-summer for asking any questions about OSPP.
> 
> [1] https://summer-ospp.ac.cn
> 
> Thanks,
> Penghui
> 
> 
> Template
> 
> Project Name:
> Project Description: (at most 1000 words)
> Difficulty Level:
> - [ ] Basic
> - [ ] Advanced
> Project Validation Items:
> Item 1:__
> Item 2:__
> Item 3:__
> …
> Project Mentor:
> Your Name:
> Your Email:
> Your Apache ID:
> 

Re: [ANNOUNCE] new Committer Qiang Zhao (mattisonchao)

[Ran Gao]

Congratulations! Qiang

best,
Ran

On 2022/05/04 06:28:24 Enrico Olivelli wrote:
> The Project Management Committee (PMC) for Apache Pulsar
> has invited Qiang Zhao  to become a committer and we are pleased
> to announce that he has accepted.
> 
> Qiang Zhao (with GH id mattisonchao) contributed lots of improvements
> and bugfixes to Pulsar.
> 
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
> A PMC member helps manage and guide the direction of the project.
> 
> Please join me in congratulating with him !
> 
> Best regards
> Enrico Olivelli
> 

Re: [DISCUSSION] PIP-156: Enable system topic by default

[Ran Gao]

+1 (non-binding)

Best,
Ran

On 2022/04/26 12:07:32 guo jiwei wrote:
> Hi community:
>In PIP-39
> ,
> we have introduced namespace event to support topic level policy. This
> feature was released in Pulsar 2.6.
> 
>Then PIP-92
> ,
> we support topic policy to replicate across clusters and release in Pulsar
> 2.10.
> 
> And with version iteration, the corresponding topic policy CLI and
> functions are mature and stable, but users need to enable system topic to
> support these features.  So it's better to enable system topic by default.
> 
> 
> So I have opened apache/pulsar#15333
>  to discuss this.
> 
> 
> Regards
> Jiwei Guo (Tboy)
> 

Re: [VOTE] [PIP-153] Optimize metadataPositions in MLPendingAckStore

[Ran Gao]

+1 (non-binding)

Best,
Ran

On 2022/04/25 06:54:58 一苇以航 wrote:
> Hi Pulsar Community.
> 
> This is the voting thread for PIP-153. It will stay open for at least 48 
> hours.
> 
> The proposal can be found:https://github.com/apache/pulsar/issues/15073
> 
> Discuss thread:
> https://lists.apache.org/thread/svmbp8ybn6l8o0o8dmvsysnb86qj77r3
> 
> Thanks,
> Xiangying
> 
>  

Re: [VOTE] [PIP-158] Split client TLS transport encryption from authentication

[Ran Gao]

+1 (non-binding)

Best,
Ran

On 2022/05/10 08:46:26 Zixuan Liu wrote:
> Hi Pulsar community,
> 
> Voting for https://github.com/apache/pulsar/issues/15289
> 
> Discussion thread:
> https://lists.apache.org/thread/fblmm8oc7h907cfnppvk71o2cbp5mk8q
> 
> Thanks,
> Zixuan
> 
> --
> 
> ## Motivation
> 
> The client supports TLS transport encryption and TLS authentication, this
> code so like:
> 
> ```java
> PulsarClient client = PulsarClient.builder()
> .serviceUrl("pulsar+ssl://localhost:6651")
> .tlsTrustCertsFilePath("/path/to/cacert.pem")
> .authentication(AuthenticationTls.class.getName(),
> authParams)
> .build()
> ```
> 
> This causes an issue that cannot use other authentication with TLS
> transport encryption, and also made our confusion if we use TLS transport
> encryption by setting `authentication`.
> 
> ## Goal
> 
> Split client TLS transport encryption from authentication is used to
> support TLS transport encryption with any authentication.
> 
> ## API Changes
> 
> - Add new methods in `org.apache.pulsar.client.api.ClientBuilder`
> 
> ```java
> public interface ClientBuilder extends Serializable, Cloneable {
> /**
>  * Set the path to the TLS key file.
>  *
>  * @param tlsKeyFilePath
>  * @return the client builder instance
>  */
> ClientBuilder tlsKeyFilePath(String tlsKeyFilePath);
> 
> /**
>  * Set the path to the TLS certificate file.
>  *
>  * @param tlsCertificateFilePath
>  * @return the client builder instance
>  */
> ClientBuilder tlsCertificateFilePath(String tlsCertificateFilePath);
> 
> /**
>  * The file format of the key store file.
>  *
>  * @param tlsKeyStoreType
>  * @return the client builder instance
>  */
> ClientBuilder tlsKeyStoreType(String tlsKeyStoreType);
> 
> /**
>  * The location of the key store file.
>  *
>  * @param tlsTrustStorePath
>  * @return the client builder instance
>  */
> ClientBuilder tlsKeyStorePath(String tlsTrustStorePath);
> 
> /**
>  * The store password for the key store file.
>  *
>  * @param tlsKeyStorePassword
>  * @return the client builder instance
>  */
> ClientBuilder tlsKeyStorePassword(String tlsKeyStorePassword);
> }
> ```
> 
> ## Implementation
> 
> ### TLS transport encryption
> 
> We can call the `tlsKeyFilePath()`, `tlsCertificateFilePath()` and
> `tlsTrustCertsFilePath()` to configurate the TLS transport encryption, the
> code so like:
> 
> ```java
> PulsarClient client = PulsarClient.builder()
> .serviceUrl("pulsar+ssl://my-host:6650")
> .tlsTrustCertsFilePath("/path/to/cacert.pem")
> .tlsKeyFilePath("/path/to/client-key.pem")
> .tlsCertificateFilePath("/path/to/client-cert.pem")
> .build();
> ```
> 
> 
> Call the `tlsKeyFilePath()`, `tlsTrustStorePath()` to configurate the JKS
> TLS transport encryption, the code so like:
> 
> ```java
> PulsarClient client = PulsarClient.builder()
> .serviceUrl("pulsar+ssl://my-host:6650")
> .tlsKeyFilePath("/path/key.jks")
> .tlsKeyStorePassword("hello")
> .tlsTrustStorePath("/path/trust.jks")
> .tlsTrustStorePassword("hello")
> .build();
> ```
> 
> 
> ### TLS transport encryption with any authentication
> 
> We can call the `tlsKeyFilePath()`, `tlsCertificateFilePath()`,
> `tlsTrustCertsFilePath()` and `authentication()` to configurate  the TLS
> transport encryption with any authentication, the code so like:
> 
> ```java
> PulsarClient client = PulsarClient.builder()
> .serviceUrl("pulsar+ssl://my-host:6650")
> .tlsTrustCertsFilePath("/path/to/cacert.pem")
> .tlsKeyFilePath("/path/to/client-key.pem")
> .tlsCertificateFilePath("/path/to/client-cert.pem")
> .authentication(AuthenticationTls.class.getName() /*
> AuthenticationToken.class.getName()*/, authParams)
> .builder()
> ```
> 
> Call the `tlsKeyFilePath()`, `tlsTrustStorePath()` to configurate the JKS
> TLS transport encryption, the code so like:
> 
> ```java
> PulsarClient client = PulsarClient.builder()
> .serviceUrl("pulsar+ssl://my-host:6650")
> .tlsKeyFilePath("/path/key.jks")
> .tlsKeyStorePassword("hello")
> .tlsTrustStorePath("/path/trust.jks")
> .authentication(AuthenticationTls.class.getName() /*
> AuthenticationToken.class.getName()*/, authParams)
> .build();
> ```
> 
> For `AuthenticationTls`, we need to check the authParams, when the
> authParams is empty, we need to read TLS config from `ClientBuilder`,
> otherwise read from the authParams, the authParams can override the config
> from `ClientBuilder`, if still is empty, we read TLS config from
> `ClientBuilder`.
> 
> ### Plan test
> 
> - Verify TLS transport encryption without authentication
> - Verify TLS transport encryption with token authentication
> - Verify TLS transport encryption with TL

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Ran Gao]

This is a good idea, but I have a concern, Pulsar has the config 
`brokerMaxConnections` to control max connection count against one broker. If 
the connection is closed, it will re-connect when consumers or producers start 
to consume and produce messages again, but this time the max connection count 
will reach the max count.


On 2022/05/26 06:31:37 Yubiao Feng wrote:
> I open a pip to discuss Auto release client useless connections, could you
> help me review
> 
> 
> ## Motivation
> Currently, the Pulsar client keeps the connection even if no producers or
> consumers use this connection.
> If a client produces messages to topic A and we have 3 brokers 1, 2, 3. Due
> to the bundle unloading(load manager)
> topic ownership will change from A to B and finally to C. For now, the
> client-side will keep 3 connections to all 3 brokers.
> We can optimize this part to reduce the broker side connections, the client
> should close the unused connections.
> 
> So a mechanism needs to be added to release unwanted connections.
> 
> ### Why are there idle connections?
> 
> 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> connection is not closed at all.
> The design is to hold a fixed number of connections per Host, avoiding
> frequent closing and creation.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> 
> 2-1. When clients receive `command-close`, will reconnect immediately.
> It's designed to make it possible to reconnect, rebalance, and unload.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> 
> 2-2. The broker will close client connections before writing ownership info
> to the ZK. Then clients will get deprecated broker address when it tries
> lookup.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> 
> ## Goal
> Automatically release connections that are no longer used.
> 
> - Scope
>   - **Pulsar client**
> Contains connections used by consumers, Producers, and Transactions.
> 
>   - **Pulsar proxy**
> Contains only the connection between Proxy and broker
> 
> ## Approach
> Periodically check for idle connections and close them.
> 
> ## Changes
> 
> ### API changes
> **ClientCnx** added an idle check method to mark idle time.
> 
> ```java
> /** Create time. **/
> private final long createTime;
> /** The time when marks the connection is idle. **/
> private long IdleMarkTime;
> /** The time when the last valid data was transmitted. **/
> private long lastWorkTime;
> /** Stat. enumerated values: using, idle_marked, before_release, released**/
> private int stat;
> /**
>   * Check client connection is now free. This method may change the state
> to idle.
>   * This method will not change the state to idle.
>   */
> public boolen doIdleCheck();
> /** Get stat **/
> public int getStat();
> /** Change stat **/
> public int setStat(int originalStat, int newStat);
> ```
> 
> ### Configuration changes
> We can set the check frequency and release rule for idle connections at
> `ClientConfigurationData`.
> 
> ```java
> @ApiModelProperty(
> name = "autoReleaseIdleConnectionsEnabled",
> value = "Do you want to automatically clean up unused connections"
> )
> private boolean autoReleaseIdleConnectionsEnabled = true;
> 
> @ApiModelProperty(
> name = "connectionMaxIdleSeconds",
> value = "Release the connection if it is not used for more than
> [connectionMaxIdleSeconds] seconds"
> )
> private int connectionMaxIdleSeconds = 180;
> 
> @ApiModelProperty(
> name = "connectionIdleDetectionIntervalSeconds",
> value = "How often check idle connections"
> )
> private int connectionIdleDetectionIntervalSeconds = 60;
> ```
> 
> ## Implementation
> 
> - **Pulsar client**
> If no consumer, producer, or transaction uses the current connection,
> release it.
> 
> - **Pulsar proxy**
> If the connection has not transmitted valid data for a long time, release
> it.
> 
> 
> Yubiao Feng
> Thanks
> 

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Ran Gao]

Ok, thanks for your explanation, make sense to me.

+1

On 2022/06/05 17:08:22 Yubiao Feng wrote:
> Hi Ran
> 
> I think you mean that: Producer/Consumer failed to establish a connection
> when he tried to work again.
> 
> There are two places in the Broker configuration that limit the maximum
> number of connections:
> - Broker config : maxConnectionsLimitEnabled
> - Broker config: maxConnectionsLimitPerIpEnabled
> 
> At client side:
> We only release connections that are not registered with producer or
> Consumer or Transaction. So when a new producer creates it will get an
> error (NotAllowedError because reached the maximum number of
> connections)  same as original design.
> 
> At proxy side:
> I'm sorry I didn't think it through before. I changed the proxy part of the
> proposal:
> 
> The connection between proxy and broker has two parts: For lookup commands;
> For consumers, producers commands  and other commands.
> The connection "For consumers, producers commands  and other commands" is
> managed by DirectProxyHandler, which holds the connection until the client
> is closed, so it does not affect of producers or consumers, These
> connections do not require additional closing.
> The connection "For lookup commands": When the proxy is configured
> `metadataStoreUrl`, the Lookup Command will select the registered broker by
> rotation training and create a connection. If we do not optimize the broker
> load balancing algorithm, all connections are considered useful connections.
> When the cluster is large, holds so many connections becomes redundant.
> Later, I will try to put forward other proposals to improve this
> phenomenon, so this proposal does not involve proxy connection release.
> 
> 
> 
> 
> On Fri, Jun 3, 2022 at 11:44 AM Ran Gao  wrote:
> 
> > This is a good idea, but I have a concern, Pulsar has the config
> > `brokerMaxConnections` to control max connection count against one broker.
> > If the connection is closed, it will re-connect when consumers or producers
> > start to consume and produce messages again, but this time the max
> > connection count will reach the max count.
> >
> >
> > On 2022/05/26 06:31:37 Yubiao Feng wrote:
> > > I open a pip to discuss Auto release client useless connections, could
> > you
> > > help me review
> > >
> > >
> > > ## Motivation
> > > Currently, the Pulsar client keeps the connection even if no producers or
> > > consumers use this connection.
> > > If a client produces messages to topic A and we have 3 brokers 1, 2, 3.
> > Due
> > > to the bundle unloading(load manager)
> > > topic ownership will change from A to B and finally to C. For now, the
> > > client-side will keep 3 connections to all 3 brokers.
> > > We can optimize this part to reduce the broker side connections, the
> > client
> > > should close the unused connections.
> > >
> > > So a mechanism needs to be added to release unwanted connections.
> > >
> > > ### Why are there idle connections?
> > >
> > > 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> > > connection is not closed at all.
> > > The design is to hold a fixed number of connections per Host, avoiding
> > > frequent closing and creation.
> > >
> > >
> > https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> > >
> > > 2-1. When clients receive `command-close`, will reconnect immediately.
> > > It's designed to make it possible to reconnect, rebalance, and unload.
> > >
> > >
> > https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> > >
> > > 2-2. The broker will close client connections before writing ownership
> > info
> > > to the ZK. Then clients will get deprecated broker address when it tries
> > > lookup.
> > >
> > >
> > https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> > >
> > > ## Goal
> > > Automatically release connections that are no longer used.
> > >
> > > - Scope
> > >   - **Pulsar client**
> > > Contains connections used by consumers, Producers, and Transactions.
> > >
> > >   - **Pulsar proxy**
> &g

Re: [VOTE] Pulsar Release 2.10.1 Candidate 1

[Ran Gao]

+1 (non-binding)

Verification list:
1. Build source code. (MacOS 12.2.1, Chip Apple M1 Pro)
2. Start standalone.
3. Producing and consuming tests based on perf tool.
4. Query bytes data by Pulsar SQL.
5. Using the binary package execute same operations.

On 2022/06/13 11:42:59 PengHui Li wrote:
> This is the first release candidate for Apache Pulsar 2.10.1
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.1+
> 
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
> 
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.1-candidate-1/
> 
> SHA-512 checksums:
> 
> 87045067cf123574b2b7d578f020efd177b0d9ff12a75d9f738e135cde35a034000c03e21cc8a264a6ef9e2b1df2c4493ffa3b9170b0f368d7424f1160df38cb
>  ./apache-pulsar-2.10.1-bin.tar.gz
> 
> 19c078f65b9105728adda3926a9a1527d6498f0e2337d37497a6d965a38acf94e217e58939bc5b0a8b8166511ae9ee176ee3d0c764d4777e285b7bcaeb2731e1
>  ./apache-pulsar-2.10.1-src.tar.gz
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1156/
> 
> The tag to be voted upon:
> v2.10.1-candidate-1 (0702b51ce09cc99044f4110e6bf98e82a0e86283)
> https://github.com/apache/pulsar/releases/tag/v2.10.1-candidate-1
> 
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Docker images:
> 
> https://hub.docker.com/layers/233293315/lph890127/pulsar/2.10.1-rc1/images/sha256-4a952b3c662b94247ffc4ff17be16ef176c293baaf346db13a095970f43adfd6?context=repo
> 
> https://hub.docker.com/layers/233303469/lph890127/pulsar-all/2.10.1-rc1/images/sha256-ac8180f4ba796291e961581821a1318437e49b0fa684597f5b67265a502822ee?context=repo
> 
> Please download the source package, and follow the Release Candidate
> Validation[1]
> to validate the release
> 
> [1] https://github.com/apache/pulsar/wiki/Release-Candidate-Validation
> 
> Thanks,
> Penghui
> 

Re: Pulsar Documentation Guides

[Ran Gao]

Great work!

On 2022/06/13 16:00:00 Yu wrote:
> Hi team,
> 
> This is a soft reminder:
> 
> If you make contributions to Pulsar documentation, please follow these
> guides to improve your PR quality and get your PR merged quickly.
> 
> - Pulsar Documentation Writing Style Guide [1]
> - Pulsar Documentation Writing Syntax Guide [2]
> - Pulsar Documentation Preview Guide [3]
> - Pulsar Documentation Label Guide [4]
> - Pulsar Pull Request Naming Convention Guide [5]
> - Pulsar Design Style Guide [6]
> 
> I'm continuously updating these guides and working on the documentation
> contribution guide. Stay tuned!
> 
> Feel free to comment, thank you!
> 
> [1]
> https://docs.google.com/document/d/1lc5j4RtuLIzlEYCBo97AC8-U_3Erzs_lxpkDuseU0n4/edit#
> [2]
> https://docs.google.com/document/d/12De2btkDHQVaqUlHjTqERmroMLKGhHdiC7rEttFTyqc/edit#
> [3]
> https://docs.google.com/document/d/1wszdtMRo6MhKbVaggPK7_bnKaC4TewuT--GWZZxJNGg/edit#heading=h.tng8x5vsxlpi
> [4]
> https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0/edit#
> [5]
> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.wu6ygjne8e35
> [6]
> https://docs.google.com/document/d/16Hp7Sc86MQtL0m8fc2w_TrcKXAuglwRwHmdmwfk00mI/edit#heading=h.b8ogodj5sj0
> 
> Yu
> 

Re: Required and recommended JavaVersion for Pulsar broker

[Ran Gao]

One problem is that the presto 334 needs jdk11+ and it didn't support jdk8, 
maybe 2.11 is a good time for Pulsar to upgrade the Presto to 334+ or the 
latest stable version.

On 2022/06/22 18:22:09 Enrico Olivelli wrote:
> For Pulsar 2.9 and 2.10 you should use jdk11
> you can checkout the official docker images
> 
> for 2.11 we decided to go with jdk17
> 
> Enrico
> 
> Il giorno mer 22 giu 2022 alle ore 17:42 ZhangJian He
>  ha scritto:
> >
> > Enrico commented on 31 Mar. Now it's required jdk17 on master branch.
> > `branch-2.10` is compatible with jdk8.
> >
> > Thanks
> > ZhangJian He
> >
> > tison  于2022年6月22日周三 23:37写道：
> >
> > > Hi devs.
> > >
> > > During a review of PR-16163[1] I notice a comment from @Enrico Olivelli
> > >   on PR-14953 said:
> > >
> > > > We are currently still supporting jdk8 (on the server side)
> > >
> > > However, on our README.md it recommends:
> > >
> > > Pulsar Runtime Java Version Recommendation
> > > pulsar master branch
> > > Broker JavaVersion: 17
> > >
> > > Also from the pom.xml file it shows:
> > >
> > > 17
> > > 17
> > >
> > >
> > > ${maven.compiler.target}
> > > 8
> > >
> > > I'm unsure what JavaVersion required and recommend for Pulsar broker.
> > >
> > > IIUC https://github.com/apache/pulsar/pull/15264 and PIP-156 (
> > > https://github.com/apache/pulsar/issues/15207) are ongoing efforts to run
> > > pulsar broker with JDK17.
> > >
> > > Best,
> > > tison.
> > >
> > > [1] https://github.com/apache/pulsar/pull/16163
> > >
> 

Re: [VOTE] PIP-195: New bucket based delayed message tracker

[Ran Gao]

+1 Great work!

Thanks,
Ran Gao

On 2022/08/29 10:37:31 Cong Zhao wrote:
> Hi Pulsar Community,
> 
> In order to facilitate community review, I copied this proposal to Google 
> Docs, please review it and complete the vote.
> 
> PIP: 
> https://docs.google.com/document/d/17HE88w4WuEsLoz0DExPpRN555UC_Tc3MEIKckVWksDc/edit
> 
> On 2022/08/08 06:51:31 Cong Zhao wrote:
> >  Hi Pulsar Community,
> > 
> > I would like to start a VOTE on "New bucket based delayed message tracker"
> > (PIP-195).
> > 
> > The proposal can be read at https://github.com/apache/pulsar/issues/16763
> > and the discussion thread is available at
> > https://lists.apache.org/thread/1krdhrvs803kb6rqzdh17q0f199nroz4
> > 
> > Voting will stay open for at least 48h.
> > 
> > Thanks,
> > Cong Zhao
> > 
> 

Re: Virtual Pulsar Community Meetings

[Ran Gao]

+1 @sijie

The virtual Pulsar community meeting is so cool and I think Sijie's proposal is 
more reasonable, thanks.

Best Regards,
Ran

On 2021/01/29 08:53:40, Sijie Guo  wrote: 
> That's a super great idea! Thank you for bringing this up!
> 
> Given there are a lot of committers/contributors are from North America and
> Asia, I think we should pick up a better time that would be suitable for
> people from Asia (Japan and China).
> In the bookkeeper community, we used to run two events. One is to cover NA
> and Asia, and the other one is to cover NA and EU.
> 
> I would suggest running biweekly meetings.
> 
> - Tuesday 4 PM - 5 PM PST for NA and Asia
> - Thursday 8 AM - 9 AM PST for NA and EU
> 
> All the events can be recorded and uploaded to Youtube. So people are able
> to watch the recordings as well.
> 
> If people are good with this proposal, I am happy to set up and coordinate
> the meetings given I have run many meetings in the BookKeeper community
> before.
> 
> We can get started next Tuesday and formalize the process as we go. Please
> vote for your idea.
> 
> Thanks,
> Sijie
> 
> 
> On Fri, Jan 29, 2021 at 12:05 AM Enrico Olivelli 
> wrote:
> 
> > Hello everyone,
> > in the BookKeeper community we used to have "Community Meetings" in order
> > to meet each other, discuss current works on the project, share knowledge
> > about current problems.
> >
> > What about having Community meetings for Pulsar ?
> > I would be happy to organize and to host the first meeting.
> >
> > In Apache we keep the decisions and the discussions on mailing lists, so
> > these meetings would be only to share information and we are not going to
> > make decisions.
> >
> > The Pulsar community is distributed all over the world, we have people from
> > China, the US, Europe, it will be hard to find a good time for everyone.
> > I suggest for the first meeting to meet at 8:30 PST
> >
> > For reference, this is the link to the "minutes" of the Community Meetings
> > in BK,
> > Sijie and Matteo will remember those days
> > https://cwiki.apache.org/confluence/display/BOOKKEEPER/Community+Meetings
> >
> >
> > Enrico
> >
> 

Re: [ANNOUNCE] New Committer: Jiwei Guo

[Ran Gao]

Congrats Jiwei!

Ran

On 2021/11/22 04:35:08 PengHui Li wrote:
> The Apache Pulsar Project Management Committee (PMC) has invited Jiwei Guo
> https://github.com/Technoboy- to become a committer and we are pleased to
> announce
> that he has accepted.
> 
> Jiwei has joined the community for more than 1 year now and he is active in
> the Pulsar community for more than 5 months, and he is a PMC member of
> Apache dolphinscheduler
> https://dolphinscheduler.apache.org/en-us/community/team.html
> 
> Welcome and Congratulations, Jiwei!
> 
> Please join us in congratulating and welcoming Jiwei onboard!
> 
> Best Regards,
> Penghui Li on behalf of the Pulsar PMC
> 

Re: [VOTE] Apache Pulsar 2.9.0 candidate 4

[Ran Gao]

Hi Enrico,

Thanks for your great work!

We also make some chaos tests for 2.9.0 candidate 4.

We opened a new Pulsar cluster based on 2.9.0-candidate-4 and injected some 
chaos experiments by chaos-mesh, such as broker pod failure, kill, and 
zookeeper pod failure, in this process, the entire Pulsar cluster could provide 
service normally, the producer and consumer could work continuously. But maybe 
the test is not very strict, we will improve it persistently.

Thanks


On 2021/11/17 13:04:52 Enrico Olivelli wrote:
> This is the fourth release candidate for Apache Pulsar, version 2.9.0.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/milestone/30?closed=1
> 
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
> 
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.0-candidate-4/
> 
> SHA-512 checksums:
> 
> 659afa98e2475fc95f3126aab47d365e3e43a9d419c9cbe20c99520029f0b28aec2f471da5ccf47b92017a4c7943fc113a96232b075eb675e2b9df3601b2a7ad
>  apache-pulsar-2.9.0-SNAPSHOT-bin.tar.gz
> ed93aa1b57411c5d153bcd1433db6679ddb9b4d23896b39c1a1ad8edc70aaa7e95842a5770d6d55b466ab90500e17f50549673601556e767f7c12135ae360e85
>  apache-pulsar-2.9.0-SNAPSHOT-src.tar.gz
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1104/
> 
> The tag to be voted upon:
> v2.9.0-candidate-4 (bdd57b21a66b81aab72c4ec39d516ffd2a769c35)
> https://github.com/apache/pulsar/releases/tag/v2.9.0-candidate-4
> 
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
> 
> Enrico Olivelli
> 

Re: [DISCUSS] Migrate CLI parser from jcommander to picocli

[Ran Gao]

+1 (non-binding), it's more elegant. There will be a lot of changes, we can 
create some tasks based on commands.

- Ran Gao

On 2024/02/21 05:07:46 Zixuan Liu wrote:
> Hi Pulsar Community,
> 
> The pulsar CLI(pulsar-admin, pulsar-client, pulsar-shell, and so on) uses
> the jcommander [1] as CLI parser, which is an awesome project, but the
> maintainer is not active and cannot keep up with modern CLI
> features(auto-completion, sub-command, native-images, suggest commands, and
> so on). Then I found the picocli [2] project to meet these needs, which is
> active and powerful. For comparison, please see [3].
> 
> Error prompt:
> ```
> bin/pulsar-admin clusters update cluster-a -b
> 
> # jcommander
> Need to provide just 1 parameter
> 
> # picocli
> Unknown option: '-b'
> ```
> 
> Suggest commands:
> ```
> bin/pulsar-admin cluste
> 
> # jcommander
> Expected a command, got cluste
> 
> # picocli
> Unmatched argument at index 0: 'cluste'
> Did you mean: pulsar-admin clusters?
> ```
> 
> What do you think about migrating CLI parse from jcommander to picocli?
> 
> Thanks,
> Zixuan
> 
> [1] - https://github.com/cbeust/jcommander
> [2] - https://picocli.info/
> [3] - https://github.com/remkop/picocli/wiki/picocli-vs-JCommander
> 

Re: [ANNOUNCE] New Committer: Asaf Mesika

[Ran Gao]

Congratulations, Asaf!

Best Regards,
Ran Gao

On 2024/02/20 16:50:29 Lari Hotari wrote:
> The Apache Pulsar Project Management Committee (PMC) has invited
> Asaf Mesika https://github.com/asafm to become a committer and we
> are pleased to announce that he has accepted.
> 
> Welcome and Congratulations, Asaf Mesika!
> 
> Please join us in congratulating and welcoming Asaf onboard!
> 
> Best Regards,
> 
> Lari Hotari
> on behalf of the Pulsar PMC
> 

[VOTE] Pulsar Release 3.1.3 Candidate 1

[Ran Gao]

This is the first release candidate for Apache Pulsar version 3.1.3.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.1.3+label%3Acherry-picked%2Fbranch-3.1+

*** Please download, test and verify on this release. This vote will stay
open for at least 72 hours ***

Note that we are verifying upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.3-candidate-1/

SHA-512 checksums:

66e2cdbc28af2c1bc5aafa8f0a9c726c2d3bb33128f81632dadbb49d51397c9295d82c797094988eda80494ad217053e609e135fca3506e135f198f7d3ce18ed
 apache-pulsar-3.1.3-bin.tar.gz
ca4ffeeb13b77b1d36f2e8e121a7698f2b989f73f88d1f708745cf45875daf033a8395bb2941f43b5e0f014ee482dfdc01ad805938456511ddfa9777a07c
 apache-pulsar-3.1.3-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1268

The tag to verify:
v3.1.3-candidate-1 (7e28e8404f50d065ed37a590c188b8934cae4a75)
https://github.com/apache/pulsar/commits/v3.1.3-candidate-1

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://downloads.apache.org/pulsar/KEYS

Docker images:

https://hub.docker.com/layers/gaoran10/pulsar/3.1.3-7e28e84/images/sha256-fe637cca3c98137f70237568f12d589b62d00483b5563ac8c63ac0a6a5d33b5c?context=repo

https://hub.docker.com/layers/gaoran10/pulsar-all/3.1.3-7e28e84/images/sha256-219cf93ea289254de127bb52e0c0f1d324a6891fda583ec2ade02c0b0b5a508f?context=repo


Please download the source package, and follow the README to build
and run the Pulsar standalone service.


Regards
Ran Gao (gaoran10)

Re: [VOTE] Pulsar Release 3.2.1 Candidate 1

[Ran Gao]

+1(non-binding)

1. verify the GPG signature for the below files
- apache-pulsar-3.2.1-bin.tar.gz.asc
- apache-pulsar-3.2.1-src.tar.gz.asc
2. build the source code
3. test following the release verifying doc.

Regards,
Ran Gao

On 2024/03/05 07:05:05 guo jiwei wrote:
> This is the first release candidate for Apache Pulsar version 3.2.1.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.2.1+label%3Acherry-picked%2Fbranch-3.2+
> 
> *** Please download, test and verify on this release. This vote will stay
> open for at least 72 hours ***
> 
> Note that we are verifying upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.1-candidate-1/
> 
> SHA-512 checksums:
> 
> d500ba21305d56b0f7b4355e1270e50ae5f60e5632b6632d66fec6d1178bb9fdf3f24caa709b6b36ea8273e5a2c094868cf30e389154bc8bb6397e7de4f1bf1d
> 
> apache-pulsar-3.2.1-bin.tar.gz
> 
> 7aee1623db6dc95058cd0e7a4f108f8a3f43163d798a8eeeaecf5f335225c04f5a0518ce2dfb2d4fa29b7b5e54d27ba24af82fbb3542eb3ffc9e3602cb577878
> 
> apache-pulsar-3.2.1-src.tar.gz
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1267/
> 
> The tag to verify:
> v3.2.1-candidate-1 (158d5eb670c9fd7b123c204533ac6cf8cb439ccd)
> https://github.com/apache/pulsar/commits/v3.2.1-candidate-1/
> 
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Docker images:
> 
> pulsar images:
> https://hub.docker.com/layers/technoboy8/pulsar/3.2.1-158d5eb/images/sha256-23f0cbd54b1fb504dcab16dfcce562ed735ab94db1c5f6fabbe9145f3a2d0fa8?context=repo
> <https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore>
> pulsar-all images:
> https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.1-158d5eb/images/sha256-80ab1d748eff18655a9c247beba74aa107624b9d3e7cc3a2f22e1246f0d3de83?context=repo
> 
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
> 
> 
> 
> Regards
> Jiwei Guo (Tboy)
> 

[VOTE] Pulsar Release 3.1.3 Candidate 2

[Ran Gao]

This is the second release candidate for Apache Pulsar version 3.1.3.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.1.3+label%3Acherry-picked%2Fbranch-3.1+

*** Please download, test and verify on this release. This vote will stay
open for at least 24 hours ***

Note that we are verifying upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.3-candidate-2/

SHA-512 checksums:

057cc9dd391aab98e0607a7f240f96c191a3c195ab404572530dec0045725d11ffdeb491c55db6ad87e36cdb72748ad70160f2b768a9f2e94e7ba515f8c0a5ff
 apache-pulsar-3.1.3-bin.tar.gz
b657fc45e385ffd8b73372ec21616ea181d9bb04221864104ce9b507f993a2c6f69e3e2f617535473d5032815a564a7a472fd2f2187764ad9acd1eac1ffcdab4
 apache-pulsar-3.1.3-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1277

The tag to verify:
v3.1.3-candidate-2 (dcaf508f8e381107125e98722f4ddab76f9303ad)
https://github.com/apache/pulsar/commits/v3.1.3-candidate-2

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://downloads.apache.org/pulsar/KEYS

Docker images:

https://hub.docker.com/layers/gaoran10/pulsar/3.1.3-dcaf508/images/sha256-ef03162480a2710d8fc83dd22635ad401cc48c0ba8ebefa413bdfd794ae3af4f?context=repo

https://hub.docker.com/layers/gaoran10/pulsar-all/3.1.3-dcaf508/images/sha256-091214bca4985694a2e95d6e8432d3790c64e83af1ef0e4f2423833e66073685?context=repo


Please download the source package, and follow the README to build
and run the Pulsar standalone service.



Regards
Ran Gao (gaoran10)

Re: [DISCUSS] Retire pulsar-all Docker image and spin-off Python Functions runtime

[Ran Gao]

+1

Most users don't need all built-in connectors, it's too bloated.

Best Regards,
Ran Gao

On 2024/03/05 23:02:37 Matteo Merli wrote:
> The docker image `pulsar-all` is a convenience image that is created on top
> of the base `pulsar` image, including all the Pulsar IO connectors as well
> as the tiered storage offloaders.
> 
> The Dockerfile for `pulsar-all` can be found here:
> https://github.com/apache/pulsar/blob/master/docker/pulsar-all/Dockerfile
> 
> The resulting image is very big:
> 
> ```
> apachepulsar/pulsar-all   3.1.2
>  3d1aa250bf6c   2 months ago3.68GB
> ```
> 
> This poses a challenge in many ways:
>  1. Our CI pipeline needs to build these images and cache them across
> different stages of the pipeline
>  2. It takes a lot of time for release managers to build and push these
> images to Docker Hub
>  3. Users using this image in production see very long download times,
> something that can affect the availability of the system (eg: more chances
> of a 2nd broker to crash if a restart takes a very long time).
>  4. It's very unlikely that one user will require all the connectors, most
> likely, it would use just 2-3 of them.
> 
> The problem is that `pulsar-all` was introduced at a time when there were
> ~3 Pulsar IO connectors. Right now we do have 35 connectors, with a 1.9 GB
> total size.
> 
> The proposal here is to drop this image altogether. Users will be able to
> construct their own targeted images in a very simple way:
> 
> ```
> FROM apachepulsar/pulsar:latest
> RUN mkdir -p connectors && \
> cd connectors && \
> wget
> https://downloads.apache.org/pulsar/pulsar-3.2.0/connectors/pulsar-io-elastic-search-3.2.0.nar
> ```
> 
> 
> 
> ### Pulsar Functions Python Runtime
> 
> In order to support Python functions runtime, we have been including the
> Pulsar base image with quite a bit of dependencies, from `pulsar-client`
> Python SDK, to gRPC which is quite a heavy package with many transitive
> dependencies.
> 
> Given that the vast majority would be using the `pulsar` base image to run
> brokers and not python functions, it would make sense to split the Python
> support into a different image, like `pulsar-functions-python`, which
> extends from the base image and adds all the needed Python dependencies.
> 
> This way it will be very easy for users to select the appropriate image and
> we wouldn't be carrying a big amount of useless Python dependencies to
> users who don't need them.
> 
> 
> What are people's opinions with respect to this?
> 
> Matteo
> 
> --
> Matteo Merli
> 
> 

Re: [VOTE] Pulsar Release 3.1.3 Candidate 2

[Ran Gao]

Close this vote with 3 bindings, thanks for voting.

Regards,
gaoran10

On 2024/03/07 18:00:26 Ran Gao wrote:
> This is the second release candidate for Apache Pulsar version 3.1.3.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.1.3+label%3Acherry-picked%2Fbranch-3.1+
> 
> *** Please download, test and verify on this release. This vote will stay
> open for at least 24 hours ***
> 
> Note that we are verifying upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.3-candidate-2/
> 
> SHA-512 checksums:
> 
> 057cc9dd391aab98e0607a7f240f96c191a3c195ab404572530dec0045725d11ffdeb491c55db6ad87e36cdb72748ad70160f2b768a9f2e94e7ba515f8c0a5ff
>  apache-pulsar-3.1.3-bin.tar.gz
> b657fc45e385ffd8b73372ec21616ea181d9bb04221864104ce9b507f993a2c6f69e3e2f617535473d5032815a564a7a472fd2f2187764ad9acd1eac1ffcdab4
>  apache-pulsar-3.1.3-src.tar.gz
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1277
> 
> The tag to verify:
> v3.1.3-candidate-2 (dcaf508f8e381107125e98722f4ddab76f9303ad)
> https://github.com/apache/pulsar/commits/v3.1.3-candidate-2
> 
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
> 
> Docker images:
> 
> https://hub.docker.com/layers/gaoran10/pulsar/3.1.3-dcaf508/images/sha256-ef03162480a2710d8fc83dd22635ad401cc48c0ba8ebefa413bdfd794ae3af4f?context=repo
> 
> https://hub.docker.com/layers/gaoran10/pulsar-all/3.1.3-dcaf508/images/sha256-091214bca4985694a2e95d6e8432d3790c64e83af1ef0e4f2423833e66073685?context=repo
> 
> 
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
> 
> 
> 
> Regards
> Ran Gao (gaoran10)
> 

[ANNOUNCE] Apache Pulsar 3.1.3 released

[Ran Gao]

The Apache Pulsar team is proud to announce Apache Pulsar version 3.1.3.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:
https://pulsar.apache.org/download

Release Notes are at:
https://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Re: [DISCUSS] Upgrading to Jetty 12 since Jetty 9.4.x has reached end-of-line in June 2022

[Ran Gao]

3.3.x +1

The branch-3.2 is the latest branch, and the latest version is 3.2.2, it 
contains functionality improvements, bug fixes, and CVE fixes, if we happen to 
release a new version based on branch-3.3, why not add this change to it?

Best
- Ran Gao

On 2024/04/17 17:30:27 Lari Hotari wrote:
> I'd not like to do this change for 3.3.x, but the fact is that there's no 
> support for Jetty 9.4.x . 
> A new CVE in Jetty 9.4.x could cause a problem to us if the support policy is 
> actually followed in Jetty and users are forced to upgrade to get the fix. We 
> just have to tolerate this risk, I guess.
> 
> I agree that 3.3.x release preparations should start and we are already 
> behind schedule.
> It would be great to have BK 4.17.0 in Pulsar 3.3.x since it's not possible 
> to upgrade GRPC and Protobuf in Pulsar unless that first happens in BK. [1] 
> 
> -Lari
> 
> 1 - https://lists.apache.org/thread/s9g6w31vtwzgqf162hhlcr2nx3y68gv5
> 
> On 2024/04/17 15:40:30 Nicolò Boschi wrote:
> > I personally agree with NOT including it in 3.3.x
> > 
> > Following the release policy, it should be time to start preparing the
> > 3.3.x branch soon.
> > Also I wonder if the release line after 3.3.x will be 4.0, given that
> > 3.0 has been released almost 1y ago and the LTS should be cut every 18
> > months.
> > Having this upgrade in 4.0 seems like a good move to me.
> > 
> > Nicolò
> > 
> > Il giorno mer 17 apr 2024 alle ore 17:32 Zixuan Liu
> >  ha scritto:
> > >
> > > > do we first release 3.3.x asap and then tackle this upgrade for Pulsar
> > > 3.4.x
> > > +1
> > >
> > > Lari Hotari  于2024年4月17日周三 21:57写道：
> > >
> > > > Hi,
> > > >
> > > > Pulsar Broker and Pulsar Proxy use Jetty 9.4.x. Jetty 9.4.x has reached
> > > > end-of-line already in June 2022. Jetty 9.4.x has been receiving 
> > > > security
> > > > updates until now although it is not officially supported anymore. This
> > > > might change soon. That's why we need to start preparing to upgrade to
> > > > Jetty 12.
> > > >
> > > > It is recommended to skip Jetty 10 and Jetty 11 since they are also end 
> > > > of
> > > > support.
> > > > More details about this recommendation is in this Jetty GitHub issue
> > > > message by the Jetty maintainer:
> > > > https://github.com/jetty/jetty.project/issues/11644#issuecomment-2048516903
> > > > .
> > > >
> > > > Could we target Jetty 12 upgrade for the master branch so that it gets
> > > > included in Pulsar 3.3.x ?
> > > > Or, do we first release 3.3.x asap and then tackle this upgrade for 
> > > > Pulsar
> > > > 3.4.x?
> > > >
> > > > -Lari
> > > >
> > 
> 

Re: [DISCUSS] PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis

[Ran Gao]

+1

Best,
Ran Gao

On 2024/05/10 03:31:13 Yong Zhang wrote:
> Hi all,
> 
> I pushed a new proposal to allow to use -1 to disable
> the managedLedgerOffloadDeletionLagInMillis to not delete the data from
> bookkeeper.
> 
> Please take a look and share your thoughts. Thanks!
> 
> PIP: https://github.com/apache/pulsar/pull/22688
> 
> Best Regards,
> Yong
> 

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

[Ran Gao]

LGTM

Not all topics use the transaction feature, it can save a lot of resources to 
help reduce broker recovery time, and it's convenient to calculate topics count 
that use transactions.

Thanks,
Ran Gao

On 2024/08/21 14:27:25 Yubiao Feng wrote:
> Hi all
> 
> I drafted a proposal to add a topic's system prop that indicates whether
> users have published TXN messages in before.
> 
> https://github.com/apache/pulsar/pull/23210
> 
> I'm looking forward to hearing from you.
> 
> Thanks
> Yubiao Feng
> 

Re: [VOTE] PIP-222: Add CommandPartitionedTopicMetadata metrics

[Ran Gao]

+1 (unbinding)

Thanks,
Ran

On 2022/11/17 09:06:50 Jiuming Tao wrote:
> Dear Pulsar Community,
> 
> Please review and vote on this PIP.
> 
> PIP link: https://github.com/apache/pulsar/issues/18319 
> 
> 
> Discuss thread: 
> https://lists.apache.org/thread/sybl4nno4503w42hzt7b5lsyk6m2rbo6 
> 
> 
> Thanks,
> Tao Jiuming

Re: [DISCUSS] Change the default IO threads and listener threads of Java Client

[Ran Gao]

It seems this change will not affect the consumer, because the default executor 
is a single-thread executor service, increasing the listener threads number 
will allocate different single-thread executor services to different consumers.

On 2022/12/21 09:01:39 Yunze Xu wrote:
> I have a concern about the message ordering. If we have more than 1
> listener thread by default, could messages from the same topic be
> passed to different listener threads?
> 
> Thanks,
> Yunze
> 
> On Wed, Dec 21, 2022 at 11:12 AM 丛搏  wrote:
> >
> > +1
> > Our default configuration is best for most users. Multiple clients are
> > a few cases.
> >
> > Thanks,
> > Bo
> >
> > houxiaoyu  于2022年12月20日周二 16:02写道：
> > >
> > > +1
> > >
> > > This change might bring thread number increment in case users create many
> > > clients, but too many pulsar clients run in one machine is not a good use
> > > case I think,  so this change looks good to me.
> > >
> > > Thanks,
> > > Xiaoyu Hou
> > >
> > >  于2022年12月20日周二 12:25写道：
> > >
> > > > +1
> > > >
> > > > My concern is whether this change will affect some users who are 
> > > > creating
> > > > many clients. I think we can wait for other users to confirm it. (If 
> > > > this
> > > > will be affected, maybe we can give it a max_io_thread_num and then 
> > > > expand
> > > > the size from 1 to max_io_thread_num when adding a new consumer or 
> > > > producer)
> > > >
> > > >
> > > > Best,
> > > > Mattison
> > > > On Dec 20, 2022, 11:17 +0800, PengHui Li , wrote:
> > > > > Hi all,
> > > > >
> > > > > I noticed the Java Client (I haven't checked other clients) uses 1 IO
> > > > > thread and 1 listener
> > > > > thread by default. It will require users to update the thread
> > > > configuration
> > > > > if they have
> > > > > multiple cores and desired high throughput.
> > > > >
> > > > > Here is the example that we change to 16 IO threads in
> > > > > openmessaging benchmark
> > > > >
> > > > https://github.com/openmessaging/benchmark/blob/master/driver-pulsar/pulsar.yaml#L22
> > > > >
> > > > > We can apply the configuration of the threads based on the CPU cores. 
> > > > > So
> > > > > that for the
> > > > > most common cases, users don't need to touch the thread configuration.
> > > > >
> > > > > ```
> > > > > private int numIoThreads = Runtime.getRuntime().availableProcessors();
> > > > > private int numListenerThreads =
> > > > Runtime.getRuntime().availableProcessors();
> > > > > ```
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Thanks,
> > > > > Penghui
> > > >
> 

Re: [VOTE] Pulsar Node.js Client Release 1.8.0 Candidate 2

[Ran Gao]

+1 (non-binding)

- Checked npm package https://www.npmjs.com/package/pulsar-client/v/1.8.0-rc.2
- Checked source code + napi-darwin-unknown-arm64 work on macOS 12.6 (M1 Pro 
chip)

Thanks,
Ran Gao

On 2022/12/27 08:29:35 Zike Yang wrote:
> Hi everyone,
> 
> This is the second release candidate for the Apache Pulsar Node.js
> client, version 1.8.0.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar-client-node/milestone/13?closed=1
> 
> Please download the source files and review this release candidate:
> - Download the source package, verify shasum and asc
> - Follow the README.md to build and run the Pulsar Node.js client.
> 
> The rc package has been published to the npm registry:
> https://www.npmjs.com/package/pulsar-client/v/1.8.0-rc.2
> You can install it by `npm i pulsar-client@1.8.0-rc.2` and verify the package.
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Source files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.0-candidate-2/
> 
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> SHA-512 checksum:
> 80f9de8ca73b9a0bc7d083c6ba92e86721d381aa0689858270e074ff505f887533c40c5625b85a8b0943ced352e93eaafe67eb9bcb5afefadb4bd29b594620c4
>  pulsar-client-node-1.8.0.tar.gz
> 
> The tag to be voted upon:
> v1.8.0-rc.2
> https://github.com/apache/pulsar-client-node/releases/tag/v1.8.0-rc.2
> 
> Please review and vote on the release candidate #2 for the version
> 1.8.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> Thanks,
> Zike Yang
> 

Re: [DISCUSS] Introduce oshi library to sensory OS resources

[Ran Gao]

+1

Thanks,
Ran Gao

On 2022/12/21 09:38:01 Zike Yang wrote:
> +1
> 
> Thanks,
> Zike Yang
> 
> On Wed, Dec 21, 2022 at 5:06 PM Yunze Xu  wrote:
> >
> > +1
> >
> > Thanks,
> > Yunze
> >
> > On Wed, Dec 21, 2022 at 5:02 PM houxiaoyu  wrote:
> > >
> > > +1
> > >
> > > Yubiao Feng  于2022年12月21日周三 11:59写道：
> > >
> > > > +1
> > > >
> > > > On Mon, Dec 19, 2022 at 6:19 PM  wrote:
> > > >
> > > > >
> > > > > Hi, All
> > > > >
> > > > > I would like to introduce a new library oshi[1] to help Apache Pulsar
> > > > > sensory OS resources. It can help us to get away from the complex file
> > > > > manipulation and cross-platform compatibility issues in some operating
> > > > > systems.
> > > > >
> > > > > code example:   https://github.com/apache/pulsar/pull/18984
> > > > >
> > > > > Please feel free to left comments.
> > > > >
> > > > >
> > > > > Best,
> > > > > Mattison
> > > > >
> > > > >
> > > > > [1] https://github.com/oshi/oshi
> > > > >
> > > >
> 

Re: [VOTE] PIP-239: Retry Mechanism per Message title

[Ran Gao]

+1 (non-binding)

Thanks,
Ran Gao

On 2023/01/08 15:05:31 Zixuan Liu wrote:
> +1 (non-binding)
> 
> Thanks,
> Zixuan
> 
> Yunze Xu  于2023年1月8日周日 22:38写道：
> 
> > +1 (binding)
> >
> > Thanks,
> > Yunze
> >
> > On Sun, Jan 8, 2023 at 4:46 PM Zike Yang  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Zike Yang
> > >
> > > On Sun, Jan 8, 2023 at 3:22 PM Haiting Jiang 
> > wrote:
> > > >
> > > > +1 binding
> > > >
> > > > Although I think we should use more unique keys for system properties
> > in pulsar,
> > > > e.g. reserve all properties prefixed with "PULSAR_" for system
> > internal usage.
> > > > But it's another topic as we already have "RECONSUMETIMES".
> > > >
> > > >
> > > > On Sun, Jan 8, 2023 at 12:27 PM Nitin Goyal 
> > wrote:
> > > > >
> > > > > Hello community,
> > > > >
> > > > > I'm starting the VOTE for PIP-239: Retry Mechanism per Message title.
> > > > >
> > > > > Motivation: We are working on a project where each message has their
> > retry
> > > > > as per the requirements. like one message 100 times and other
> > messages 5
> > > > > times and so on. This feature also adds an extra functionality while
> > > > > comparing existing queueing services like RabbitMQ or SQS etc.
> > > > >
> > > > > For more details please find the detailed PIP at:
> > > > > https://github.com/apache/pulsar/issues/19136
> > > > >
> > > > > Discussion Threads:
> > > > > 1. https://lists.apache.org/thread/rn6114xxtx1j57yy2jbmdv4xzt80o1hz
> > > > > 2. https://lists.apache.org/thread/b9rfv6t307z439xx3zt2ym4p140qzp06
> > > > >
> > > > > Proposed changes in pulsar go client library.
> > > > > https://github.com/apache/pulsar-client-go/pull/939
> > > > >
> > > > > Thanks
> > > > > Nitin Goyal
> >
> 

Re: [VOTE] PIP-280 : Refactor CLI Argument Parsing Logic for Measurement Units using JCommander's custom converter

[Ran Gao]

+1 (non-binding)

Thanks,
Ran Gao

On 2023/07/07 09:25:22 Joo Hyuk Kim wrote:
> Hi community,
> 
> This PIP has received a couple of approvals in github PR link [1]
> So I thought it's time to vote.
> 
> ## Motivation
> 
> In the current Pulsar codebase, the logic to parse CLI arguments for
> measurement units like time and bytes is
> 
> scattered across various CLI classes. Each value read has its distinct
> parsing implementation, leading to a lack of code
> 
> reuse.
> 
> 
> ## Goals
> 
> 
> This PIP is to refactor the argument parsing logic to leverage the
> `@Parameter.converter`
> 
> functionality provided by JCommander [link 3]. This will isolate the
> measurement-specific parsing logic and increase
> 
> code
> 
> reusability.
> 
> 
> ### In Scope
> 
> 
> - Refactor all `Cmd` classes to utilize the converter functionality of
> JCommander. This will streamline the parsing
> 
>   logic and simplify the codebase.
> 
> - Refer to bottom section "Concrete Example", before "Links"
> 
> - Or on-going PR with small use case in
> https://github.com/apache/pulsar/pull/20663
> 
> 
> ## links
> 
> 
> [1] PR : https://github.com/apache/pulsar/pull/20691
> 
> 
> 
> Best regards,
> 
> JooHyukKim (Vince)
> 

[DISCUSS]PIP-417: Provide ability for Pulsar clients to integrate with third-party schema registry service

[Ran Gao]

Hi, Pulsar Community.

I opened a new PIP to provide the Pulsar client with the ability to
integrate with a third-party schema registry service. I'm looking
forward to your suggestions!

link: https://github.com/apache/pulsar/pull/24328

Thanks,
Ran Gao

-- 
This email and any attachments are intended solely for the recipient(s) 
named above and may contain confidential, privileged, or proprietary 
information. If you are not the intended recipient, you are hereby notified 
that any disclosure, copying, distribution, or reproduction of this 
information is strictly prohibited. If you have received this email in 
error, please notify the sender immediately by replying to this email and 
delete it from your system.

Re: [DISCUSS]PIP-417: Provide ability for Pulsar clients to integrate with third-party schema registry service

[Ran Gao]

Thanks for the suggestions, I'll add them to the PIP.

Thanks,
Ran Gao

On 2025/06/03 00:15:58 PengHui Li wrote:
> Thanks for the proposal.
> 
> The motivation looks good to me, users can connect to their preferred or
> customized schema registry with this proposal, which can get rid of the
> limitation from the Pulsar built-in schema registry.
> 
> I have a few questions about the migration or compatibility.
> 
> - For a topic that already uses pulsar's built-in schema, will the client
> be able to switch to an external schema registry? As I understand, we
> should reject this case since it will mess up the schema compatibility with
> 2 schema registries
> - And how about the old version(without external schema registry support)
> consumers connected to the topic that has schema from external schema
> registry?
> 
> We probably need to consider adding another schema type instead of using
> the bytes schema. If the topic has a schema from an external schema
> registry, which means the Pulsar broker will not manage schemas for this
> topic.
> 
> We should add more details about the compatibility to let users understand
> the proper way to move to the external schema registry.
> 
> Regards,
> Penghui
> 
> On Wed, May 21, 2025 at 9:23 AM Ran Gao  wrote:
> 
> > Hi, Pulsar Community.
> >
> > I opened a new PIP to provide the Pulsar client with the ability to
> > integrate with a third-party schema registry service. I'm looking
> > forward to your suggestions!
> >
> > link: https://github.com/apache/pulsar/pull/24328
> >
> > Thanks,
> > Ran Gao

Re: [DISCUSS] Whether to recycle failed recyclable objects

[Ran Gao]

Thanks for your explanation. Reusing the corrupted recycle object is dangerous, 
and it's hard to investigate issues from the heap dump.

Thanks,
Ran Gao

On 2025/07/17 13:38:38 Lari Hotari wrote:
> Recycling can cause bugs that are very hard to detect. The main reason to 
> avoid recycling in exceptional cases is to prevent such bugs. One possible 
> scenario is when another thread continues to hold a reference to an object 
> instance. After the object instance has been recycled, it could refer to a 
> completely different object instance. We have encountered such bugs in the 
> past with Pulsar, which is why it's better to avoid the risk of these bugs 
> altogether.
> 
> Netty doesn't recycle all object instances when recycling is enabled. There's 
> currently a per-thread limit of 4,096 object instances by default. Omitting 
> recycling in exceptional paths won't add noticeable extra GC pressure because 
> many instances will be discarded in any case, due to recycling cache overflow 
> when using default Netty settings.
> 
> -Lari
> 
> On 2025/07/17 12:47:22 Yunze Xu wrote:
> > Hi all,
> > 
> > I noticed there is a difference of opinion for whether to recycle
> > failed recyclable objects. [1][2][3] So I'd like to open the
> > discussion in the mail list.
> > 
> > Netty recycler is an object pool to reduce GC pressure for frequently
> > created and discarded objects. If a recyclable object fails, e.g. when
> > an `OpAddEntry` encounters an unexpected exception, IMO, it's
> > reasonable not to recycle it.
> > 
> > When a recyclable object is not recycled, it just won't be reused from
> > the object pool. The worst case is that the memory of the whole object
> > pool is exhausted. After that
> > 
> > It's true that if such objects are not recycled, there will be a
> > "memory leak". However, the worst result is that the pool memory is
> > exhausted. In this case, it will fall back to allocating objects via
> > the `newObject` method with a dummy handle [4]. The implementation of
> > this method usually allocates memory from JVM. In such cases, the
> > recyclable object will eventually be garbage collected.
> > 
> > It's actually not a memory leak. Recyclable objects usually have short
> > life timing, for example, an `OpAddEntry` object is created when
> > starting an asynchronous send and recycled after the send is done. The
> > object is never referenced by any other object, so even if it's
> > allocated from JVM, it will eventually be garbage collected.
> > 
> > The benefit to skip recycling objects for failures is that for rare
> > cases, retaining the object could help diagnose issues that are hard
> > to reproduce. Still take `OpAddEntry` for example, if it encountered
> > an unexpected exception, recycling it could set all fields with null,
> > so it's hard to know which ledger it belongs to (and other useful
> > fields) from the heap dump.
> > 
> > [1] https://github.com/apache/pulsar/pull/24515#discussion_r2212602435
> > [2] https://github.com/apache/pulsar/pull/24522#discussion_r2210814071
> > [3] https://github.com/apache/pulsar/pull/24522#discussion_r2213100905
> > [4] 
> > https://github.com/netty/netty/blob/ab2d2cf3ff3a6f055368405af7f6e9dd5b8d144e/common/src/main/java/io/netty/util/Recycler.java#L189
> > 
> > Thanks,
> > Yunze
> > 
>