Re: [DISCUSS] Pulsar Year in Review blog post

[Julien Jakubowski]

Hi,
The blog post is published! Thank you everyone
https://pulsar.apache.org/blog/2024/01/12/pulsar-2023-year-in-review/

On Mon, Jan 22, 2024 at 5:52 PM Julien Jakubowski <
julien.jakubow...@streamnative.io> wrote:

> Thank you! I added your suggestion. I'd love the community to add more
> input. Otherwise, let's merge the PR tomorrow morning.
>
> On Mon, Jan 22, 2024 at 4:14 PM Lari Hotari  wrote:
>
>> I added a comment about "What’s next in 2024". It would be great to
>> have more concrete and interesting things highlighted since the
>> current content isn't very convincing.
>> https://github.com/apache/pulsar-site/pull/770#discussion_r1461997301
>> One possibility would be to look at the in-progress PIPs and add some
>> content based on that.
>> I made one opinionated suggestion about "PIP-322: Pulsar Rate Limiting
>> Refactoring". :)
>> I guess we don't have to get stuck on this point and let's just get
>> the blog post published. :)
>>
>> -Lari
>>
>> On Mon, 22 Jan 2024 at 15:34, Julien Jakubowski
>>  wrote:
>> >
>> > Thank you, Lari. And thank you, Chris, Kiryl & Dave for your input.
>> > Just a reminder for the community: could you please give me your input
>> by
>> > EOD?
>> > I believe the blog post should be published ASAP because it's about
>> 2023,
>> > and we're already at the end of January :-)
>> > Thanks,
>> > Julien
>> >
>> > On Thu, Jan 18, 2024 at 4:40 PM Lari Hotari  wrote:
>> >
>> > > Thank you, Julien.
>> > >
>> > > This sounds like a good plan. It's great to share news about all
>> Apache
>> > > Pulsar OSS community related projects, not just about the official ASF
>> > > Apache Pulsar projects. There are many exciting things that happened
>> last
>> > > years around Apache Pulsar and putting that in a blog post is a great
>> way
>> > > to get the word out there.
>> > >
>> > > -Lari
>> > >
>> > > On 2024/01/18 12:29:38 Julien Jakubowski wrote:
>> > > > Hi Pulsar Community,
>> > > >
>> > > > I am working on a blog post titled 'Apache Pulsar 2023 Year in
>> Review,'
>> > > > similar to the one we published last year:
>> > > >
>> https://pulsar.apache.org/blog/2023/01/10/pulsar-2022-year-in-review/.
>> > > > Here is my first draft:
>> https://github.com/apache/pulsar-site/pull/770.
>> > > >
>> > > > I would greatly appreciate your contributions to this blog post,
>> > > especially
>> > > > any news about the OSS Pulsar community that you think should be
>> shared.
>> > > > Additionally, I welcome any feedback for improvements.
>> > > >
>> > > > Could you please provide your contributions by the end of the day on
>> > > > Monday? Please feel free to just reply to this thread with your
>> input.
>> > > >
>> > > > As Dave mentioned, it's essential to clearly distinguish what is
>> part of
>> > > > Apache Pulsar and what is not. Last year, we included a 'Pulsar
>> > > Ecosystem'
>> > > > section highlighting contributions external to Pulsar but still
>> within
>> > > its
>> > > > ecosystem. I think we should keep this approach this year if you all
>> > > agree.
>> > > >
>> > > > I look forward to your contributions.
>> > > >
>> > > > Thanks,
>> > > > Julien
>> > > >
>> > >
>>
>

Re: [VOTE] Pulsar Client Go Release 0.12.0 Candidate 3

[PengHui Li]

+1 (binding)

- Build from source
- Run tests on 3.1.2
- Run tests on 3.0.2

Regards,
Penghui

On Wed, Jan 24, 2024 at 2:59 PM Zike Yang  wrote:

> Hi everyone,
> Please review and vote on the release candidate #3 for the version
> 0.12.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> This is the third release candidate for Apache Pulsar Go client,
> version 0.12.0.
>
> The release note/changelog for Go client 0.12.0:
> https://github.com/apache/pulsar-client-go/pull/1153/files
>
> Pulsar Client Go's KEYS file contains PGP keys we used to sign this
> release:
> https://downloads.apache.org/pulsar/KEYS
>
> Please download these packages and review this release candidate:
> - Review release notes:
> https://github.com/apache/pulsar-client-go/pull/1153
> - Download the source package (verify shasum, and asc) and follow the
> README.md to build and run the pulsar-client-go.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Source file:
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-3/
>
> The tag to be voted upon:
> v0.12.0-candidate-3
> https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-3
>
> SHA-512 checksums:
>
> 66c37c4439549a02d7e3d21ecc02c4a2f44bad2232fbf15fb04c42f02fc4caad758b7a1b1663d0df47414af99daa4e3a19d33c29c0decbc43ac4d17e46a126bc
>  apache-pulsar-client-go-0.12.0-src.tar.gz
>

Re: [VOTE] Release Apache Pulsar Helm Chart 3.2.0 based on 3.2.0-candidate-3

[PengHui Li]

+1 (binding)

- Deploy pulsar cluster on local docker-desktop(with Kubernetes)
- Checked the signature
- Checked the license

Regards,
Penghui

On Sun, Jan 21, 2024 at 4:45 PM Lari Hotari  wrote:

> Hello Apache Pulsar Community,
>
> This is a call for the vote to release the Apache Pulsar Helm Chart
> version 3.2.0.
>
> Release notes for 3.2.0-candidate-3:
>
> https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-3.2.0-candidate-3
>
> The release candidate is available at:
> https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.2.0-candidate-3/
>
> pulsar-chart-3.2.0-source.tar.gz - is the "main source release".
> pulsar-3.2.0.tgz - is the binary Helm Chart release.
>
> Public keys are available at: https://www.apache.org/dist/pulsar/KEYS
>
> For convenience "index.yaml" has been uploaded (though excluded from
> voting), so you can also run the below commands.
>
> helm repo add --force-update apache-pulsar-dist-dev
> https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.2.0-candidate-3/
> helm repo update
> helm install pulsar apache-pulsar-dist-dev/pulsar --version 3.2.0 --set
> affinity.anti_affinity=false
>
> pulsar-3.2.0.tgz.prov - is also uploaded for verifying Chart Integrity,
> though it is not strictly required for releasing the artifact based on ASF
> Guidelines.
>
> You can optionally verify this file using this helm plugin
> https://github.com/technosophos/helm-gpg, or by using helm --verify (
> https://helm.sh/docs/helm/helm_verify/).
>
> helm fetch --prov apache-pulsar-dist-dev/pulsar
> helm plugin install https://github.com/technosophos/helm-gpg
> helm gpg verify pulsar-3.2.0.tgz
>
> The vote will be open for at least 72 hours.
>
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
>
> For license checks, the .rat-excludes files is included, so you can run
> the following to verify licenses (just update ):
>
> tar -xvf pulsar-chart-3.2.0-source.tar.gz
> cd pulsar-chart-3.2.0
> java -jar /apache-rat-0.15/apache-rat-0.15.jar . -E .rat-excludes
>
> Please note that the version number excludes the `-candidate-X` string, so
> it's now
> simply 3.2.0. This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release it.
>
> Thanks,
> Lari
>

Re: Ability to decrease partition count in pulsar

[Lari Hotari]

Hi Girish,

Very useful proposal.

Would it be possible to enable comments on the Google Doc? It's pretty
hard to comment on the doc since copying is also disabled.

In the scope definition 4.2,
"The initial scope is to target unordered consumption flows. Even in
the current world, there are challenges with normal partition scale up
for ordered consumption based topics, so keeping the partition scale
down out of scope for that as well."

If we don't care about ordered consumption and re-keying, I guess the
feature isn't very hard to implement.
Pulsar already contains the topic termination feature which will let
consumers to consume messages while publishers cannot publish more
messages. This is the "ready-only topic" feature that could be used as
one of the building blocks for implementing the decrease of the
partition count for a topic.

For the final design, it would be great to have a design for ordered
consumption flows. It might not be trivial to design it. I happened to
be at a local Kafka meetup a few months ago and this particular
challenge was discussed in the context of Kafka and how painful it is
to handle manually and what problems could happen in production when
large scale streaming applications assume that a specific key is
contained in a specific partition.

There's a similar challenge also when the number of partitions are
increased so this problem isn't specific to decreasing partitions.
In ordered consumption flows, there is most likely an ordering key and
a specific key is assigned to a specific partition. If the partition
count changes, there would have to be some rekeying/reassignment that
happens.

I guess it should be part of the metadata of the message key to know
what partition count did the producing application use when assigning
the message to a specific partition. That information could be used in
consuming application to detect when keying or number of partitions
changes. This feels like a generic problem so perhaps we should look
for existing ways to solve this challenge in streaming applications.

-Lari

On Wed, 24 Jan 2024 at 09:00, Girish Sharma  wrote:
>
> Bumping this up! Hoping this can be discussed so that I get rule out that
> this approach has any fatal flaws.
>
> Regards
>
> On Fri, Jan 19, 2024 at 11:58 AM Girish Sharma 
> wrote:
>
> > Hello everyone,
> >
> > A a true cloud native platform, which supports scale up and scale down, I
> > feel like there is a need to be able to reduce partition count in pulsar to
> > truly achieve a scale down after events like sales (akin to black friday,
> > etc) or huge temporary publish burst due to backfill.
> >
> > I looked through the archives (upto 2021) and did not find any prior
> > discussion on the same topic.
> >
> > I have given this an initial thought to figure out what would it need to
> > support such a feature in the lowest footprint possible. I am attaching the
> > document explaining the need, requirements and initial high level details
> > [0]. What I would like is to understand if the community also finds this
> > feature helpful and does the approach described in the document have some
> > fatal flaw? Summarizing the approach here as well:
> >
> >- Introduce an ability to convert a normal topic object into a
> >read-only topic via admin api and an additional partitioned-topic 
> > metadata
> >property (just like shadow source, etc)
> >- Add logic to block produce but allow new consumers and dispatch call
> >based on this flag
> >- Add logic in GC to clean out read only topics when all of their
> >ledgers expire (TTL/retention)
> >
> > Goal is that there is no data movement involved and no impact on existing
> > partitions during this scale down.
> >
> > Looking forward to the discussion.
> >
> > [0]
> > https://docs.google.com/document/d/1sbGQSwDihQftIRsxAXg5Zm4uxKQ0kRk9HadKYRFTswI/edit?usp=sharing
> >
> > Regards
> > --
> > Girish Sharma
> >
>
>
> --
> Girish Sharma

Re: Ability to decrease partition count in pulsar

[Girish Sharma]

Hello Lari, thanks for the comments. replies inline.


On Wed, Jan 24, 2024 at 7:36 PM Lari Hotari  wrote:

> Hi Girish,
>
> Very useful proposal.
>
> Would it be possible to enable comments on the Google Doc? It's pretty
> hard to comment on the doc since copying is also disabled.
>
> I've enabled them now. Thank you for going through the doc.


> In the scope definition 4.2,
> "The initial scope is to target unordered consumption flows. Even in
> the current world, there are challenges with normal partition scale up
> for ordered consumption based topics, so keeping the partition scale
> down out of scope for that as well."
>
> If we don't care about ordered consumption and re-keying, I guess the
> feature isn't very hard to implement.
> Pulsar already contains the topic termination feature which will let
> consumers to consume messages while publishers cannot publish more
> messages. This is the "ready-only topic" feature that could be used as
> one of the building blocks for implementing the decrease of the
> partition count for a topic.
>

Yes, terminated topic is already very close to the read-only topic barring
the grace period and maybe the scope of un-terminating a topic. I will
merge my read-only with the existing terminate API/feature.


>
> For the final design, it would be great to have a design for ordered
> consumption flows. It might not be trivial to design it. I happened to
> be at a local Kafka meetup a few months ago and this particular
> challenge was discussed in the context of Kafka and how painful it is
> to handle manually and what problems could happen in production when
> large scale streaming applications assume that a specific key is
> contained in a specific partition.
>
> There's a similar challenge also when the number of partitions are
> increased so this problem isn't specific to decreasing partitions.
> In ordered consumption flows, there is most likely an ordering key and
> a specific key is assigned to a specific partition. If the partition
> count changes, there would have to be some rekeying/reassignment that
> happens.
>
> I agree that this is an existing problem in both kafka and pulsar for both
partition count scale up (and scale down in kafka via re-mapping). For that
purpose, I've kept it out of scope. But what I would ensure is that adding
this new feature of partitions scale down is not increasing the complexity
or difficulty of providing seamless partition count change for ordered
consumption in future.


-- 
Girish Sharma

Re: [VERIFY] Pulsar Release 3.2.0 Candidate 4

[Lari Hotari]

+1 (binding)

- checked the source and binary files
  - checksums matched
  - signatures were ok
- ran validation steps in 
https://pulsar.apache.org/contribute/validate-release-candidate/

-Lari



On 2024/01/22 14:43:14 guo jiwei wrote:
> This is the fourth release candidate for Apache Pulsar version 3.2.0.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar/milestone/36?closed=1
> 
> *** Please download, test and verify on this release. This release
> candidate verification will stay open until Jan 28 ***
> 
> Note that we are verifying upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.0-candidate-4/
> 
> SHA-512 checksums:
> 
> de4beeb266dea2bdce0b6a3cf213bd36235f89003ed42cb212eb2eab2c784255ca8d82a4c8ed7d02f1ae7598dae114f7717f0828af41ce15573ab2606b0205f9
> 
> apache-pulsar-3.2.0-bin.tar.gz
> 
> 0e7d0f3043b85d54a3b346d5b702db358437ee178596967f02d79472ce2797d46a4af4692463c349187afed2ea8f084b071e5711b311fd332d23ac4f84a3d895
> 
> apache-pulsar-3.2.0-src.tar.gz
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1264/
> 
> The tag to verify:
> v3.2.0-candidate-4 (bca08d98ab742d241d31b2e8d7673273605b2e16)
> https://github.com/apache/pulsar/commits/v3.2.0-candidate-4/
> 
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Docker images:
> 
> pulsar images:
> https://hub.docker.com/layers/technoboy8/pulsar/3.2.0-bca08d9/images/sha256-714e977706b44a2ed9376a8d7328ab1bf72217553e9c127ef9f7a33c0f3d8e4b?context=repo
> 
> pulsar-all images:
> https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.0-bca08d9/images/sha256-668ab662d6506bc154a12da5359b944d8115596e8d09035c7213fe073fbf5078?context=repo
> 
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
> 
> Note that this RC doesn't require a formal vote, but we would also
> appreciate your feedback with +1/-1. And please provide specific
> comments if your feedback is not +1.
> 
> 
> Regards
> Jiwei Guo (Tboy)
> 

Re: [VOTE] Release Apache Pulsar Helm Chart 3.2.0 based on 3.2.0-candidate-3

[Lari Hotari]

+1 (binding)

-Lari

On 2024/01/21 08:45:04 Lari Hotari wrote:
> Hello Apache Pulsar Community,
> 
> This is a call for the vote to release the Apache Pulsar Helm Chart version 
> 3.2.0.
> 
> Release notes for 3.2.0-candidate-3:
> https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-3.2.0-candidate-3
> 
> The release candidate is available at:
> https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.2.0-candidate-3/
> 
> pulsar-chart-3.2.0-source.tar.gz - is the "main source release".
> pulsar-3.2.0.tgz - is the binary Helm Chart release.
> 
> Public keys are available at: https://www.apache.org/dist/pulsar/KEYS
> 
> For convenience "index.yaml" has been uploaded (though excluded from voting), 
> so you can also run the below commands.
> 
> helm repo add --force-update apache-pulsar-dist-dev 
> https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.2.0-candidate-3/
> helm repo update
> helm install pulsar apache-pulsar-dist-dev/pulsar --version 3.2.0 --set 
> affinity.anti_affinity=false
> 
> pulsar-3.2.0.tgz.prov - is also uploaded for verifying Chart Integrity, 
> though it is not strictly required for releasing the artifact based on ASF 
> Guidelines. 
> 
> You can optionally verify this file using this helm plugin 
> https://github.com/technosophos/helm-gpg, or by using helm --verify 
> (https://helm.sh/docs/helm/helm_verify/).
> 
> helm fetch --prov apache-pulsar-dist-dev/pulsar
> helm plugin install https://github.com/technosophos/helm-gpg
> helm gpg verify pulsar-3.2.0.tgz
> 
> The vote will be open for at least 72 hours.
> 
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
> 
> For license checks, the .rat-excludes files is included, so you can run the 
> following to verify licenses (just update ):
> 
> tar -xvf pulsar-chart-3.2.0-source.tar.gz
> cd pulsar-chart-3.2.0
> java -jar /apache-rat-0.15/apache-rat-0.15.jar . -E .rat-excludes
> 
> Please note that the version number excludes the `-candidate-X` string, so 
> it's now
> simply 3.2.0. This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release it.
> 
> Thanks,
> Lari
> 

Re: [VERIFY] Pulsar Release 3.2.0 Candidate 4

[guo jiwei]

Thanks Lari.
We have confirmed there is a break change from the function part.
Before, the output of querystate was

  "key": "hello",
  "numberValue": 20,
  "version": 19

And now, it was changed to

{
  "key": "hello",
  "stringValue": "\u\u\u\u\u\u\u\n",
  "numberValue": 10
}


We need to drop this candidate, I will raise a new one when it is fixed.


Regards
Jiwei Guo (Tboy)


On Thu, Jan 25, 2024 at 12:17 AM Lari Hotari  wrote:

> +1 (binding)
>
> - checked the source and binary files
>   - checksums matched
>   - signatures were ok
> - ran validation steps in
> https://pulsar.apache.org/contribute/validate-release-candidate/
>
> -Lari
>
>
>
> On 2024/01/22 14:43:14 guo jiwei wrote:
> > This is the fourth release candidate for Apache Pulsar version 3.2.0.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar/milestone/36?closed=1
> >
> > *** Please download, test and verify on this release. This release
> > candidate verification will stay open until Jan 28 ***
> >
> > Note that we are verifying upon the source (tag), binaries are provided
> for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.0-candidate-4/
> >
> > SHA-512 checksums:
> >
> >
> de4beeb266dea2bdce0b6a3cf213bd36235f89003ed42cb212eb2eab2c784255ca8d82a4c8ed7d02f1ae7598dae114f7717f0828af41ce15573ab2606b0205f9
> >
> > apache-pulsar-3.2.0-bin.tar.gz
> >
> >
> 0e7d0f3043b85d54a3b346d5b702db358437ee178596967f02d79472ce2797d46a4af4692463c349187afed2ea8f084b071e5711b311fd332d23ac4f84a3d895
> >
> > apache-pulsar-3.2.0-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1264/
> >
> > The tag to verify:
> > v3.2.0-candidate-4 (bca08d98ab742d241d31b2e8d7673273605b2e16)
> > https://github.com/apache/pulsar/commits/v3.2.0-candidate-4/
> >
> > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Docker images:
> >
> > pulsar images:
> >
> https://hub.docker.com/layers/technoboy8/pulsar/3.2.0-bca08d9/images/sha256-714e977706b44a2ed9376a8d7328ab1bf72217553e9c127ef9f7a33c0f3d8e4b?context=repo
> > <
> https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore
> >
> > pulsar-all images:
> >
> https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.0-bca08d9/images/sha256-668ab662d6506bc154a12da5359b944d8115596e8d09035c7213fe073fbf5078?context=repo
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> > Note that this RC doesn't require a formal vote, but we would also
> > appreciate your feedback with +1/-1. And please provide specific
> > comments if your feedback is not +1.
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
>