[ANNOUNCE] Apache Pulsar 3.2.0 released

[guo jiwei]

The Apache Pulsar team is proud to announce Apache Pulsar version 3.2.0.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
https://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards
Jiwei Guo (Tboy)

Re: [ANNOUNCE] Apache Pulsar 3.2.0 released

[Enrico Olivelli]

Congrats ! this is a great milestone

Thank you for driving the release

Enrico

Il giorno lun 5 feb 2024 alle ore 13:52 guo jiwei
 ha scritto:
>
> The Apache Pulsar team is proud to announce Apache Pulsar version 3.2.0.
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> For Pulsar release details and downloads, visit:
>
> https://pulsar.apache.org/download
>
> Release Notes are at:
> https://pulsar.apache.org/release-notes
>
> We would like to thank the contributors that made the release possible.
>
> Regards
> Jiwei Guo (Tboy)

Re: [DISCUSS] PIP-335: Oxia metadata support

[Matteo Merli]

On Sun, Feb 4, 2024 at 11:55 AM Dave Fisher  wrote:

> 2. Other questions. If the base Pulsar will now depend on Oxia and no
> longer use Zookeeper? Or is Oxia an option? Will Oxia be included as a
> Pulsar dependency in Pulsar convenience binaries?
>

It stated clearly in the proposal that: "It's not in the scope of this
proposal to change any default behavior or configuration of Pulsar."

This is an extra option.

Oxia server does not need to be included in Pulsar distribution, the Helm
chart will be able to point to Oxia server Docker images.

> Can you explain how this would be *any* different here?
>
> Adding dependencies should be more than getting the LICENSE and NOTICE
> correct. It should be a deliberate choice.
>
> What’s a little different here is that Oxia may not be tracked for CVEs by
> dependabot. Another difference I’m OK with is that it is maintained by so
> many who are also members of this community.
>

It's already tracking CVEs for all dependencies and Oxia images are
automatically scanned for vulnerabilities.


Thanks,
Matteo

Re: [DISCUSS] PIP-335: Oxia metadata support

[Matteo Merli]

Thanks to everyone,

moving to VOTE thread.


--
Matteo Merli



On Mon, Feb 5, 2024 at 11:13 AM Matteo Merli  wrote:

>
> On Sun, Feb 4, 2024 at 11:55 AM Dave Fisher  wrote:
>
>> 2. Other questions. If the base Pulsar will now depend on Oxia and no
>> longer use Zookeeper? Or is Oxia an option? Will Oxia be included as a
>> Pulsar dependency in Pulsar convenience binaries?
>>
>
> It stated clearly in the proposal that: "It's not in the scope of this
> proposal to change any default behavior or configuration of Pulsar."
>
> This is an extra option.
>
> Oxia server does not need to be included in Pulsar distribution, the Helm
> chart will be able to point to Oxia server Docker images.
>
> > Can you explain how this would be *any* different here?
>>
>> Adding dependencies should be more than getting the LICENSE and NOTICE
>> correct. It should be a deliberate choice.
>>
>> What’s a little different here is that Oxia may not be tracked for CVEs
>> by dependabot. Another difference I’m OK with is that it is maintained by
>> so many who are also members of this community.
>>
>
> It's already tracking CVEs for all dependencies and Oxia images are
> automatically scanned for vulnerabilities.
>
>
> Thanks,
> Matteo
>
>

[VOTE] PIP-335: Oxia metadata support

[Matteo Merli]

https://github.com/apache/pulsar/pull/22009

-

# PIP-335: Support Oxia metadata store plugin

# Motivation

Oxia is a scalable metadata store and coordination system that can be used
as the core infrastructure
to build large scale distributed systems.

Oxia was created with the primary goal of providing an alternative Pulsar
to replace ZooKeeper as the
long term preferred metadata store, overcoming all the current limitations
in terms of metadata
access throughput and data set size.

# Goals

Add a Pulsar MetadataStore plugin that uses Oxia client SDK.

Users will be able to start a Pulsar cluster using just Oxia, without any
ZooKeeper involved.

## Not in Scope

It's not in the scope of this proposal to change any default behavior or
configuration of Pulsar.

# Detailed Design

## Design & Implementation Details

Oxia semantics and client SDK were already designed with Pulsar and
MetadataStore plugin API in mind, so
there is not much integration work that needs to be done here.

Just few notes:
 1. Oxia client already provides support for transparent batching of read
and write operations,
so there will be no use of the batching logic in
`AbstractBatchedMetadataStore`
 2. Oxia does not treat keys as a walkable file-system like interface, with
directories and files. Instead
all the keys are independent. Though Oxia sorting of keys is aware of
'/' and provides efficient key
range scanning operations to identify the first level children of a
given key
 3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
exist. eg: we can create `/a/b/c` key
without `/a/b` and `/a` existing.
In the Pulsar integration for Oxia we're forcing to create all parent
keys when they are not there. This
is due to several places in BookKeeper access where it does not create
the parent keys, though it will
later make `getChildren()` operations on the parents.

## Other notes

Unlike in the ZooKeeper implementation, the notification of events is
guaranteed in Oxia, because the Oxia
client SDK will use the transaction offset after server reconnections and
session restarted events. This
will ensure that brokers cache will always be properly invalidated. We will
then be able to remove the
current 5minutes automatic cache refresh which is in place to prevent the
ZooKeeper missed watch issue.

# Links

Oxia: https://github.com/streamnative/oxia
Oxia Java Client SDK: https://github.com/streamnative/oxia-java


--
Matteo Merli

Re: [VOTE] Pulsar Node.js Client Release 1.10.0 Candidate 1

[Yubiao Feng]

+1 (binding)

- verified checksums and signatures
- confirmed that the build was successful
- ran producer/consumer

Thanks
Yubaio Feng

On Mon, Jan 29, 2024 at 6:11 PM Baodi Shi  wrote:

> Hi everyone,
>
> This is the first release candidate for Apache Pulsar Node.js client,
> version 1.10.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar-client-node/milestone/15?closed=1
>
> Please download the source files and review this release candidate:
> - Download the source package, verify shasum and asc
> - Follow the README.md to build and run the Pulsar Node.js client.
>
> The release candidate package has been published to the npm registry:
> https://www.npmjs.com/package/pulsar-client/v/1.10.0-rc.1
> You can install it by `npm i pulsar-client@1.10.0-rc.1
> --pulsar_binary_host_mirror=
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>  and
> verify the package.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Source files:
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.10.0-rc.1/
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
>
> SHA-512 checksum:
>
> 3e2c76ff8340597129eec1ab2a9cdbdd48fb162257426c474bd637bb02a45b49011e6b31e08679022f18d700e73f49cbb8adc59f75e345db9220a2b755eccac4
> ./apache-pulsar-client-node-1.10.0.tar.gz
>
> The tag to be voted upon:
> v1.10.0-rc.1
> https://github.com/apache/pulsar-client-node/releases/tag/v1.10.0-rc.1
>
> Please review and vote on the release candidate #1 for the version 1.10.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>

Re: [VOTE] PIP-335: Oxia metadata support

[Haiting Jiang]

+1

Thanks,
Haiting

On Tue, Feb 6, 2024 at 4:40 AM Matteo Merli  wrote:
>
> https://github.com/apache/pulsar/pull/22009
>
> -
>
> # PIP-335: Support Oxia metadata store plugin
>
> # Motivation
>
> Oxia is a scalable metadata store and coordination system that can be used
> as the core infrastructure
> to build large scale distributed systems.
>
> Oxia was created with the primary goal of providing an alternative Pulsar
> to replace ZooKeeper as the
> long term preferred metadata store, overcoming all the current limitations
> in terms of metadata
> access throughput and data set size.
>
> # Goals
>
> Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
>
> Users will be able to start a Pulsar cluster using just Oxia, without any
> ZooKeeper involved.
>
> ## Not in Scope
>
> It's not in the scope of this proposal to change any default behavior or
> configuration of Pulsar.
>
> # Detailed Design
>
> ## Design & Implementation Details
>
> Oxia semantics and client SDK were already designed with Pulsar and
> MetadataStore plugin API in mind, so
> there is not much integration work that needs to be done here.
>
> Just few notes:
>  1. Oxia client already provides support for transparent batching of read
> and write operations,
> so there will be no use of the batching logic in
> `AbstractBatchedMetadataStore`
>  2. Oxia does not treat keys as a walkable file-system like interface, with
> directories and files. Instead
> all the keys are independent. Though Oxia sorting of keys is aware of
> '/' and provides efficient key
> range scanning operations to identify the first level children of a
> given key
>  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> exist. eg: we can create `/a/b/c` key
> without `/a/b` and `/a` existing.
> In the Pulsar integration for Oxia we're forcing to create all parent
> keys when they are not there. This
> is due to several places in BookKeeper access where it does not create
> the parent keys, though it will
> later make `getChildren()` operations on the parents.
>
> ## Other notes
>
> Unlike in the ZooKeeper implementation, the notification of events is
> guaranteed in Oxia, because the Oxia
> client SDK will use the transaction offset after server reconnections and
> session restarted events. This
> will ensure that brokers cache will always be properly invalidated. We will
> then be able to remove the
> current 5minutes automatic cache refresh which is in place to prevent the
> ZooKeeper missed watch issue.
>
> # Links
>
> Oxia: https://github.com/streamnative/oxia
> Oxia Java Client SDK: https://github.com/streamnative/oxia-java
>
>
> --
> Matteo Merli
> 

Re: [DISCUSS] PIP-334:Create a new admin API: pck (Pulsar Consistency Checker)

[Haiting Jiang]

The motivation LGTM.
But I am not sure if it's possible to implement at client side,
without adding a new admin API?
My main concern is this method may run for quite a while. And it's not
easy to see the progress if it's implemented on the server.

Thanks,
Haiting

On Tue, Jan 30, 2024 at 6:50 PM Jie crossover  wrote:
>
> Hi dev,
> I proposed a PIP: https://github.com/apache/pulsar/pull/21988
>
> Sometimes we need to check if there are any orphan ledgers in the
> bookkeeper.
> So I'm planning to create an admin API.
>
> Specific implementation logic is referred to in this link
> 
> .
>
> Please take a look and give your feedback.
> --
> Best Regards！
> crossoverJie

Re: [VOTE] PIP-335: Oxia metadata support

[Hang Chen]

+1

Best,
Hang

Haiting Jiang  于2024年2月6日周二 11:22写道：
>
> +1
>
> Thanks,
> Haiting
>
> On Tue, Feb 6, 2024 at 4:40 AM Matteo Merli  wrote:
> >
> > https://github.com/apache/pulsar/pull/22009
> >
> > -
> >
> > # PIP-335: Support Oxia metadata store plugin
> >
> > # Motivation
> >
> > Oxia is a scalable metadata store and coordination system that can be used
> > as the core infrastructure
> > to build large scale distributed systems.
> >
> > Oxia was created with the primary goal of providing an alternative Pulsar
> > to replace ZooKeeper as the
> > long term preferred metadata store, overcoming all the current limitations
> > in terms of metadata
> > access throughput and data set size.
> >
> > # Goals
> >
> > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> >
> > Users will be able to start a Pulsar cluster using just Oxia, without any
> > ZooKeeper involved.
> >
> > ## Not in Scope
> >
> > It's not in the scope of this proposal to change any default behavior or
> > configuration of Pulsar.
> >
> > # Detailed Design
> >
> > ## Design & Implementation Details
> >
> > Oxia semantics and client SDK were already designed with Pulsar and
> > MetadataStore plugin API in mind, so
> > there is not much integration work that needs to be done here.
> >
> > Just few notes:
> >  1. Oxia client already provides support for transparent batching of read
> > and write operations,
> > so there will be no use of the batching logic in
> > `AbstractBatchedMetadataStore`
> >  2. Oxia does not treat keys as a walkable file-system like interface, with
> > directories and files. Instead
> > all the keys are independent. Though Oxia sorting of keys is aware of
> > '/' and provides efficient key
> > range scanning operations to identify the first level children of a
> > given key
> >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> > exist. eg: we can create `/a/b/c` key
> > without `/a/b` and `/a` existing.
> > In the Pulsar integration for Oxia we're forcing to create all parent
> > keys when they are not there. This
> > is due to several places in BookKeeper access where it does not create
> > the parent keys, though it will
> > later make `getChildren()` operations on the parents.
> >
> > ## Other notes
> >
> > Unlike in the ZooKeeper implementation, the notification of events is
> > guaranteed in Oxia, because the Oxia
> > client SDK will use the transaction offset after server reconnections and
> > session restarted events. This
> > will ensure that brokers cache will always be properly invalidated. We will
> > then be able to remove the
> > current 5minutes automatic cache refresh which is in place to prevent the
> > ZooKeeper missed watch issue.
> >
> > # Links
> >
> > Oxia: https://github.com/streamnative/oxia
> > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> >
> >
> > --
> > Matteo Merli
> >