Re: [VOTE] PIP-120: Enable client memory limit by default

[Enrico Olivelli]

1 (binding)

Enrico

Il giorno mer 22 dic 2021 alle ore 03:39 mattison chao <
mattisonc...@gmail.com> ha scritto:

> +1
>
> On Wed, 22 Dec 2021 at 07:46, Matteo Merli  wrote:
>
> > This is the voting thread for PIP-120. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13306
> >
> > 
> >
> > ## Motivation
> >
> > In Pulsar 2.8, we have introduced a setting to control the amount of
> memory
> > used by a client instance.
> >
> > ```java
> > interface ClientBuilder {
> > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > }
> > ```
> >
> > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> > being
> > enforced.
> >
> > I think it's a good time for 2.10 to enable this setting by default and,
> > correspondingly, to disable by default the producer queue size limit.
> >
> > This will simplify a lot the configuration that a producer application
> will
> > have to come up with, when publishing with many topic/partitions or
> > when messages
> > are bigger than expected.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the `ClientBuilder`, change
> >   * `memoryLimit`: 0 -> 64 MB
> >
> > For the `ProducerBuilder`, changes
> >   * `maxPendingMessages`: 1000 -> 0
> >
> > 64MB is picked because it's a small enough memory size that will
> guarantee
> > a very high producer throughput, irrespective of the individual messages
> > size.
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

[GitHub] [pulsar-manager] sourabhaggrawal opened a new pull request #435: Fix for Sidebar items not visible to admin and superadmin

[GitBox]

sourabhaggrawal opened a new pull request #435:
URL: https://github.com/apache/pulsar-manager/pull/435


   
   ### Motivation
   
   
   *All newly added sidebar items (Roles,Users etc)  are not visible to 
superadmin. Also when admin user login, only Token tab is visible to them.* 
   
   ### Modifications
   
   *Added a new method isAccessbile in SidebarItem.vue. It compares roles which 
are set in Item.metadata.roles with logged-in user role, based on which the 
item is either visible or hidden for user*
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the `./gradlew build` checks.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

> We can avoid confusion by only attempting to create the subscription
> when `initSubscriptionName` is not `null`. As a result, this PIP
> wouldn't change the behavior for current implementations.
I think there is an issue with the current behavior. When the init
subscription
is not created before creating deadLetterProducer, it may result in the
loss of messages.
And this is not the expected behavior. So the purpose of this PIP is to fix
this issue.
IMO, we need to create that subscription by default to fix the current
behavior.

> +1 - I think we should expand the binary protocol to add a
> `CommandSubscription` or possibly `CommandCreateSubscription`
> (`CommandSubscribe` is already taken). Without a new command, we will
> introduce race conditions related to consumer creation and will
> definitely generate unnecessary errors.
+1. `CommandCreateSubscription` seems more reasonable.

> Regarding default authorization, this new command would require the
> same level of permission for the client since the act of creating a
> consumer and the act of creating a subscription both require the same
> `AuthAction` permission: Consume.
+1

Thanks,
Zike Yang

On Wed, Dec 22, 2021 at 2:40 AM Michael Marshall 
wrote:
>
> > IMO, we won't create the init subscription if
`allowAutoSubscriptionCreation`
> > is false. Otherwise, it will confuse the user. Users can explicitly
create that
> > subscription to handle this case.
>
> That is a good point, but I don't think we should attempt to create
> the subscription implicitly via consumer creation because that will
> lead to additional failure, which adds unnecessary load on the broker.
>
> We can avoid confusion by only attempting to create the subscription
> when `initSubscriptionName` is not `null`. As a result, this PIP
> wouldn't change the behavior for current implementations.
>
> > Do you mean to create a private method called `createSubscription` in
> > the non-admin client? But we don't have a protocol command for
> > creating subscription
>
> Yes, that was what I meant, and you're right, we don't have a protocol
> command for this case.
>
> > Or the other way is to create a new command `CommandSubscription` in
> > the protocol to handle this.
>
> +1 - I think we should expand the binary protocol to add a
> `CommandSubscription` or possibly `CommandCreateSubscription`
> (`CommandSubscribe` is already taken). Without a new command, we will
> introduce race conditions related to consumer creation and will
> definitely generate unnecessary errors.
>
> Regarding default authorization, this new command would require the
> same level of permission for the client since the act of creating a
> consumer and the act of creating a subscription both require the same
> `AuthAction` permission: Consume.
>
> Thanks,
> Michael
>
> On Tue, Dec 21, 2021 at 4:57 AM Zike Yang
>
>  wrote:
> >
> > Thanks for your review Michael.
> >
> > On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall 
wrote:
> > >
> > > Thanks for creating this PIP Zike Yang. This sounds like an important
> > > improvement. I have a couple thoughts.
> > >
> > > 1. Does the `retryLetterTopic` have the same problem with immediate
> > > message expiration? Based on a quick glance, I don't see anything
> > > creating a subscription for the retry topic.
> > >
> > Yes. I think we also need to handle the retryLetterTopic. I will update
the PIP.
> >
> > > 2. Your prototype exposes a lacking in our existing client
> > > implementation: the non-admin client cannot directly create
> > > subscriptions. IIUC, this implementation does not work if
> > > `allowAutoSubscriptionCreation` is false.
> > IMO, we won't create the init subscription if
`allowAutoSubscriptionCreation`
> >  is false. Otherwise, it will confuse the user. Users can explicitly
create that
> > subscription to handle this case.
> >
> > > Further, it is inefficient
> > > to create and immediately close a consumer just to create a
> > > subscription. Perhaps we should just update the non-admin client so
> > > that it can create subscriptions? It could be private so that we don't
> > > create confusion with the non-admin and the admin clients.
> > Do you mean to create a private method called `createSubscription` in
> > the non-admin client? But we don't have a protocol command for
> > creating subscription.
> > So in that method, we can send the `CommandSubscribe` and
`CommandUnsubscribe`
> > to the broker to simulate the creation and closure of a consumer. But
> > I think it is not
> > straightforward to create a subscription using two commands.
> > Or the other way is to create a new command `CommandSubscription` in
> > the protocol
> > to handle this. But I wonder if there are more use cases for this
> > command that would
> > make it worthwhile to use this solution.
> > What do you think about these two ways?
> >
> > >
> > > Thanks,
> > > Michael
> > >
> > >
> > > On Mon, Dec 20, 2021 at 4:18 AM Zike Yang
> > >  wrote:
> > > >
> > > > https://github.com/apache/pu

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

[Nicolò Boschi]

+1 (non binding)

Checks:
- Checksum and signatures
- Apache Rat check passes
- Compile from source w JDK8
- Build docker image from source
- Run Pulsar standalone and produce-consume from CLI
- Verified Log4J inside lib/

-rw-r--r-- 1 root root   208235 Jan 22  2020
org.apache.logging.log4j-log4j-1.2-api-2.17.0.jar

-rw-r--r-- 1 root root   301776 Jan 22  2020
org.apache.logging.log4j-log4j-api-2.17.0.jar

-rw-r--r-- 1 root root  1789339 Jan 22  2020
org.apache.logging.log4j-log4j-core-2.17.0.jar

-rw-r--r-- 1 root root24252 Jan 22  2020
org.apache.logging.log4j-log4j-slf4j-impl-2.17.0.jar

-rw-r--r-- 1 root root35920 Jan 22  2020
org.apache.logging.log4j-log4j-web-2.17.0.jar

Il giorno mer 22 dic 2021 alle ore 06:37 Lin Lin  ha
scritto:

>
>
> On 2021/12/21 10:48:41 Shivji Kumar Jha wrote:
> > Hi LinLin,
> >
> > Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we
> move
> > to Log4j 2.17.0 in 2.8.2?
> >
> > Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did
> not
> > > protect from uncontrolled recursion from self-referential lookups. This
> > > allows an attacker with control over Thread Context Map data to cause a
> > > denial of service when a crafted string is interpreted. This issue was
> > > fixed in Log4j 2.17.0 and 2.12.3.
>
>
> Already included
>


-- 
Nicolò Boschi

Re: [VOTE] PIP-120: Enable client memory limit by default

[Christophe Bornet]

+1

Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a écrit :

> This is the voting thread for PIP-120. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13306
>
> 
>
> ## Motivation
>
> In Pulsar 2.8, we have introduced a setting to control the amount of memory
> used by a client instance.
>
> ```java
> interface ClientBuilder {
> ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> }
> ```
>
> By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> being
> enforced.
>
> I think it's a good time for 2.10 to enable this setting by default and,
> correspondingly, to disable by default the producer queue size limit.
>
> This will simplify a lot the configuration that a producer application will
> have to come up with, when publishing with many topic/partitions or
> when messages
> are bigger than expected.
>
> ## Proposed changes
>
> In 2.10 release, for the `ClientBuilder`, change
>   * `memoryLimit`: 0 -> 64 MB
>
> For the `ProducerBuilder`, changes
>   * `maxPendingMessages`: 1000 -> 0
>
> 64MB is picked because it's a small enough memory size that will guarantee
> a very high producer throughput, irrespective of the individual messages
> size.
>
>
> --
> Matteo Merli
> 
>

Re: [VOTE] PIP-120: Enable client memory limit by default

[ZhangJian He]

+1

Christophe Bornet  于2021年12月22日周三 17:38写道：

> +1
>
> Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a écrit :
>
> > This is the voting thread for PIP-120. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13306
> >
> > 
> >
> > ## Motivation
> >
> > In Pulsar 2.8, we have introduced a setting to control the amount of
> memory
> > used by a client instance.
> >
> > ```java
> > interface ClientBuilder {
> > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > }
> > ```
> >
> > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> > being
> > enforced.
> >
> > I think it's a good time for 2.10 to enable this setting by default and,
> > correspondingly, to disable by default the producer queue size limit.
> >
> > This will simplify a lot the configuration that a producer application
> will
> > have to come up with, when publishing with many topic/partitions or
> > when messages
> > are bigger than expected.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the `ClientBuilder`, change
> >   * `memoryLimit`: 0 -> 64 MB
> >
> > For the `ProducerBuilder`, changes
> >   * `maxPendingMessages`: 1000 -> 0
> >
> > 64MB is picked because it's a small enough memory size that will
> guarantee
> > a very high producer throughput, irrespective of the individual messages
> > size.
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

[PengHui Li]

+1 (binding)

- check signatures
- build from sources
- verify Cassandra connector
- verify functions

Thanks for the great work.

Penghui

On Tue, Dec 21, 2021 at 7:28 PM Enrico Olivelli  wrote:

> +1 (binding)
>
> - built from sources, JDK8 in MacOS
> - run pulsar standalone, smoke tests
> - verified checksum and digital signatures
> - I took a look to the Maven staging repository
>
> Thanks for driving the release
>
> Enrico
>
>
> Il giorno mar 21 dic 2021 alle ore 11:27 Nozomi Kurihara <
> nkuri...@apache.org> ha scritto:
>
> > +1 (binding)
> >
> > - Verified checksums and signatures
> > - Verified license headers by apache rat
> > - Built the source
> > - Ran standalone, producer, and consumer
> > - Verified functions, cassandra connectors and stateful functions
> >
> > Thanks for driving the release.
> > Nozomi
> >
> > 2021年12月21日(火) 18:34 guo jiwei :
> >
> > > This is the second release candidate for Apache Pulsar, version 2.7.4.
> > >
> > > It fixes the following issues:
> > >
> > >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
> > >
> > > Release note:
> > > https://github.com/apache/pulsar/pull/13391
> > >
> > > *** Please download, test and vote on this release. This vote will
> > > stay openfor at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided
> for
> > > convenience.
> > >
> > > Source and binary files:
> > >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-2/
> > > <
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-1/
> > >
> > >
> > > SHA-512 checksums:
> > >
> > >
> > >
> >
> a56a794bd14c8f4b6d7320496b2be4640a836a7aa940c37aac4f53ae057f8b4887f3f8716f475d2a03c79e3e44bf2ad9234fd05e2b0f58b005a0a4f7f376fc15
> > >  apache-pulsar-2.7.4-bin.tar.gz
> > >
> > >
> >
> 03ec488d59d281fdbbbee48f1f62b266cd1163181decbf962d0d58eacc746731d31739955d8afac8bc392547caf2ae76e864d0ad2f2a460cd53920a707a7b40c
> > >  apache-pulsar-2.7.4-src.tar.gz
> > >
> > >
> > > Maven staging repo:
> > > https://repository.apache.org/content/repositories/orgapachepulsar-1
> > > <
> https://repository.apache.org/content/repositories/orgapachepulsar-1113
> > > >130
> > >
> > > The tag to be voted upon:
> > > v2.7.4-candidate-2 (2774b464f3b7e1ccfdc84d6d390efe7260cbb7fa)
> > > https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-
> > > 2
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download the source package, and follow the README to build
> > > and run the Pulsar standalone service.
> > >
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >
> >
>

Re: [VOTE] PIP-120: Enable client memory limit by default

[PengHui Li]

+1

Penghui

On Wed, Dec 22, 2021 at 6:57 PM ZhangJian He  wrote:

> +1
>
> Christophe Bornet  于2021年12月22日周三 17:38写道：
>
> > +1
> >
> > Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a écrit :
> >
> > > This is the voting thread for PIP-120. It will stay open for at least
> > 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13306
> > >
> > > 
> > >
> > > ## Motivation
> > >
> > > In Pulsar 2.8, we have introduced a setting to control the amount of
> > memory
> > > used by a client instance.
> > >
> > > ```java
> > > interface ClientBuilder {
> > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > }
> > > ```
> > >
> > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit
> is
> > > being
> > > enforced.
> > >
> > > I think it's a good time for 2.10 to enable this setting by default
> and,
> > > correspondingly, to disable by default the producer queue size limit.
> > >
> > > This will simplify a lot the configuration that a producer application
> > will
> > > have to come up with, when publishing with many topic/partitions or
> > > when messages
> > > are bigger than expected.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the `ClientBuilder`, change
> > >   * `memoryLimit`: 0 -> 64 MB
> > >
> > > For the `ProducerBuilder`, changes
> > >   * `maxPendingMessages`: 1000 -> 0
> > >
> > > 64MB is picked because it's a small enough memory size that will
> > guarantee
> > > a very high producer throughput, irrespective of the individual
> messages
> > > size.
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> >
>

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[PengHui Li]

I think we can create a consumer with the DLQ init subscription and then
close the consumer?
This will make the implementation easier.

Penghui

On Wed, Dec 22, 2021 at 4:49 PM Zike Yang 
wrote:

> > We can avoid confusion by only attempting to create the subscription
> > when `initSubscriptionName` is not `null`. As a result, this PIP
> > wouldn't change the behavior for current implementations.
> I think there is an issue with the current behavior. When the init
> subscription
> is not created before creating deadLetterProducer, it may result in the
> loss of messages.
> And this is not the expected behavior. So the purpose of this PIP is to fix
> this issue.
> IMO, we need to create that subscription by default to fix the current
> behavior.
>
> > +1 - I think we should expand the binary protocol to add a
> > `CommandSubscription` or possibly `CommandCreateSubscription`
> > (`CommandSubscribe` is already taken). Without a new command, we will
> > introduce race conditions related to consumer creation and will
> > definitely generate unnecessary errors.
> +1. `CommandCreateSubscription` seems more reasonable.
>
> > Regarding default authorization, this new command would require the
> > same level of permission for the client since the act of creating a
> > consumer and the act of creating a subscription both require the same
> > `AuthAction` permission: Consume.
> +1
>
> Thanks,
> Zike Yang
>
> On Wed, Dec 22, 2021 at 2:40 AM Michael Marshall 
> wrote:
> >
> > > IMO, we won't create the init subscription if
> `allowAutoSubscriptionCreation`
> > > is false. Otherwise, it will confuse the user. Users can explicitly
> create that
> > > subscription to handle this case.
> >
> > That is a good point, but I don't think we should attempt to create
> > the subscription implicitly via consumer creation because that will
> > lead to additional failure, which adds unnecessary load on the broker.
> >
> > We can avoid confusion by only attempting to create the subscription
> > when `initSubscriptionName` is not `null`. As a result, this PIP
> > wouldn't change the behavior for current implementations.
> >
> > > Do you mean to create a private method called `createSubscription` in
> > > the non-admin client? But we don't have a protocol command for
> > > creating subscription
> >
> > Yes, that was what I meant, and you're right, we don't have a protocol
> > command for this case.
> >
> > > Or the other way is to create a new command `CommandSubscription` in
> > > the protocol to handle this.
> >
> > +1 - I think we should expand the binary protocol to add a
> > `CommandSubscription` or possibly `CommandCreateSubscription`
> > (`CommandSubscribe` is already taken). Without a new command, we will
> > introduce race conditions related to consumer creation and will
> > definitely generate unnecessary errors.
> >
> > Regarding default authorization, this new command would require the
> > same level of permission for the client since the act of creating a
> > consumer and the act of creating a subscription both require the same
> > `AuthAction` permission: Consume.
> >
> > Thanks,
> > Michael
> >
> > On Tue, Dec 21, 2021 at 4:57 AM Zike Yang
> >
> >  wrote:
> > >
> > > Thanks for your review Michael.
> > >
> > > On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall  >
> wrote:
> > > >
> > > > Thanks for creating this PIP Zike Yang. This sounds like an important
> > > > improvement. I have a couple thoughts.
> > > >
> > > > 1. Does the `retryLetterTopic` have the same problem with immediate
> > > > message expiration? Based on a quick glance, I don't see anything
> > > > creating a subscription for the retry topic.
> > > >
> > > Yes. I think we also need to handle the retryLetterTopic. I will update
> the PIP.
> > >
> > > > 2. Your prototype exposes a lacking in our existing client
> > > > implementation: the non-admin client cannot directly create
> > > > subscriptions. IIUC, this implementation does not work if
> > > > `allowAutoSubscriptionCreation` is false.
> > > IMO, we won't create the init subscription if
> `allowAutoSubscriptionCreation`
> > >  is false. Otherwise, it will confuse the user. Users can explicitly
> create that
> > > subscription to handle this case.
> > >
> > > > Further, it is inefficient
> > > > to create and immediately close a consumer just to create a
> > > > subscription. Perhaps we should just update the non-admin client so
> > > > that it can create subscriptions? It could be private so that we
> don't
> > > > create confusion with the non-admin and the admin clients.
> > > Do you mean to create a private method called `createSubscription` in
> > > the non-admin client? But we don't have a protocol command for
> > > creating subscription.
> > > So in that method, we can send the `CommandSubscribe` and
> `CommandUnsubscribe`
> > > to the broker to simulate the creation and closure of a consumer. But
> > > I think it is not
> > > straightforward to create a subscription using two commands.
> > > Or t

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

[PengHui Li]

+1

Penghui

On Wed, Dec 22, 2021 at 3:31 PM Enrico Olivelli  wrote:

> very good
>
> Enrico
>
> Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
> mattisonc...@gmail.com> ha scritto:
>
> > +1
> >
> > On Wed, 22 Dec 2021 at 07:59, Matteo Merli  wrote:
> >
> > > This is the voting thread for PIP-123. It will stay open for at least
> > 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13346
> > >
> > >
> > > -
> > > ## Motivation
> > >
> > > For a very long time, we have included a CLI command to start the
> > ZooKeeper
> > > shell utility: `pulsar zookeeper-shell`, which is essentially a
> > repackaging
> > > of the ZooKeeper tool `zkCli.sh`.
> > >
> > > This is useful in some cases to either verify the content of metadata
> or
> > > to perform cleanup and modification tasks for which there is not an
> > > available
> > > option through the Pulsar REST APIs.
> > >
> > > While it's very useful, there are some drawbacks with the
> > `zookeeper-shell`
> > > as it is today:
> > >
> > >  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > > metadata backends, we should have a tool that works across all the
> > > implementations and presents a single consistent interface.
> > >
> > >  2. ZooKeeper shell is designed to be an interactive shell and it's not
> > > very
> > > good when trying to do non-interactive scriptable operations.
> > >
> > >  3. ZooKeeper is a bit clunky when using it and it requires the user to
> > > retype
> > > paths many times. The commands are not very intuitive or
> documented.
> > > It's not possible to update z-node with multi-lines content.
> > >
> > >  4. We cannot easily add functionality or improvements into ZooKeeper
> > > shell,
> > > since it belongs to a different project and the tool has been
> > > stagnating
> > > for many years.
> > >
> > >  5. In cases where the z-nodes content is binary (Protobuf) or
> > compressed,
> > > there
> > > is no easy way to inspect the content from the ZooKeeper shell.
> > > Additionally, we can format and colorize JSON content to make it
> > > easier to
> > > read.
> > >
> > >  6. The paths used for metadata resources are also often using
> encodings
> > > that
> > > make it more difficult to construct on the shell tool.
> > >
> > > Part of what is described here is in the `pulsar-managed-ledger-admin`
> > CLI
> > > tool,
> > > though that is a Python script that requires additional dependencies
> that
> > > are
> > > not typically installed, it only works with ZooKeeper, and it only
> > targets
> > > accessing the managed ledger metadata.
> > >
> > > ## Goal
> > >
> > > Introduce a new Java CLI tool to access, inspect and modify metadata
> that
> > > solves
> > > all the issues described above.
> > >
> > > We would leave the `zookeeper-shell` command for now. In the future,
> once
> > > the
> > > new tool is proven, we can consider removing the `zookeeper-shell`
> > command.
> > >
> > >
> > > ## Proposed changes
> > >
> > > Add a new command:
> > > ```bash
> > > bin/pulsar metadata
> > > ```
> > >
> > > with several subcommands:
> > >
> > >
> > >  Get
> > >
> > > Examples:
> > > ```bash
> > > # General path get
> > > $ pulsar metadata get /my-path
> > >
> > >
> > > # Topic metadata
> > > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > > {
> > >   # Managed ledger metadata
> > > }
> > >
> > > # Namespace get
> > > $ pulsar metadata get namespace my-tenant/my-namespace
> > > {
> > >   # Namespace metadata
> > > }
> > >
> > > $ pulsar metadata get ledger 12345
> > > {
> > >   # BK ledger metadata
> > > }
> > > ```
> > >
> > >  Delete
> > >
> > > Examples:
> > > ```bash
> > > # General path delete
> > > $ pulsar metadata delete /my-path
> > >
> > > # Topic metadata
> > > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > > ```
> > >
> > >  Scan
> > >
> > > Examples:
> > > ```bash
> > > $ pulsar metadata scan /my-path
> > > /my-path
> > > /my-path/1
> > > /my-path/2
> > > /my-path/3
> > > /my-path/3/1
> > >
> > > $ pulsar metadata scan --values /my-path
> > > /my-path
> > > {value}
> > >
> > > /my-path/1
> > > {value}
> > >
> > > /my-path/2
> > > {value}
> > > ```
> > >
> > >  Shell
> > >
> > > ```bash
> > > $ pulsar metadata shell
> > > > get topic my-tenant/my-namespace/my-topic
> > > {
> > >   # Managed ledger metadata
> > > }
> > >
> > > > delete topic my-tenant/my-namespace/my-topic
> > >
> > > > cd /my-path
> > > > ls
> > > 1
> > > 2
> > > 3
> > > > delete 1 # Delete keys with relative paths
> > >
> > > ```
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> >
>

Re: [VOTE] PIP-117: Change Pulsar standalone defaults

[PengHui Li]

+1

Penghui

On Wed, Dec 22, 2021 at 7:49 AM Matteo Merli  wrote:

> This is the voting thread for PIP-117. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13302
>
> 
>
> ## Motivation
>
> Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster,
> where
> all the components are started within the context of a single JVM process.
>
> Users are using the standalone as a way to get quickly started with Pulsar
> or
> in all the cases where it makes sense to have a single node deployment.
>
> Right now, the standalone is starting by default with many components,
> several of
> which are quite complex, since they are designed to be deployed in a
> distributed
> fashion.
>
> ## Goal
>
> Simplify the components of Pulsar standalone to achieve:
>
>  1. Reduce complexity
>  2. Reduce startup time
>  3. Reduce memory and CPU footprint of running standalone
>
> ## Proposed changes
>
> The proposal here is to change some of the default implementations that are
> used for the Pulsar standalone.
>
>  1. **Metadata Store implementation** -->
>   Change from ZooKeeper to RocksDB
>
>  2. **Pulsar functions package backend** -->
>   Change from using DistributedLog to using local filesystem, storing
> the
>   jars directly in the data folder instead of uploading them into BK.
>
>  3. **Pulsar functions state store implementation** -->
>   Change the state store to be backed by a MetadataStore based backed,
>   with the RocksDB implementation.
>
>  4. **Table Service** -->
>   Do not start BK table service by default
>
> ## Compatibility considerations
>
> In order to avoid compatibility issues where users have existing Pulsar
> standalone services that they want to upgrade without conflicts, we will
> follow the principle of keeping the old defaults where there is existing
> data on the disk.
>
> We will add a file, serving the purpose as a flag, in the `data/standalone`
> directory, for example `new-2.10-defaults`.
>
> If the file is present, or if the data directory is completely missing, we
> will adopt the new set of default configuration settings.
>
> If the file is not there, we will continue to use existing defaults and we
> will
> not break the upgrade operation.
>
>
> --
> Matteo Merli
> 
>

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[PengHui Li]

+1

Penghui

On Wed, Dec 22, 2021 at 7:23 AM Matteo Merli  wrote:

> This is the voting thread for PIP-119. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13305
>
> ---
>
> ## Motivation
>
> The consistent hashing implementation to uniformly assign keys to consumers
> in the context of a KeyShared subscription, was introduced in
> https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> 2.6.0.
>
> While consistent hashing can use slightly more memory in certain cases, it
> is
> more suitable as a general default implementation, as it leads to a fairer
> distribution of keys across consumers, and avoiding corner cases that
> depend
> on the sequence of addition/removal of consumers.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> splitting ranges or
> # consistent hashing to reassign keys to new consumers
> subscriptionKeySharedUseConsistentHashing=false
> ```
>
> Change its default value to `true`.
>
> The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> be
> able to use the old implementation.
>
> --
> Matteo Merli
> 
>

Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[PengHui Li]

+1

Penghui

On Wed, Dec 22, 2021 at 7:22 AM Matteo Merli  wrote:

> https://github.com/apache/pulsar/issues/13304
>
> Following the discussion, I have updated the proposal to also include
> the deprecation and renaming of the config setting name to
> `metadataSessionExpiredPolicy`.
>
>
>
> ---
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata
> operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled to stabilize in 2.9 the configuration setting that
> allows
> brokers to continue operating in a safe mode when the session with
> ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane
> will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid
> any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and
> thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Deprecate the old setting and rename it to
> `metadataSessionExpiredPolicy`, with default value set to `reconnect`.
>

Re: [VOTE] PIP-117: Change Pulsar standalone defaults

[ZhangJian He]

+1

PengHui Li  于2021年12月22日周三 19:33写道：

> +1
>
> Penghui
>
> On Wed, Dec 22, 2021 at 7:49 AM Matteo Merli  wrote:
>
> > This is the voting thread for PIP-117. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13302
> >
> > 
> >
> > ## Motivation
> >
> > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster,
> > where
> > all the components are started within the context of a single JVM
> process.
> >
> > Users are using the standalone as a way to get quickly started with
> Pulsar
> > or
> > in all the cases where it makes sense to have a single node deployment.
> >
> > Right now, the standalone is starting by default with many components,
> > several of
> > which are quite complex, since they are designed to be deployed in a
> > distributed
> > fashion.
> >
> > ## Goal
> >
> > Simplify the components of Pulsar standalone to achieve:
> >
> >  1. Reduce complexity
> >  2. Reduce startup time
> >  3. Reduce memory and CPU footprint of running standalone
> >
> > ## Proposed changes
> >
> > The proposal here is to change some of the default implementations that
> are
> > used for the Pulsar standalone.
> >
> >  1. **Metadata Store implementation** -->
> >   Change from ZooKeeper to RocksDB
> >
> >  2. **Pulsar functions package backend** -->
> >   Change from using DistributedLog to using local filesystem, storing
> > the
> >   jars directly in the data folder instead of uploading them into BK.
> >
> >  3. **Pulsar functions state store implementation** -->
> >   Change the state store to be backed by a MetadataStore based
> backed,
> >   with the RocksDB implementation.
> >
> >  4. **Table Service** -->
> >   Do not start BK table service by default
> >
> > ## Compatibility considerations
> >
> > In order to avoid compatibility issues where users have existing Pulsar
> > standalone services that they want to upgrade without conflicts, we will
> > follow the principle of keeping the old defaults where there is existing
> > data on the disk.
> >
> > We will add a file, serving the purpose as a flag, in the
> `data/standalone`
> > directory, for example `new-2.10-defaults`.
> >
> > If the file is present, or if the data directory is completely missing,
> we
> > will adopt the new set of default configuration settings.
> >
> > If the file is not there, we will continue to use existing defaults and
> we
> > will
> > not break the upgrade operation.
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [VOTE] PIP-117: Change Pulsar standalone defaults

[Hang Chen]

+1

Thanks,
Hang

ZhangJian He  于2021年12月22日周三 20:02写道：
>
> +1
>
> PengHui Li  于2021年12月22日周三 19:33写道：
>
> > +1
> >
> > Penghui
> >
> > On Wed, Dec 22, 2021 at 7:49 AM Matteo Merli  wrote:
> >
> > > This is the voting thread for PIP-117. It will stay open for at least
> > 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13302
> > >
> > > 
> > >
> > > ## Motivation
> > >
> > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster,
> > > where
> > > all the components are started within the context of a single JVM
> > process.
> > >
> > > Users are using the standalone as a way to get quickly started with
> > Pulsar
> > > or
> > > in all the cases where it makes sense to have a single node deployment.
> > >
> > > Right now, the standalone is starting by default with many components,
> > > several of
> > > which are quite complex, since they are designed to be deployed in a
> > > distributed
> > > fashion.
> > >
> > > ## Goal
> > >
> > > Simplify the components of Pulsar standalone to achieve:
> > >
> > >  1. Reduce complexity
> > >  2. Reduce startup time
> > >  3. Reduce memory and CPU footprint of running standalone
> > >
> > > ## Proposed changes
> > >
> > > The proposal here is to change some of the default implementations that
> > are
> > > used for the Pulsar standalone.
> > >
> > >  1. **Metadata Store implementation** -->
> > >   Change from ZooKeeper to RocksDB
> > >
> > >  2. **Pulsar functions package backend** -->
> > >   Change from using DistributedLog to using local filesystem, storing
> > > the
> > >   jars directly in the data folder instead of uploading them into BK.
> > >
> > >  3. **Pulsar functions state store implementation** -->
> > >   Change the state store to be backed by a MetadataStore based
> > backed,
> > >   with the RocksDB implementation.
> > >
> > >  4. **Table Service** -->
> > >   Do not start BK table service by default
> > >
> > > ## Compatibility considerations
> > >
> > > In order to avoid compatibility issues where users have existing Pulsar
> > > standalone services that they want to upgrade without conflicts, we will
> > > follow the principle of keeping the old defaults where there is existing
> > > data on the disk.
> > >
> > > We will add a file, serving the purpose as a flag, in the
> > `data/standalone`
> > > directory, for example `new-2.10-defaults`.
> > >
> > > If the file is present, or if the data directory is completely missing,
> > we
> > > will adopt the new set of default configuration settings.
> > >
> > > If the file is not there, we will continue to use existing defaults and
> > we
> > > will
> > > not break the upgrade operation.
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> >

Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[Hang Chen]

+1

Thanks,
Hang

PengHui Li  于2021年12月22日周三 19:34写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 22, 2021 at 7:22 AM Matteo Merli  wrote:
>
> > https://github.com/apache/pulsar/issues/13304
> >
> > Following the discussion, I have updated the proposal to also include
> > the deprecation and renaming of the config setting name to
> > `metadataSessionExpiredPolicy`.
> >
> >
> >
> > ---
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and 2.9
> > releases, it enabled the concept of re-acquirable resource locks and leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> > operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled to stabilize in 2.9 the configuration setting that
> > allows
> > brokers to continue operating in a safe mode when the session with
> > ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data plane
> > will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> > any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> > thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Deprecate the old setting and rename it to
> > `metadataSessionExpiredPolicy`, with default value set to `reconnect`.
> >

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Hang Chen]

+1

Thanks,
Hang

PengHui Li  于2021年12月22日周三 19:33写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 22, 2021 at 7:23 AM Matteo Merli  wrote:
>
> > This is the voting thread for PIP-119. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13305
> >
> > ---
> >
> > ## Motivation
> >
> > The consistent hashing implementation to uniformly assign keys to consumers
> > in the context of a KeyShared subscription, was introduced in
> > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> > 2.6.0.
> >
> > While consistent hashing can use slightly more memory in certain cases, it
> > is
> > more suitable as a general default implementation, as it leads to a fairer
> > distribution of keys across consumers, and avoiding corner cases that
> > depend
> > on the sequence of addition/removal of consumers.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > splitting ranges or
> > # consistent hashing to reassign keys to new consumers
> > subscriptionKeySharedUseConsistentHashing=false
> > ```
> >
> > Change its default value to `true`.
> >
> > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> > be
> > able to use the old implementation.
> >
> > --
> > Matteo Merli
> > 
> >

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

[Hang Chen]

+1

Thanks,
Hang

PengHui Li  于2021年12月22日周三 19:33写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 22, 2021 at 3:31 PM Enrico Olivelli  wrote:
>
> > very good
> >
> > Enrico
> >
> > Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
> > mattisonc...@gmail.com> ha scritto:
> >
> > > +1
> > >
> > > On Wed, 22 Dec 2021 at 07:59, Matteo Merli  wrote:
> > >
> > > > This is the voting thread for PIP-123. It will stay open for at least
> > > 48h.
> > > >
> > > > https://github.com/apache/pulsar/issues/13346
> > > >
> > > >
> > > > -
> > > > ## Motivation
> > > >
> > > > For a very long time, we have included a CLI command to start the
> > > ZooKeeper
> > > > shell utility: `pulsar zookeeper-shell`, which is essentially a
> > > repackaging
> > > > of the ZooKeeper tool `zkCli.sh`.
> > > >
> > > > This is useful in some cases to either verify the content of metadata
> > or
> > > > to perform cleanup and modification tasks for which there is not an
> > > > available
> > > > option through the Pulsar REST APIs.
> > > >
> > > > While it's very useful, there are some drawbacks with the
> > > `zookeeper-shell`
> > > > as it is today:
> > > >
> > > >  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > > > metadata backends, we should have a tool that works across all the
> > > > implementations and presents a single consistent interface.
> > > >
> > > >  2. ZooKeeper shell is designed to be an interactive shell and it's not
> > > > very
> > > > good when trying to do non-interactive scriptable operations.
> > > >
> > > >  3. ZooKeeper is a bit clunky when using it and it requires the user to
> > > > retype
> > > > paths many times. The commands are not very intuitive or
> > documented.
> > > > It's not possible to update z-node with multi-lines content.
> > > >
> > > >  4. We cannot easily add functionality or improvements into ZooKeeper
> > > > shell,
> > > > since it belongs to a different project and the tool has been
> > > > stagnating
> > > > for many years.
> > > >
> > > >  5. In cases where the z-nodes content is binary (Protobuf) or
> > > compressed,
> > > > there
> > > > is no easy way to inspect the content from the ZooKeeper shell.
> > > > Additionally, we can format and colorize JSON content to make it
> > > > easier to
> > > > read.
> > > >
> > > >  6. The paths used for metadata resources are also often using
> > encodings
> > > > that
> > > > make it more difficult to construct on the shell tool.
> > > >
> > > > Part of what is described here is in the `pulsar-managed-ledger-admin`
> > > CLI
> > > > tool,
> > > > though that is a Python script that requires additional dependencies
> > that
> > > > are
> > > > not typically installed, it only works with ZooKeeper, and it only
> > > targets
> > > > accessing the managed ledger metadata.
> > > >
> > > > ## Goal
> > > >
> > > > Introduce a new Java CLI tool to access, inspect and modify metadata
> > that
> > > > solves
> > > > all the issues described above.
> > > >
> > > > We would leave the `zookeeper-shell` command for now. In the future,
> > once
> > > > the
> > > > new tool is proven, we can consider removing the `zookeeper-shell`
> > > command.
> > > >
> > > >
> > > > ## Proposed changes
> > > >
> > > > Add a new command:
> > > > ```bash
> > > > bin/pulsar metadata
> > > > ```
> > > >
> > > > with several subcommands:
> > > >
> > > >
> > > >  Get
> > > >
> > > > Examples:
> > > > ```bash
> > > > # General path get
> > > > $ pulsar metadata get /my-path
> > > >
> > > >
> > > > # Topic metadata
> > > > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > > > {
> > > >   # Managed ledger metadata
> > > > }
> > > >
> > > > # Namespace get
> > > > $ pulsar metadata get namespace my-tenant/my-namespace
> > > > {
> > > >   # Namespace metadata
> > > > }
> > > >
> > > > $ pulsar metadata get ledger 12345
> > > > {
> > > >   # BK ledger metadata
> > > > }
> > > > ```
> > > >
> > > >  Delete
> > > >
> > > > Examples:
> > > > ```bash
> > > > # General path delete
> > > > $ pulsar metadata delete /my-path
> > > >
> > > > # Topic metadata
> > > > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > > > ```
> > > >
> > > >  Scan
> > > >
> > > > Examples:
> > > > ```bash
> > > > $ pulsar metadata scan /my-path
> > > > /my-path
> > > > /my-path/1
> > > > /my-path/2
> > > > /my-path/3
> > > > /my-path/3/1
> > > >
> > > > $ pulsar metadata scan --values /my-path
> > > > /my-path
> > > > {value}
> > > >
> > > > /my-path/1
> > > > {value}
> > > >
> > > > /my-path/2
> > > > {value}
> > > > ```
> > > >
> > > >  Shell
> > > >
> > > > ```bash
> > > > $ pulsar metadata shell
> > > > > get topic my-tenant/my-namespace/my-topic
> > > > {
> > > >   # Managed ledger metadata
> > > > }
> > > >
> > > > > delete topic my-tenant/my-namespace/my-topic
> > > >
> > > > > cd /my-path
> > > > > ls
> > > > 1
> > > > 2
> > > > 3
> > > > > delete 1 # Delete keys with relative paths
>

Re: [VOTE] PIP-120: Enable client memory limit by default

[Hang Chen]

+1

Thanks,
Hang

PengHui Li  于2021年12月22日周三 19:28写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 22, 2021 at 6:57 PM ZhangJian He  wrote:
>
> > +1
> >
> > Christophe Bornet  于2021年12月22日周三 17:38写道：
> >
> > > +1
> > >
> > > Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a écrit :
> > >
> > > > This is the voting thread for PIP-120. It will stay open for at least
> > > 48h.
> > > >
> > > > https://github.com/apache/pulsar/issues/13306
> > > >
> > > > 
> > > >
> > > > ## Motivation
> > > >
> > > > In Pulsar 2.8, we have introduced a setting to control the amount of
> > > memory
> > > > used by a client instance.
> > > >
> > > > ```java
> > > > interface ClientBuilder {
> > > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > > }
> > > > ```
> > > >
> > > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit
> > is
> > > > being
> > > > enforced.
> > > >
> > > > I think it's a good time for 2.10 to enable this setting by default
> > and,
> > > > correspondingly, to disable by default the producer queue size limit.
> > > >
> > > > This will simplify a lot the configuration that a producer application
> > > will
> > > > have to come up with, when publishing with many topic/partitions or
> > > > when messages
> > > > are bigger than expected.
> > > >
> > > > ## Proposed changes
> > > >
> > > > In 2.10 release, for the `ClientBuilder`, change
> > > >   * `memoryLimit`: 0 -> 64 MB
> > > >
> > > > For the `ProducerBuilder`, changes
> > > >   * `maxPendingMessages`: 1000 -> 0
> > > >
> > > > 64MB is picked because it's a small enough memory size that will
> > > guarantee
> > > > a very high producer throughput, irrespective of the individual
> > messages
> > > > size.
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
> > >
> >

OWASP dependencies check on active branches

[Nicolò Boschi]

Hello everyone,

I created a couple of pull requests in order to run a periodic check on
Pulsar active branches. In this way we can proactively update dependencies
whenever is needed (for fixing CVE's purpose)

The first one [0] is to make the check pass on branch-2.8
The second one [1] is to make the check pass on master and branch-2.9
The third one [2] is to make the periodic job running against master,
branch-2.8 and branch-2.9.

We also have to port this PR [3] to branch-2.9

I left out 2.7 branch because I have the impression (please confirm it) we
are no longer cherry-picking dependency upgrades. Also the check doesn't
exist at all in that branch.

Let me know what you think.

Thanks,
Nicolò Boschi

[0] https://github.com/apache/pulsar/pull/13455
[1] https://github.com/apache/pulsar/pull/13451
[2] https://github.com/apache/pulsar/pull/13366
[3] https://github.com/apache/pulsar/pull/13364

Re: [VOTE] PIP-120: Enable client memory limit by default

[Michael Marshall]

+1

- Michael

On Wed, Dec 22, 2021 at 6:10 AM Hang Chen  wrote:
>
> +1
>
> Thanks,
> Hang
>
> PengHui Li  于2021年12月22日周三 19:28写道：
> >
> > +1
> >
> > Penghui
> >
> > On Wed, Dec 22, 2021 at 6:57 PM ZhangJian He  wrote:
> >
> > > +1
> > >
> > > Christophe Bornet  于2021年12月22日周三 17:38写道：
> > >
> > > > +1
> > > >
> > > > Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a écrit :
> > > >
> > > > > This is the voting thread for PIP-120. It will stay open for at least
> > > > 48h.
> > > > >
> > > > > https://github.com/apache/pulsar/issues/13306
> > > > >
> > > > > 
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > In Pulsar 2.8, we have introduced a setting to control the amount of
> > > > memory
> > > > > used by a client instance.
> > > > >
> > > > > ```java
> > > > > interface ClientBuilder {
> > > > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > > > }
> > > > > ```
> > > > >
> > > > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit
> > > is
> > > > > being
> > > > > enforced.
> > > > >
> > > > > I think it's a good time for 2.10 to enable this setting by default
> > > and,
> > > > > correspondingly, to disable by default the producer queue size limit.
> > > > >
> > > > > This will simplify a lot the configuration that a producer application
> > > > will
> > > > > have to come up with, when publishing with many topic/partitions or
> > > > > when messages
> > > > > are bigger than expected.
> > > > >
> > > > > ## Proposed changes
> > > > >
> > > > > In 2.10 release, for the `ClientBuilder`, change
> > > > >   * `memoryLimit`: 0 -> 64 MB
> > > > >
> > > > > For the `ProducerBuilder`, changes
> > > > >   * `maxPendingMessages`: 1000 -> 0
> > > > >
> > > > > 64MB is picked because it's a small enough memory size that will
> > > > guarantee
> > > > > a very high producer throughput, irrespective of the individual
> > > messages
> > > > > size.
> > > > >
> > > > >
> > > > > --
> > > > > Matteo Merli
> > > > > 
> > > > >
> > > >
> > >

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Michael Marshall]

+1

- Michael

On Wed, Dec 22, 2021 at 6:09 AM Hang Chen  wrote:
>
> +1
>
> Thanks,
> Hang
>
> PengHui Li  于2021年12月22日周三 19:33写道：
> >
> > +1
> >
> > Penghui
> >
> > On Wed, Dec 22, 2021 at 7:23 AM Matteo Merli  wrote:
> >
> > > This is the voting thread for PIP-119. It will stay open for at least 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13305
> > >
> > > ---
> > >
> > > ## Motivation
> > >
> > > The consistent hashing implementation to uniformly assign keys to 
> > > consumers
> > > in the context of a KeyShared subscription, was introduced in
> > > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> > > 2.6.0.
> > >
> > > While consistent hashing can use slightly more memory in certain cases, it
> > > is
> > > more suitable as a general default implementation, as it leads to a fairer
> > > distribution of keys across consumers, and avoiding corner cases that
> > > depend
> > > on the sequence of addition/removal of consumers.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the setting:
> > >
> > > ```properties
> > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > > splitting ranges or
> > > # consistent hashing to reassign keys to new consumers
> > > subscriptionKeySharedUseConsistentHashing=false
> > > ```
> > >
> > > Change its default value to `true`.
> > >
> > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> > > be
> > > able to use the old implementation.
> > >
> > > --
> > > Matteo Merli
> > > 
> > >

Re: OWASP dependencies check on active branches

[Lari Hotari]

Good work Nicolò! It's great to have OWASP dependency check handled for all
active branches.

-Lari

On Wed, Dec 22, 2021 at 5:05 PM Nicolò Boschi  wrote:

> Hello everyone,
>
> I created a couple of pull requests in order to run a periodic check on
> Pulsar active branches. In this way we can proactively update dependencies
> whenever is needed (for fixing CVE's purpose)
>
> The first one [0] is to make the check pass on branch-2.8
> The second one [1] is to make the check pass on master and branch-2.9
> The third one [2] is to make the periodic job running against master,
> branch-2.8 and branch-2.9.
>
> We also have to port this PR [3] to branch-2.9
>
> I left out 2.7 branch because I have the impression (please confirm it) we
> are no longer cherry-picking dependency upgrades. Also the check doesn't
> exist at all in that branch.
>
> Let me know what you think.
>
> Thanks,
> Nicolò Boschi
>
> [0] https://github.com/apache/pulsar/pull/13455
> [1] https://github.com/apache/pulsar/pull/13451
> [2] https://github.com/apache/pulsar/pull/13366
> [3] https://github.com/apache/pulsar/pull/13364
>

[GitHub] [pulsar-helm-chart] sdwerwed edited a comment on pull request #127: Antiaffinity with zone topology for high availability

[GitBox]

sdwerwed edited a comment on pull request #127:
URL: https://github.com/apache/pulsar-helm-chart/pull/127#issuecomment-859123386


   Hello @wolfstudy, @sijie, can you have a look at this PR, let me know if any 
other action is required from my side.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] sdwerwed commented on pull request #127: Antiaffinity with zone topology for high availability

[GitBox]

sdwerwed commented on pull request #127:
URL: https://github.com/apache/pulsar-helm-chart/pull/127#issuecomment-999727380


Hello @wolfstudy, @sijie, can you have a look at this PR, let me know if 
any other action is required from my side.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] sdwerwed removed a comment on pull request #127: Antiaffinity with zone topology for high availability

[GitBox]

sdwerwed removed a comment on pull request #127:
URL: https://github.com/apache/pulsar-helm-chart/pull/127#issuecomment-859123386


   Hello @wolfstudy, @sijie, can you have a look at this PR, let me know if any 
other action is required from my side.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Pulsar non root docker image

[Michael Marshall]

Thanks for raising this important topic, Enrico.

> Basically if you are running as non root, you cannot add tools on demand,
> so we need to add basic tools, like netstat/vim/unzip otherwise when
> you have a problem you are trapped.

I agree that running as a non root user presents challenges for
debugging. However, I don't think we should optimize our production
image for debugging. We should instead optimize for a minimal docker
image with as few dependencies as possible to decrease the image's
attack surface.

Also, I think it would be valuable to audit the current
dependencies we already ship in our docker image. For example, this
single `RUN` command [0] adds 1 GB of dependencies to our
docker image.

If there is a need for an image with debug tools, we could publish a
"debug" image that extends our production image with extra tooling.
Users could swap out the prod image with the debug image, as needed.
However, it is pretty easy and quick to extend a public docker image
to add your own tooling. By documenting how to extend our docker
image, we could avoid decisions about which tools should be in our
image.

> there are ways to inject tools, but they are very hard to execute for
> people who is not very experienced

We can solve this through additional documentation. Freeznet asked a
similar question about debugging on the PR. I provided some methods for
debugging here [1].

Thanks,
Michael

[0] 
https://github.com/apache/pulsar/blob/5dd60dbd748e446f8da396b448a5bb16a2ae6902/docker/pulsar/Dockerfile#L46-L55
[1] https://github.com/apache/pulsar/pull/13376#discussion_r773580954




On Wed, Dec 22, 2021 at 1:29 AM Enrico Olivelli  wrote:
>
> Michael,
>
> I would like to add this item to the list
> https://github.com/apache/pulsar/pull/10815
>
> Basically if you are running as non root, you cannot add tools on demand,
> so we need to add basic tools, like netstat/vim/unzip otherwise when
> you have a problem you are trapped.
>
> there are ways to inject tools, but they are very hard to execute for
> people who is not very experienced
>
> Enrico
>
>
> Il giorno mer 22 dic 2021 alle ore 04:40 Haiting Jiang <
> jianghait...@apache.org> ha scritto:
>
> > > 1. Pulsar configuration is read in only from configuration files in
> > > `/pulsar/conf`. A non root user must be able to update these files to
> > > have run with custom configuration.
> >
> > About the configurations, I also see similar require like this lately [0].
> > IMHO, update any configs without redeploy service is a useful feature.
> > I would like post a PIP for this later.
> > Basic idea is like make all configs dynamic by default except
> > `metadataStoreUrl` and all configs are stored under path
> > "/admin/configuration" in metadata store.
> >
> > [0] https://github.com/apache/pulsar/pull/13074
> >
> > On 2021/12/21 20:16:44 Michael Marshall wrote:
> > > All tests are now passing for this PR [0]. I built the docker image
> > > and pushed it to my personal repository to simplify testing [1] for
> > > anyone interested in verifying the changes.
> > >
> > > I would like our docker image to be as close to immutable as possible.
> > > As far as I can tell, here are the only reasons the image cannot be
> > > immutable:
> > >
> > > 1. Pulsar configuration is read in only from configuration files in
> > > `/pulsar/conf`. A non root user must be able to update these files to
> > > have run with custom configuration.
> > > 2. The Pulsar function worker unpacks nar files to
> > > `/pulsar/download/pulsar_functions` by default.
> > > 3. Pulsar tiered storage writes to `/pulsar` by default when using
> > > filesystem storage.
> > > 4. The Presto SQL worker writes to `/pulsar/lib/presto/` by default.
> > > 5. Pulsar-admin and functions write to `/pulsar/log` by default
> > > (possibly other components too).
> > >
> > > Note that even though bookkeepers and zookeepers write to
> > > `/pulsar/data`, they should be writing to docker volumes, in which
> > > case, the host's file system permissions are all that matter.
> > >
> > > If we can remove any of the above reasons, we can decrease the
> > > privilege in the docker image.
> > >
> > > The PR has more detail. Please take a look.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar/pull/13376
> > > [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT-1dec8b9
> > >
> > >
> > > On Fri, Dec 17, 2021 at 12:33 AM Michael Marshall 
> > wrote:
> > > >
> > > > Hi Pulsar Community,
> > > >
> > > > I opened a PR to make our pulsar and pulsar-all docker images non root
> > > > and OpenShift compliant [0]. As some may remember, we had issues with
> > > > these changes before due to lack of testing. I plan to test thoroughly
> > > > before we merge this PR, and it'd be great to have others test too. I
> > > > published a build of my PR [1]. I also have an issue [2] tracking this
> > > > work.
> > > >
> > > > Please take a look. I hope to make our 2.10 release a non root release!
> > > >
> > 

Re: WebSite build CI job does work - blocker for announcing release !

[Dave Fisher]

Hi -

A WebSite Build CI Job just completed. Please check that it created the new 
docs.

I see the release notes for 2.9.1 - https://pulsar.apache.org/en/release-notes/

Regards,
Dave

> On Dec 21, 2021, at 11:33 PM, Nicolò Boschi  wrote:
> 
> I think the problem is that it times out (currently there's a job timeout
> of 2h)
> I created this PR to increase it to 3h
> https://github.com/apache/pulsar/pull/13449
> 
> 
> Il giorno mer 22 dic 2021 alle ore 08:23 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
> 
>> Hello,
>> the WebSite build CI job is stuck
>> https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true
>> 
>> This is probably the problem, it is due to "crowdin" that is timing out
>> 
>> File 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
>> SKIPPED
>> File 'master/website/i18n/en.json'/ - - OK
>> Done in 2452.38s.
>> + yarn run crowdin-download
>> yarn run v1.22.5
>> warning package.json: No license field
>> $ crowdin --config ../crowdin.yaml download -b master
>> Error: The operation was canceled.
>> 
>> 
>> Can anyone help ?
>> we are stuck and if we do not update the website we cannot announce the
>> releases with the security fixes
>> 
>> 
>> Enrico
>> 
> 
> 
> -- 
> Nicolò Boschi

Re: WebSite build CI job does work - blocker for announcing release !

[Enrico Olivelli]

Thanks Dave,
I don't see 2.9.1 in the download page.
Did I miss to update some files?

Enrico

Il Mer 22 Dic 2021, 21:19 Dave Fisher  ha scritto:

> Hi -
>
> A WebSite Build CI Job just completed. Please check that it created the
> new docs.
>
> I see the release notes for 2.9.1 -
> https://pulsar.apache.org/en/release-notes/
>
> Regards,
> Dave
>
> > On Dec 21, 2021, at 11:33 PM, Nicolò Boschi 
> wrote:
> >
> > I think the problem is that it times out (currently there's a job timeout
> > of 2h)
> > I created this PR to increase it to 3h
> > https://github.com/apache/pulsar/pull/13449
> >
> >
> > Il giorno mer 22 dic 2021 alle ore 08:23 Enrico Olivelli <
> > eolive...@gmail.com> ha scritto:
> >
> >> Hello,
> >> the WebSite build CI job is stuck
> >> https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true
> >>
> >> This is probably the problem, it is due to "crowdin" that is timing out
> >>
> >> File
> 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
> >> SKIPPED
> >> File 'master/website/i18n/en.json'/ - - OK
> >> Done in 2452.38s.
> >> + yarn run crowdin-download
> >> yarn run v1.22.5
> >> warning package.json: No license field
> >> $ crowdin --config ../crowdin.yaml download -b master
> >> Error: The operation was canceled.
> >>
> >>
> >> Can anyone help ?
> >> we are stuck and if we do not update the website we cannot announce the
> >> releases with the security fixes
> >>
> >>
> >> Enrico
> >>
> >
> >
> > --
> > Nicolò Boschi
>
>

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mer 22 Dic 2021, 16:59 Michael Marshall  ha
scritto:

> +1
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:09 AM Hang Chen  wrote:
> >
> > +1
> >
> > Thanks,
> > Hang
> >
> > PengHui Li  于2021年12月22日周三 19:33写道：
> > >
> > > +1
> > >
> > > Penghui
> > >
> > > On Wed, Dec 22, 2021 at 7:23 AM Matteo Merli 
> wrote:
> > >
> > > > This is the voting thread for PIP-119. It will stay open for at
> least 48h.
> > > >
> > > > https://github.com/apache/pulsar/issues/13305
> > > >
> > > > ---
> > > >
> > > > ## Motivation
> > > >
> > > > The consistent hashing implementation to uniformly assign keys to
> consumers
> > > > in the context of a KeyShared subscription, was introduced in
> > > > https://github.com/apache/pulsar/pull/6791, which was released in
> Pulsar
> > > > 2.6.0.
> > > >
> > > > While consistent hashing can use slightly more memory in certain
> cases, it
> > > > is
> > > > more suitable as a general default implementation, as it leads to a
> fairer
> > > > distribution of keys across consumers, and avoiding corner cases that
> > > > depend
> > > > on the sequence of addition/removal of consumers.
> > > >
> > > > ## Proposed changes
> > > >
> > > > In 2.10 release, for the setting:
> > > >
> > > > ```properties
> > > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > > > splitting ranges or
> > > > # consistent hashing to reassign keys to new consumers
> > > > subscriptionKeySharedUseConsistentHashing=false
> > > > ```
> > > >
> > > > Change its default value to `true`.
> > > >
> > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will
> still
> > > > be
> > > > able to use the old implementation.
> > > >
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
>

Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mer 22 Dic 2021, 13:09 Hang Chen  ha scritto:

> +1
>
> Thanks,
> Hang
>
> PengHui Li  于2021年12月22日周三 19:34写道：
> >
> > +1
> >
> > Penghui
> >
> > On Wed, Dec 22, 2021 at 7:22 AM Matteo Merli  wrote:
> >
> > > https://github.com/apache/pulsar/issues/13304
> > >
> > > Following the discussion, I have updated the proposal to also include
> > > the deprecation and renaming of the config setting name to
> > > `metadataSessionExpiredPolicy`.
> > >
> > >
> > >
> > > ---
> > > ## Motivation
> > >
> > > After all the work done for PIP-45 that was already included in 2.8
> and 2.9
> > > releases, it enabled the concept of re-acquirable resource locks and
> leader
> > > election.
> > >
> > > Another important change was to avoid doing any deferrable metadata
> > > operation
> > > when we know that we are not currently connected to the metadata
> service.
> > >
> > > Finally, that enabled to stabilize in 2.9 the configuration setting
> that
> > > allows
> > > brokers to continue operating in a safe mode when the session with
> > > ZooKeeper
> > > expires.
> > >
> > > The way it works is that, when we lose a ZooKeeper session, the data
> plane
> > > will
> > > continue to work undisturbed, relying on the BookKeeper fencing to
> avoid
> > > any
> > > inconsistencies.
> > >
> > > New topics are not able to get started, but existing topics will see no
> > > impact.
> > >
> > > The original intention for shutting down the brokers was to ensure
> that we
> > > would automatically go back to a consistent state, with respect to
> which
> > > resources are "owned" in ZooKeeper by a given broker.
> > >
> > > With the re-acquirable resource locks, that problem was solved and
> > > thoroughly
> > > tested to be robust.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the setting:
> > >
> > > ```properties
> > > # There are two policies to apply when broker metadata session
> > > expires: session expired happens, "shutdown" or "reconnect".
> > > # With "shutdown", the broker will be restarted.
> > > # With "reconnect", the broker will keep serving the topics, while
> > > attempting to recreate a new session.
> > > zookeeperSessionExpiredPolicy=shutdown
> > > ```
> > >
> > > Deprecate the old setting and rename it to
> > > `metadataSessionExpiredPolicy`, with default value set to `reconnect`.
> > >
>

Re: [VOTE] PIP-120: Enable client memory limit by default

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mer 22 Dic 2021, 16:58 Michael Marshall  ha
scritto:

> +1
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:10 AM Hang Chen  wrote:
> >
> > +1
> >
> > Thanks,
> > Hang
> >
> > PengHui Li  于2021年12月22日周三 19:28写道：
> > >
> > > +1
> > >
> > > Penghui
> > >
> > > On Wed, Dec 22, 2021 at 6:57 PM ZhangJian He 
> wrote:
> > >
> > > > +1
> > > >
> > > > Christophe Bornet  于2021年12月22日周三 17:38写道：
> > > >
> > > > > +1
> > > > >
> > > > > Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a
> écrit :
> > > > >
> > > > > > This is the voting thread for PIP-120. It will stay open for at
> least
> > > > > 48h.
> > > > > >
> > > > > > https://github.com/apache/pulsar/issues/13306
> > > > > >
> > > > > > 
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > In Pulsar 2.8, we have introduced a setting to control the
> amount of
> > > > > memory
> > > > > > used by a client instance.
> > > > > >
> > > > > > ```java
> > > > > > interface ClientBuilder {
> > > > > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no
> limit
> > > > is
> > > > > > being
> > > > > > enforced.
> > > > > >
> > > > > > I think it's a good time for 2.10 to enable this setting by
> default
> > > > and,
> > > > > > correspondingly, to disable by default the producer queue size
> limit.
> > > > > >
> > > > > > This will simplify a lot the configuration that a producer
> application
> > > > > will
> > > > > > have to come up with, when publishing with many topic/partitions
> or
> > > > > > when messages
> > > > > > are bigger than expected.
> > > > > >
> > > > > > ## Proposed changes
> > > > > >
> > > > > > In 2.10 release, for the `ClientBuilder`, change
> > > > > >   * `memoryLimit`: 0 -> 64 MB
> > > > > >
> > > > > > For the `ProducerBuilder`, changes
> > > > > >   * `maxPendingMessages`: 1000 -> 0
> > > > > >
> > > > > > 64MB is picked because it's a small enough memory size that will
> > > > > guarantee
> > > > > > a very high producer throughput, irrespective of the individual
> > > > messages
> > > > > > size.
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Matteo Merli
> > > > > > 
> > > > > >
> > > > >
> > > >
>

Re: WebSite build CI job does work - blocker for announcing release !

[Dave Fisher]

See this PR for 2.8.1 - https://github.com/apache/pulsar/pull/12089/files

> On Dec 22, 2021, at 12:48 PM, Enrico Olivelli  wrote:
> 
> Thanks Dave,
> I don't see 2.9.1 in the download page.
> Did I miss to update some files?
> 
> Enrico
> 
> Il Mer 22 Dic 2021, 21:19 Dave Fisher  ha scritto:
> 
>> Hi -
>> 
>> A WebSite Build CI Job just completed. Please check that it created the
>> new docs.
>> 
>> I see the release notes for 2.9.1 -
>> https://pulsar.apache.org/en/release-notes/
>> 
>> Regards,
>> Dave
>> 
>>> On Dec 21, 2021, at 11:33 PM, Nicolò Boschi 
>> wrote:
>>> 
>>> I think the problem is that it times out (currently there's a job timeout
>>> of 2h)
>>> I created this PR to increase it to 3h
>>> https://github.com/apache/pulsar/pull/13449
>>> 
>>> 
>>> Il giorno mer 22 dic 2021 alle ore 08:23 Enrico Olivelli <
>>> eolive...@gmail.com> ha scritto:
>>> 
 Hello,
 the WebSite build CI job is stuck
 https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true
 
 This is probably the problem, it is due to "crowdin" that is timing out
 
 File
>> 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
 SKIPPED
 File 'master/website/i18n/en.json'/ - - OK
 Done in 2452.38s.
 + yarn run crowdin-download
 yarn run v1.22.5
 warning package.json: No license field
 $ crowdin --config ../crowdin.yaml download -b master
 Error: The operation was canceled.
 
 
 Can anyone help ?
 we are stuck and if we do not update the website we cannot announce the
 releases with the security fixes
 
 
 Enrico
 
>>> 
>>> 
>>> --
>>> Nicolò Boschi
>> 
>> 

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Michael Marshall]

> I think there is an issue with the current behavior. When the init
> subscription
> is not created before creating deadLetterProducer, it may result in the
> loss of messages.
> And this is not the expected behavior.

I'm not sure I agree with this assertion. The default in Pulsar is
that a topic does not retain messages until there is a subscription,
and because topic creation does not (currently) imply subscription
creation, auto topic creation can lead to message loss.

I do agree though that it would be helpful to try to remove the risk
posed by auto topic creation.

> I think we can create a consumer with the DLQ init subscription and then
> close the consumer?

This is the current design proposed by this PIP. As I mentioned
earlier, I do not think we should take this approach.

I think the underlying feature request is to make it possible to auto
create a subscription when a topic is auto created. This feature seems
similar to `force_topic_creation`, which is already a field on the
`CommandSubscribe` command. It is documented here [0]:

 >   // If true, the subscribe operation will cause a topic to be
 >   // created if it does not exist already (and if topic auto-creation
 >   // is allowed by broker.
 >   // If false, the subscribe operation will fail if the topic
 >   // does not exist.
 >   optional bool force_topic_creation = 15 [default = true];

What if we extended the `CommandProducer` command to add a
`create_subscription` field? Then, any time a topic is auto
created and this field is true, the broker would auto create a
subscription. There are some details to work out, but I think this
feature would fulfill the needs of this PIP and would also be broadly
useful for many client applications that dynamically create topics.

Thanks,
Michael

[0] 
https://github.com/apache/pulsar/blob/6657a6c2fb23be579e74bac7c7c07a70cabfb4b2/pulsar-common/src/main/proto/PulsarApi.proto#L373-L378




On Wed, Dec 22, 2021 at 5:32 AM PengHui Li  wrote:
>
> I think we can create a consumer with the DLQ init subscription and then
> close the consumer?
> This will make the implementation easier.
>
> Penghui
>
> On Wed, Dec 22, 2021 at 4:49 PM Zike Yang 
> wrote:
>
> > > We can avoid confusion by only attempting to create the subscription
> > > when `initSubscriptionName` is not `null`. As a result, this PIP
> > > wouldn't change the behavior for current implementations.
> > I think there is an issue with the current behavior. When the init
> > subscription
> > is not created before creating deadLetterProducer, it may result in the
> > loss of messages.
> > And this is not the expected behavior. So the purpose of this PIP is to fix
> > this issue.
> > IMO, we need to create that subscription by default to fix the current
> > behavior.
> >
> > > +1 - I think we should expand the binary protocol to add a
> > > `CommandSubscription` or possibly `CommandCreateSubscription`
> > > (`CommandSubscribe` is already taken). Without a new command, we will
> > > introduce race conditions related to consumer creation and will
> > > definitely generate unnecessary errors.
> > +1. `CommandCreateSubscription` seems more reasonable.
> >
> > > Regarding default authorization, this new command would require the
> > > same level of permission for the client since the act of creating a
> > > consumer and the act of creating a subscription both require the same
> > > `AuthAction` permission: Consume.
> > +1
> >
> > Thanks,
> > Zike Yang
> >
> > On Wed, Dec 22, 2021 at 2:40 AM Michael Marshall 
> > wrote:
> > >
> > > > IMO, we won't create the init subscription if
> > `allowAutoSubscriptionCreation`
> > > > is false. Otherwise, it will confuse the user. Users can explicitly
> > create that
> > > > subscription to handle this case.
> > >
> > > That is a good point, but I don't think we should attempt to create
> > > the subscription implicitly via consumer creation because that will
> > > lead to additional failure, which adds unnecessary load on the broker.
> > >
> > > We can avoid confusion by only attempting to create the subscription
> > > when `initSubscriptionName` is not `null`. As a result, this PIP
> > > wouldn't change the behavior for current implementations.
> > >
> > > > Do you mean to create a private method called `createSubscription` in
> > > > the non-admin client? But we don't have a protocol command for
> > > > creating subscription
> > >
> > > Yes, that was what I meant, and you're right, we don't have a protocol
> > > command for this case.
> > >
> > > > Or the other way is to create a new command `CommandSubscription` in
> > > > the protocol to handle this.
> > >
> > > +1 - I think we should expand the binary protocol to add a
> > > `CommandSubscription` or possibly `CommandCreateSubscription`
> > > (`CommandSubscribe` is already taken). Without a new command, we will
> > > introduce race conditions related to consumer creation and will
> > > definitely generate unnecessary errors.
> > >
> > > Regarding default author

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

[Matteo Merli]

> What if we extended the `CommandProducer` command to add a
> `create_subscription` field? Then, any time a topic is auto
> created and this field is true, the broker would auto create a
> subscription. There are some details to work out, but I think this
> feature would fulfill the needs of this PIP and would also be broadly
> useful for many client applications that dynamically create topics.

Oh, that's a very interesting point. I think it'd be easy to add that
as "internal" feature, though I'm a bit puzzled on how to add that to
the producer API

Re: [VOTE] PIP-120: Enable client memory limit by default

[Sijie Guo]

+1

On Wed, Dec 22, 2021 at 2:08 PM Enrico Olivelli  wrote:

> +1 (binding)
>
> Enrico
>
> Il Mer 22 Dic 2021, 16:58 Michael Marshall  ha
> scritto:
>
> > +1
> >
> > - Michael
> >
> > On Wed, Dec 22, 2021 at 6:10 AM Hang Chen  wrote:
> > >
> > > +1
> > >
> > > Thanks,
> > > Hang
> > >
> > > PengHui Li  于2021年12月22日周三 19:28写道：
> > > >
> > > > +1
> > > >
> > > > Penghui
> > > >
> > > > On Wed, Dec 22, 2021 at 6:57 PM ZhangJian He 
> > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Christophe Bornet  于2021年12月22日周三 17:38写道：
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a
> > écrit :
> > > > > >
> > > > > > > This is the voting thread for PIP-120. It will stay open for at
> > least
> > > > > > 48h.
> > > > > > >
> > > > > > > https://github.com/apache/pulsar/issues/13306
> > > > > > >
> > > > > > > 
> > > > > > >
> > > > > > > ## Motivation
> > > > > > >
> > > > > > > In Pulsar 2.8, we have introduced a setting to control the
> > amount of
> > > > > > memory
> > > > > > > used by a client instance.
> > > > > > >
> > > > > > > ```java
> > > > > > > interface ClientBuilder {
> > > > > > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > > > > > }
> > > > > > > ```
> > > > > > >
> > > > > > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no
> > limit
> > > > > is
> > > > > > > being
> > > > > > > enforced.
> > > > > > >
> > > > > > > I think it's a good time for 2.10 to enable this setting by
> > default
> > > > > and,
> > > > > > > correspondingly, to disable by default the producer queue size
> > limit.
> > > > > > >
> > > > > > > This will simplify a lot the configuration that a producer
> > application
> > > > > > will
> > > > > > > have to come up with, when publishing with many
> topic/partitions
> > or
> > > > > > > when messages
> > > > > > > are bigger than expected.
> > > > > > >
> > > > > > > ## Proposed changes
> > > > > > >
> > > > > > > In 2.10 release, for the `ClientBuilder`, change
> > > > > > >   * `memoryLimit`: 0 -> 64 MB
> > > > > > >
> > > > > > > For the `ProducerBuilder`, changes
> > > > > > >   * `maxPendingMessages`: 1000 -> 0
> > > > > > >
> > > > > > > 64MB is picked because it's a small enough memory size that
> will
> > > > > > guarantee
> > > > > > > a very high producer throughput, irrespective of the individual
> > > > > messages
> > > > > > > size.
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Matteo Merli
> > > > > > > 
> > > > > > >
> > > > > >
> > > > >
> >
>

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Sijie Guo]

+1

On Tue, Dec 21, 2021 at 3:23 PM Matteo Merli  wrote:

> This is the voting thread for PIP-119. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13305
>
> ---
>
> ## Motivation
>
> The consistent hashing implementation to uniformly assign keys to consumers
> in the context of a KeyShared subscription, was introduced in
> https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> 2.6.0.
>
> While consistent hashing can use slightly more memory in certain cases, it
> is
> more suitable as a general default implementation, as it leads to a fairer
> distribution of keys across consumers, and avoiding corner cases that
> depend
> on the sequence of addition/removal of consumers.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> splitting ranges or
> # consistent hashing to reassign keys to new consumers
> subscriptionKeySharedUseConsistentHashing=false
> ```
>
> Change its default value to `true`.
>
> The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> be
> able to use the old implementation.
>
> --
> Matteo Merli
> 
>

Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[Sijie Guo]

+1

On Tue, Dec 21, 2021 at 3:22 PM Matteo Merli  wrote:

> https://github.com/apache/pulsar/issues/13304
>
> Following the discussion, I have updated the proposal to also include
> the deprecation and renaming of the config setting name to
> `metadataSessionExpiredPolicy`.
>
>
>
> ---
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata
> operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled to stabilize in 2.9 the configuration setting that
> allows
> brokers to continue operating in a safe mode when the session with
> ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane
> will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid
> any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and
> thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Deprecate the old setting and rename it to
> `metadataSessionExpiredPolicy`, with default value set to `reconnect`.
>

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

[Sijie Guo]

+1

On Tue, Dec 21, 2021 at 3:59 PM Matteo Merli  wrote:

> This is the voting thread for PIP-123. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13346
>
>
> -
> ## Motivation
>
> For a very long time, we have included a CLI command to start the ZooKeeper
> shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> of the ZooKeeper tool `zkCli.sh`.
>
> This is useful in some cases to either verify the content of metadata or
> to perform cleanup and modification tasks for which there is not an
> available
> option through the Pulsar REST APIs.
>
> While it's very useful, there are some drawbacks with the `zookeeper-shell`
> as it is today:
>
>  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> metadata backends, we should have a tool that works across all the
> implementations and presents a single consistent interface.
>
>  2. ZooKeeper shell is designed to be an interactive shell and it's not
> very
> good when trying to do non-interactive scriptable operations.
>
>  3. ZooKeeper is a bit clunky when using it and it requires the user to
> retype
> paths many times. The commands are not very intuitive or documented.
> It's not possible to update z-node with multi-lines content.
>
>  4. We cannot easily add functionality or improvements into ZooKeeper
> shell,
> since it belongs to a different project and the tool has been
> stagnating
> for many years.
>
>  5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> there
> is no easy way to inspect the content from the ZooKeeper shell.
> Additionally, we can format and colorize JSON content to make it
> easier to
> read.
>
>  6. The paths used for metadata resources are also often using encodings
> that
> make it more difficult to construct on the shell tool.
>
> Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> tool,
> though that is a Python script that requires additional dependencies that
> are
> not typically installed, it only works with ZooKeeper, and it only targets
> accessing the managed ledger metadata.
>
> ## Goal
>
> Introduce a new Java CLI tool to access, inspect and modify metadata that
> solves
> all the issues described above.
>
> We would leave the `zookeeper-shell` command for now. In the future, once
> the
> new tool is proven, we can consider removing the `zookeeper-shell` command.
>
>
> ## Proposed changes
>
> Add a new command:
> ```bash
> bin/pulsar metadata
> ```
>
> with several subcommands:
>
>
>  Get
>
> Examples:
> ```bash
> # General path get
> $ pulsar metadata get /my-path
>
>
> # Topic metadata
> $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> {
>   # Managed ledger metadata
> }
>
> # Namespace get
> $ pulsar metadata get namespace my-tenant/my-namespace
> {
>   # Namespace metadata
> }
>
> $ pulsar metadata get ledger 12345
> {
>   # BK ledger metadata
> }
> ```
>
>  Delete
>
> Examples:
> ```bash
> # General path delete
> $ pulsar metadata delete /my-path
>
> # Topic metadata
> $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> ```
>
>  Scan
>
> Examples:
> ```bash
> $ pulsar metadata scan /my-path
> /my-path
> /my-path/1
> /my-path/2
> /my-path/3
> /my-path/3/1
>
> $ pulsar metadata scan --values /my-path
> /my-path
> {value}
>
> /my-path/1
> {value}
>
> /my-path/2
> {value}
> ```
>
>  Shell
>
> ```bash
> $ pulsar metadata shell
> > get topic my-tenant/my-namespace/my-topic
> {
>   # Managed ledger metadata
> }
>
> > delete topic my-tenant/my-namespace/my-topic
>
> > cd /my-path
> > ls
> 1
> 2
> 3
> > delete 1 # Delete keys with relative paths
>
> ```
>
>
> --
> Matteo Merli
> 
>

Re: [VOTE] PIP-117: Change Pulsar standalone defaults

[Sijie Guo]

+1

On Tue, Dec 21, 2021 at 3:49 PM Matteo Merli  wrote:

> This is the voting thread for PIP-117. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13302
>
> 
>
> ## Motivation
>
> Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster,
> where
> all the components are started within the context of a single JVM process.
>
> Users are using the standalone as a way to get quickly started with Pulsar
> or
> in all the cases where it makes sense to have a single node deployment.
>
> Right now, the standalone is starting by default with many components,
> several of
> which are quite complex, since they are designed to be deployed in a
> distributed
> fashion.
>
> ## Goal
>
> Simplify the components of Pulsar standalone to achieve:
>
>  1. Reduce complexity
>  2. Reduce startup time
>  3. Reduce memory and CPU footprint of running standalone
>
> ## Proposed changes
>
> The proposal here is to change some of the default implementations that are
> used for the Pulsar standalone.
>
>  1. **Metadata Store implementation** -->
>   Change from ZooKeeper to RocksDB
>
>  2. **Pulsar functions package backend** -->
>   Change from using DistributedLog to using local filesystem, storing
> the
>   jars directly in the data folder instead of uploading them into BK.
>
>  3. **Pulsar functions state store implementation** -->
>   Change the state store to be backed by a MetadataStore based backed,
>   with the RocksDB implementation.
>
>  4. **Table Service** -->
>   Do not start BK table service by default
>
> ## Compatibility considerations
>
> In order to avoid compatibility issues where users have existing Pulsar
> standalone services that they want to upgrade without conflicts, we will
> follow the principle of keeping the old defaults where there is existing
> data on the disk.
>
> We will add a file, serving the purpose as a flag, in the `data/standalone`
> directory, for example `new-2.10-defaults`.
>
> If the file is present, or if the data directory is completely missing, we
> will adopt the new set of default configuration settings.
>
> If the file is not there, we will continue to use existing defaults and we
> will
> not break the upgrade operation.
>
>
> --
> Matteo Merli
> 
>

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

[Sijie Guo]

Sorry. When you say "we discussed", who are "we"? Is it DataStax?

I believe we want to keep SQL until the code change lands in Trino. Because
there are still users using this component.

- Sijie

On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli 
wrote:

> Lari,
>
> Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari 
> ha scritto:
>
> > Dear Pulsar community members,
> >
> > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> separate
> > repositories" was created in April 2020. The repositories for
> > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> year
> > ago [2].
> >
> > I'd like to propose that we continue with the PIP-62 plan asap, before
> > Apache Pulsar 2.10.0 is released.
> >
> > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > waiting for the Trino Pulsar PR [3] being completed.
> >
> > I am volunteering to making the changes as per the PIP-62 plan.
> > When can we proceed with the plan? Do we need an explicit vote on the
> > mailing list about this?
> >
>
> We discussed this many times in the past year.
> I believe that there is no need to wait
>
> Enrico
>
>
>
> >
> > BR,
> >
> > Lari
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > [3] https://github.com/trinodb/trino/pull/8020
> >
> > Other email threads:
> > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> -
> > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > * Updates on Presto connector for PIP-62 -
> > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> >
>

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

[Hiroyuki Sakai]

+1 (binding)

 - Check signatures/checksums
 - Check the license headers
 - build sources
 - Validate Pub/Sub and Java Functions
 - Validate Connectors
 - Validate Stateful Functions

Regards,
Hiroyuki



From: guo jiwei 
Sent: Tuesday, December 21, 2021 18:33
To: dev@pulsar.apache.org 
Subject: [VOTE] Pulsar Release 2.7.4 Candidate 2

This is the second release candidate for Apache Pulsar, version 2.7.4.

It fixes the following issues:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpulls%3Fq%3Dis%253Apr%2Bis%253Aopen%2Blabel%253Arelease%252F2.7.4%2Bis%253Apr&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C9d0a69fb262049df020308d9c465130b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756760648860410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=NA3mo6Z5ehb7lZF3Mkq5wtFktUxG85jzBqkLY68n5jE%3D&reserved=0+

Release note:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F13391&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C9d0a69fb262049df020308d9c465130b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756760648860410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QIfI2U8xJ5txHVfe0iWrVxfp6WFIsoKL4d6lg1S3aCQ%3D&reserved=0

*** Please download, test and vote on this release. This vote will
stay openfor at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.7.4-candidate-2%2F&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C9d0a69fb262049df020308d9c465130b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756760648860410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iKt4LS6xs%2B8AM3u7kXtCS9smmmYQMrfDQ2B4HbipgBs%3D&reserved=0


SHA-512 checksums:

a56a794bd14c8f4b6d7320496b2be4640a836a7aa940c37aac4f53ae057f8b4887f3f8716f475d2a03c79e3e44bf2ad9234fd05e2b0f58b005a0a4f7f376fc15
 apache-pulsar-2.7.4-bin.tar.gz
03ec488d59d281fdbbbee48f1f62b266cd1163181decbf962d0d58eacc746731d31739955d8afac8bc392547caf2ae76e864d0ad2f2a460cd53920a707a7b40c
 apache-pulsar-2.7.4-src.tar.gz


Maven staging repo:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C9d0a69fb262049df020308d9c465130b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756760648860410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yt%2BJS2QsrpOyOAVE1u4H5GKHg75fgGJdU0ZTQn62KfQ%3D&reserved=0
130

The tag to be voted upon:
v2.7.4-candidate-2 (2774b464f3b7e1ccfdc84d6d390efe7260cbb7fa)
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.7.4-candidate-&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C9d0a69fb262049df020308d9c465130b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756760648860410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fKNc1HedzVYqU%2F2QB0muCdnYwpcrUDSC2WmT040wqZY%3D&reserved=0
2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C9d0a69fb262049df020308d9c465130b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C63775676064886

[GitHub] [pulsar-client-node] izumo27 commented on issue #99: Authenticating using a Token

[GitBox]

izumo27 commented on issue #99:
URL: 
https://github.com/apache/pulsar-client-node/issues/99#issuecomment-127066


   @Matt-Esch What kind of environment did you use? Can you tell us the details 
of the issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] shiv4289 commented on a change in pull request #434: SAML Integration for Login

[GitBox]

shiv4289 commented on a change in pull request #434:
URL: https://github.com/apache/pulsar-manager/pull/434#discussion_r774308333



##
File path: src/main/java/org/apache/pulsar/manager/controller/SSOController.java
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.pulsar.manager.controller;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import java.net.URI;
+import java.util.Map;
+import java.util.Optional;
+import javax.annotation.Resource;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.util.Asserts;
+import org.apache.pulsar.manager.entity.RoleBindingEntity;
+import org.apache.pulsar.manager.entity.RoleBindingRepository;
+import org.apache.pulsar.manager.entity.RolesRepository;
+import org.apache.pulsar.manager.entity.UserInfoEntity;
+import org.apache.pulsar.manager.entity.UsersRepository;
+import org.apache.pulsar.manager.service.JwtService;
+import org.apache.pulsar.manager.utils.SAMLParser;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.validation.annotation.Validated;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.ResponseBody;
+import org.springframework.web.bind.annotation.RestController;
+
+@Slf4j
+@RestController
+@RequestMapping(value = "/pulsar-manager/saml")
+@Api(description = "Calling the request below this class does not require 
authentication because " +
+"the user has not logged in yet.")
+@Validated
+public class SSOController {
+
+
+@Autowired
+private UsersRepository usersRepository;
+
+@Autowired
+private JwtService jwtService;
+
+@Resource
+private RolesRepository rolesRepository;
+
+@Resource
+private RoleBindingRepository roleBindingRepository;
+
+@Value("${sso.redirect.host}")
+private String redirectHost;
+
+@Value("${sso.redirect.port}")
+private int redirectPort;
+
+@Value("${sso.redirect.scheme}")
+private String redirectUrlScheme;
+
+@Value("${sso.certificate}")
+private String certificate;
+
+@Value("${sso.whitelisted.domain}")
+private String whiteListedDomain;
+
+@ApiOperation(value = "Verify SAML Response")
+@ApiResponses({
+@ApiResponse(code = 200, message = "ok"),
+@ApiResponse(code = 404, message = "Not found"),
+@ApiResponse(code = 500, message = "Internal server error")
+})
+@RequestMapping(value = "/sso", method = RequestMethod.POST)
+@ResponseBody
+public void loginByIDP(HttpServletRequest request, HttpServletResponse 
response) throws Exception {
+String samlResponse = request.getParameter("SAMLResponse");
+SAMLParser samlParser = new SAMLParser(certificate);
+Map userDetails = 
samlParser.getUserDetailsFromSAMLResponse(samlResponse);
+
Asserts.check(StringUtils.contains(userDetails.get("email"),whiteListedDomain),"Login
 with domain not supported");

Review comment:
   Makes sense.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] shiv4289 commented on a change in pull request #434: SAML Integration for Login

[GitBox]

shiv4289 commented on a change in pull request #434:
URL: https://github.com/apache/pulsar-manager/pull/434#discussion_r774308333



##
File path: src/main/java/org/apache/pulsar/manager/controller/SSOController.java
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.pulsar.manager.controller;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import java.net.URI;
+import java.util.Map;
+import java.util.Optional;
+import javax.annotation.Resource;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.util.Asserts;
+import org.apache.pulsar.manager.entity.RoleBindingEntity;
+import org.apache.pulsar.manager.entity.RoleBindingRepository;
+import org.apache.pulsar.manager.entity.RolesRepository;
+import org.apache.pulsar.manager.entity.UserInfoEntity;
+import org.apache.pulsar.manager.entity.UsersRepository;
+import org.apache.pulsar.manager.service.JwtService;
+import org.apache.pulsar.manager.utils.SAMLParser;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.validation.annotation.Validated;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.ResponseBody;
+import org.springframework.web.bind.annotation.RestController;
+
+@Slf4j
+@RestController
+@RequestMapping(value = "/pulsar-manager/saml")
+@Api(description = "Calling the request below this class does not require 
authentication because " +
+"the user has not logged in yet.")
+@Validated
+public class SSOController {
+
+
+@Autowired
+private UsersRepository usersRepository;
+
+@Autowired
+private JwtService jwtService;
+
+@Resource
+private RolesRepository rolesRepository;
+
+@Resource
+private RoleBindingRepository roleBindingRepository;
+
+@Value("${sso.redirect.host}")
+private String redirectHost;
+
+@Value("${sso.redirect.port}")
+private int redirectPort;
+
+@Value("${sso.redirect.scheme}")
+private String redirectUrlScheme;
+
+@Value("${sso.certificate}")
+private String certificate;
+
+@Value("${sso.whitelisted.domain}")
+private String whiteListedDomain;
+
+@ApiOperation(value = "Verify SAML Response")
+@ApiResponses({
+@ApiResponse(code = 200, message = "ok"),
+@ApiResponse(code = 404, message = "Not found"),
+@ApiResponse(code = 500, message = "Internal server error")
+})
+@RequestMapping(value = "/sso", method = RequestMethod.POST)
+@ResponseBody
+public void loginByIDP(HttpServletRequest request, HttpServletResponse 
response) throws Exception {
+String samlResponse = request.getParameter("SAMLResponse");
+SAMLParser samlParser = new SAMLParser(certificate);
+Map userDetails = 
samlParser.getUserDetailsFromSAMLResponse(samlResponse);
+
Asserts.check(StringUtils.contains(userDetails.get("email"),whiteListedDomain),"Login
 with domain not supported");

Review comment:
   Makes sense. Lets drop this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Add definition to our cherry picking process

[Michael Marshall]

I am very interested in Jonathan's proposal. I think we could really
benefit from a workflow that keeps branches in a (near) ready to
release state and prioritizes the stability of our active branches.

It would be great to hear from other community members regarding our
current process, its benefits and challenges, and potential
alternatives.

For what it's worth, I would be happy to help transition our process
to a merge based workflow.

Thanks,
Michael



On Wed, Dec 15, 2021 at 4:54 PM Jonathan Ellis  wrote:
>
> First, it's often not an issue -- new contributors typically start with
> small patches that the reviewer or committer can easily rebase to the
> appropriate branch, and by the time they're working on more invasive fixes
> (we don't have any of those, right? :) they get used to how it works.
>
> But when the merge is NOT trivial then the reviewer will ask the
> contributor to post separate patches against the newer branches, this
> reduces the burden on the reviewer (or the release manager!) to do all the
> merging and it is less error prone for the original author to perform a
> tricky merge.
>
> Recent example from Cassandra where reviewer Joey Lynch asked contributor
> NV Harikrishna to do this:
> https://issues.apache.org/jira/browse/CASSANDRA-14898
>
>
>
> On Wed, Dec 15, 2021 at 3:21 PM Enrico Olivelli  wrote:
>
> > Jonathan,
> >
> > Il Mer 15 Dic 2021, 21:11 Jonathan Ellis  ha scritto:
> >
> > > What are the benefits of cherry picking vs committing to the oldest
> > > relevant branch and merging forwards?
> >
> >
> > We (and most of the ASF projects I am involved in) ask contributors to
> > target the master branch for the patches.
> >
> > How can we do what you suggest?
> > Asking people to send PRs against the oldest branch relevant to the patch?
> >
> > Probably the contributor is not aware of the other branches.
> >
> > Can you please explain more your idea?
> >
> > Enrico
> >
> > Git is designed around merge since
> > > that preserves the commit sha which allows it to be tracked across
> > > branches.  And a merge based workflow avoids the problem here by design.
> > >
> > > On Wed, Dec 15, 2021 at 1:54 PM Michael Marshall 
> > > wrote:
> > >
> > > > Hello Pulsar Community,
> > > >
> > > > As far as I can tell, we do not have a publicly defined process for
> > > > cherry-picking commits to release branches [0]. As a new committer,
> > > > I'd like to provide my newly gained perspective and ask that we add
> > > > some guidance to our wiki.
> > > >
> > > > Regarding cherry picking, I see two patterns in Pulsar:
> > > > 1. Some committers cherry pick commits when they merge a PR.
> > > > 2. Some committers leave PRs to be cherry picked right before tagging
> > > > the dot release. The release manager is expected to cherry pick these
> > > > commits.
> > > >
> > > > Anecdotally, pattern 2 is more common.
> > > >
> > > > I am concerned that pattern 2 creates a lot of extra work for the
> > > > release manager, leaves the community with little time to build and
> > > > test release branches before tagging release candidates, and leaves us
> > > > unable to quickly respond to security issues.
> > > >
> > > > If we switch to pattern 1, merging PRs may become more time consuming.
> > > > However, it lightens the load on release managers and makes the
> > > > project more nimble. Further, resolving conflicts should be easier, as
> > > > the committer already has the PR's context, which could reduce the
> > > > risk of regressions.
> > > >
> > > > For example, the 2.7.4 release is delayed because of pattern 2.
> > > > Penghui and I cherry picked many commits over the recent days because
> > > > branch-2.7 was not up to date. Now, the tests aren't passing for the
> > > > branch, and since we shouldn't cut a release until the tests pass, we
> > > > haven't released the patch for the Log4Shell CVE.
> > > >
> > > > Another solution is to remove the expectation that the release manager
> > > > cherry picks commits. The downside here is that we will possibly miss
> > > > important bug fixes that would improve dot release stability.
> > > >
> > > > I propose that we try to use pattern 1 (or something close to it) for
> > > > our process.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > [0] https://github.com/apache/pulsar/wiki/Committer-Guide
> > > >
> > >
> > >
> > > --
> > > Jonathan Ellis
> > > co-founder, http://www.datastax.com
> > > @spyced
> > >
> >
>
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

[Michael Marshall]

> I am volunteering to making the changes as per the PIP-62 plan.

I think Lari is proposing to complete the work to move the
Pulsar SQL code base to the repositories defined in PIP 62.

Assuming that is what he meant, I am +1 on completing that now.
Based on reading through the mailing list threads Lari referenced, we
delayed moving the code base to the other repos with the hope that
contributing the code to Trino/Presto would be done before the 2.8.0
release. I think we should move forward with the initial PIP instead
of waiting for the Trino PR to get merged.

- Michael

On Wed, Dec 22, 2021 at 6:39 PM Sijie Guo  wrote:
>
> Sorry. When you say "we discussed", who are "we"? Is it DataStax?
>
> I believe we want to keep SQL until the code change lands in Trino. Because
> there are still users using this component.
>
> - Sijie
>
> On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli 
> wrote:
>
> > Lari,
> >
> > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari 
> > ha scritto:
> >
> > > Dear Pulsar community members,
> > >
> > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > separate
> > > repositories" was created in April 2020. The repositories for
> > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> > year
> > > ago [2].
> > >
> > > I'd like to propose that we continue with the PIP-62 plan asap, before
> > > Apache Pulsar 2.10.0 is released.
> > >
> > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > > waiting for the Trino Pulsar PR [3] being completed.
> > >
> > > I am volunteering to making the changes as per the PIP-62 plan.
> > > When can we proceed with the plan? Do we need an explicit vote on the
> > > mailing list about this?
> > >
> >
> > We discussed this many times in the past year.
> > I believe that there is no need to wait
> >
> > Enrico
> >
> >
> >
> > >
> > > BR,
> > >
> > > Lari
> > >
> > > [1]
> > >
> > >
> > https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > [2]
> > >
> > >
> > https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > [3] https://github.com/trinodb/trino/pull/8020
> > >
> > > Other email threads:
> > > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> > -
> > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > * Updates on Presto connector for PIP-62 -
> > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > >
> >

Re: [VOTE] PIP-117: Change Pulsar standalone defaults

[Michael Marshall]

+1

- Michael

On Wed, Dec 22, 2021 at 6:18 PM Sijie Guo  wrote:
>
> +1
>
> On Tue, Dec 21, 2021 at 3:49 PM Matteo Merli  wrote:
>
> > This is the voting thread for PIP-117. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13302
> >
> > 
> >
> > ## Motivation
> >
> > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster,
> > where
> > all the components are started within the context of a single JVM process.
> >
> > Users are using the standalone as a way to get quickly started with Pulsar
> > or
> > in all the cases where it makes sense to have a single node deployment.
> >
> > Right now, the standalone is starting by default with many components,
> > several of
> > which are quite complex, since they are designed to be deployed in a
> > distributed
> > fashion.
> >
> > ## Goal
> >
> > Simplify the components of Pulsar standalone to achieve:
> >
> >  1. Reduce complexity
> >  2. Reduce startup time
> >  3. Reduce memory and CPU footprint of running standalone
> >
> > ## Proposed changes
> >
> > The proposal here is to change some of the default implementations that are
> > used for the Pulsar standalone.
> >
> >  1. **Metadata Store implementation** -->
> >   Change from ZooKeeper to RocksDB
> >
> >  2. **Pulsar functions package backend** -->
> >   Change from using DistributedLog to using local filesystem, storing
> > the
> >   jars directly in the data folder instead of uploading them into BK.
> >
> >  3. **Pulsar functions state store implementation** -->
> >   Change the state store to be backed by a MetadataStore based backed,
> >   with the RocksDB implementation.
> >
> >  4. **Table Service** -->
> >   Do not start BK table service by default
> >
> > ## Compatibility considerations
> >
> > In order to avoid compatibility issues where users have existing Pulsar
> > standalone services that they want to upgrade without conflicts, we will
> > follow the principle of keeping the old defaults where there is existing
> > data on the disk.
> >
> > We will add a file, serving the purpose as a flag, in the `data/standalone`
> > directory, for example `new-2.10-defaults`.
> >
> > If the file is present, or if the data directory is completely missing, we
> > will adopt the new set of default configuration settings.
> >
> > If the file is not there, we will continue to use existing defaults and we
> > will
> > not break the upgrade operation.
> >
> >
> > --
> > Matteo Merli
> > 
> >

Re: [DISCUSSION] PIP-120: Enable client memory limit by default

[Michael Marshall]

That makes sense, thank you for your explanation, Matteo.

- Michael

On Tue, Dec 21, 2021 at 4:03 PM Matteo Merli  wrote:
>
> > Did you consider determining the default limit by inspecting the JVM's 
> > runtime?
> > For example, in the broker, we set `maxMessagePublishBufferSizeInMB` to 1/2
> > of direct memory, by default.
> >
> > I concede that this config is more complicated than a broker config since
> > we're talking about a client application. However, it seems like 
> > applications
> > with 1 GB memory would probably want a different setting than one with
> > 32 GB memory, and a fractional value could make the client work better
> > "out of the box."
> >
> > We cannot make everyone happy with a default value, so if this dynamic
> > default is too complicated to implement, I'm good with 64 MB. :)
>
> The issue is that we don't know what are the application assumptions
> with respect to the memory. Eg; even if it's given 32GB to the JVM, it
> might want to use all of it.
>
> The other aspect is that after a point, increasing the memory size,
> won't increase the throughput anymore, and it will become just a waste
> of RAM.

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Michael Marshall]

+1

- Michael

On Wed, Dec 22, 2021 at 6:17 PM Sijie Guo  wrote:
>
> +1
>
> On Tue, Dec 21, 2021 at 3:23 PM Matteo Merli  wrote:
>
> > This is the voting thread for PIP-119. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13305
> >
> > ---
> >
> > ## Motivation
> >
> > The consistent hashing implementation to uniformly assign keys to consumers
> > in the context of a KeyShared subscription, was introduced in
> > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> > 2.6.0.
> >
> > While consistent hashing can use slightly more memory in certain cases, it
> > is
> > more suitable as a general default implementation, as it leads to a fairer
> > distribution of keys across consumers, and avoiding corner cases that
> > depend
> > on the sequence of addition/removal of consumers.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > splitting ranges or
> > # consistent hashing to reassign keys to new consumers
> > subscriptionKeySharedUseConsistentHashing=false
> > ```
> >
> > Change its default value to `true`.
> >
> > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> > be
> > able to use the old implementation.
> >
> > --
> > Matteo Merli
> > 
> >

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

[Michael Marshall]

+1

- Michael

On Wed, Dec 22, 2021 at 6:18 PM Sijie Guo  wrote:
>
> +1
>
> On Tue, Dec 21, 2021 at 3:59 PM Matteo Merli  wrote:
>
> > This is the voting thread for PIP-123. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13346
> >
> >
> > -
> > ## Motivation
> >
> > For a very long time, we have included a CLI command to start the ZooKeeper
> > shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> > of the ZooKeeper tool `zkCli.sh`.
> >
> > This is useful in some cases to either verify the content of metadata or
> > to perform cleanup and modification tasks for which there is not an
> > available
> > option through the Pulsar REST APIs.
> >
> > While it's very useful, there are some drawbacks with the `zookeeper-shell`
> > as it is today:
> >
> >  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > metadata backends, we should have a tool that works across all the
> > implementations and presents a single consistent interface.
> >
> >  2. ZooKeeper shell is designed to be an interactive shell and it's not
> > very
> > good when trying to do non-interactive scriptable operations.
> >
> >  3. ZooKeeper is a bit clunky when using it and it requires the user to
> > retype
> > paths many times. The commands are not very intuitive or documented.
> > It's not possible to update z-node with multi-lines content.
> >
> >  4. We cannot easily add functionality or improvements into ZooKeeper
> > shell,
> > since it belongs to a different project and the tool has been
> > stagnating
> > for many years.
> >
> >  5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> > there
> > is no easy way to inspect the content from the ZooKeeper shell.
> > Additionally, we can format and colorize JSON content to make it
> > easier to
> > read.
> >
> >  6. The paths used for metadata resources are also often using encodings
> > that
> > make it more difficult to construct on the shell tool.
> >
> > Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> > tool,
> > though that is a Python script that requires additional dependencies that
> > are
> > not typically installed, it only works with ZooKeeper, and it only targets
> > accessing the managed ledger metadata.
> >
> > ## Goal
> >
> > Introduce a new Java CLI tool to access, inspect and modify metadata that
> > solves
> > all the issues described above.
> >
> > We would leave the `zookeeper-shell` command for now. In the future, once
> > the
> > new tool is proven, we can consider removing the `zookeeper-shell` command.
> >
> >
> > ## Proposed changes
> >
> > Add a new command:
> > ```bash
> > bin/pulsar metadata
> > ```
> >
> > with several subcommands:
> >
> >
> >  Get
> >
> > Examples:
> > ```bash
> > # General path get
> > $ pulsar metadata get /my-path
> >
> >
> > # Topic metadata
> > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > {
> >   # Managed ledger metadata
> > }
> >
> > # Namespace get
> > $ pulsar metadata get namespace my-tenant/my-namespace
> > {
> >   # Namespace metadata
> > }
> >
> > $ pulsar metadata get ledger 12345
> > {
> >   # BK ledger metadata
> > }
> > ```
> >
> >  Delete
> >
> > Examples:
> > ```bash
> > # General path delete
> > $ pulsar metadata delete /my-path
> >
> > # Topic metadata
> > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > ```
> >
> >  Scan
> >
> > Examples:
> > ```bash
> > $ pulsar metadata scan /my-path
> > /my-path
> > /my-path/1
> > /my-path/2
> > /my-path/3
> > /my-path/3/1
> >
> > $ pulsar metadata scan --values /my-path
> > /my-path
> > {value}
> >
> > /my-path/1
> > {value}
> >
> > /my-path/2
> > {value}
> > ```
> >
> >  Shell
> >
> > ```bash
> > $ pulsar metadata shell
> > > get topic my-tenant/my-namespace/my-topic
> > {
> >   # Managed ledger metadata
> > }
> >
> > > delete topic my-tenant/my-namespace/my-topic
> >
> > > cd /my-path
> > > ls
> > 1
> > 2
> > 3
> > > delete 1 # Delete keys with relative paths
> >
> > ```
> >
> >
> > --
> > Matteo Merli
> > 
> >