[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #210: Remove references to tag: 2.6.0 in examples

[GitBox]

lhotari opened a new pull request #210:
URL: https://github.com/apache/pulsar-helm-chart/pull/210


   ### Motivation
   
   It's better to not maintain out-dated examples referencing the 2.6.0 tag 
version.
   
   ### Modifications
   
   - remove out-dated examples


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #211: Add -XX:+ExitOnOutOfMemoryError to Zookeeper's PULSAR_GC parameters

[GitBox]

lhotari opened a new pull request #211:
URL: https://github.com/apache/pulsar-helm-chart/pull/211


   ### Motivation
   
   - `-XX:+ExitOnOutOfMemoryError` is used for other components
   - it's better to exit the process when an OOME happens since an OOME can 
leave Zookeeper in a bad state
   
   ### Modifications
   
   - Add `-XX:+ExitOnOutOfMemoryError` to Zookeeper's `PULSAR_GC` parameters in 
the default values.yaml file
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari merged pull request #206: Upgrade default images for Grafana & Pulsar Manager

[GitBox]

lhotari merged pull request #206:
URL: https://github.com/apache/pulsar-helm-chart/pull/206


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[VOTE] PIP-124: Create init subscription before sending message to DLQ

[Zike Yang]

Hi Pulsar Community,

This is the voting thread for PIP-124. It will stay open for at least
48 hours. Pasted below for quoting convenience.

Here is the issue for this PIP: https://github.com/apache/pulsar/issues/13408



## Motivation

If we enable the DLQ when consuming messages. For some messages that
can't be processed successfully, the messages will be moved to the
DLQ, but if we do not specify the data retention for the namespace or
create a subscription for the DLQ to retain the data, the data of the
DLQ will be removed automatically. Therefore, we need to create the
initial subscription before sending messages to the DLQ.

## Goal

Users can set the initial subscription name in the DeadLetterPolicy.
The consumer will create the initial subscription before sending
messages to the DLQ. At this point, subsequent messages produced to
the DLQ are not automatically deleted unexpectedly. If
`allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
subscription won't be created automatically. Otherwise, it will
confuse the user. Users can explicitly create that subscription to
handle this case.

This PIP needs to be compatible with the previous behavior. The
initial subscription name in the DeadLetterPolicy is optional. The
default behavior is not to create the initial subscription which is
consistent with the original behavior.

## API Changes

* Add `initSubscriptionName` to the `DeadLetterPolicy`

```java
/**
 * Name of the initial subscription name of the dead letter topic.
 * If this field is not set, the initial subscription will not be created.
 */
private String initSubscriptionName;
```

* Add a new field to the `CommandProducer`. The broker will use this
field to create the initial subscription.

```proto
optional string initial_subscription_name
```

* Add a new config to the Producer Configuration. This allows the
consumer to specify the initial subscription when creating the
deadLetterProducer.

```java
/**
 * Use this config to automatically create an initial subscription
when creating the topic.
 * If this field is not set, the initial subscription will not be created.
 *
 * @param initialSubscriptionName
 *  Name of the initial subscription of the topic.
 * @return the producer builder instance
 */
ProducerBuilder initialSubscriptionName(String initialSubscriptionName);
```


## Implementation

When the deadLetterProducer is initialized, the consumer will set the
initial subscription of the deadLetterProducer according to the
DeadLetterPolicy.

When the broker creates a producer(after receiving the
CommandProducer), if it finds that the topic does not exist, it will
not only create the topic(if the topic automatically creation is
enabled) but also create the initial subscription(if the initial
subscription name is set). When creating an initial subscription, the
user needs to have the `canConsume` permission and the
`allowAutoSubscriptionCreation` needs to be enabled.

## Reject Alternatives

### Create the initial subscription using the consumer

Before the deadLetterProducer is initialized, the consumer first tries
to create a deadLetterConsumer using the initial subscription name in
the DeadLetterPolicy. When this subscription already exists, the
ConsumerBusy exception will occur. In this case, we can ignore that
exception and create the deadLetterProducer.

This is the original solution for this PIP. But this introduces too
much workload for the client. In the subsequent discussion, we try to
add a new command `CommandCreateSubscription` to avoid creating and
closing the deadLetterConsumer, but again, this introduces workload as
well as greater complexity.

### Use the retention policy to retent the data of the DLQ

Before creating the deadLetterProducer,  an admin can create a
retention policy for the topic or namespace. Then, consumers of the
topic have the duration of the retention policy to discover the topic
and create a subscription before messages are lost.

But users are not easy to set a different data retention policy or
create a new subscription for a lazy created DLQ topic.

### Add a new policy to retain data when the topic has no subscriptions

Adding a new policy for pulsar topics: a namespace or a topic level
policy that makes it possible to retain messages indefinitely when a
topic has no subscriptions.

For the auto-created topic, the subscription can only be determined at
the time of creation. It may or may not create. If users are able to
determine which consumers are, and these consumers need to receive any
message sent by the producer, they should create the topic and
subscription manually or use the consumer to trigger the topic
auto-creation, not the producer.

### Add the initial subscription name field to the metadata of the
CommandProducer

Use producer metadata to carry the init subscription name so that we
don't need to introduce a new field in CommandProducer, and the new
field looks a little confusing.

While this does not chan

Note for pulsar-client checkstyle PR

[Yunze Xu]

Hi all (especially committers),

I opened a PR to enable checkstyle plugin for pulsar-client module just now.
See https://github.com/apache/pulsar/pull/13940 
.

It’s really a huge PR but I hope it could be included in Pulsar 2.10 release
because pulsar-client is one of the most huge modules of Pulsar. And the
code style has been out of control for a long time.

For Pulsar committers, please be careful to merge PRs that modify code in
pulsar-client module after this PR is merged because it could make master
branch broken.

Thanks,
Yunze

Re: Note for pulsar-client checkstyle PR

[Enrico Olivelli]

Yunze,

Il giorno mar 25 gen 2022 alle ore 13:14 Yunze Xu
 ha scritto:
>
> Hi all (especially committers),
>
> I opened a PR to enable checkstyle plugin for pulsar-client module just now.
> See https://github.com/apache/pulsar/pull/13940 
> .


Great work.
>
> It’s really a huge PR but I hope it could be included in Pulsar 2.10 release

+1

> because pulsar-client is one of the most huge modules of Pulsar. And the
> code style has been out of control for a long time.
>
> For Pulsar committers, please be careful to merge PRs that modify code in
> pulsar-client module after this PR is merged because it could make master
> branch broken.

I am not sure if there is a way to revalidate the PRs.
IIUC when you close/reopen a PR it is not merged again with latest
master before rerunning CI

Enrico

>
> Thanks,
> Yunze

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[mattison chao]

+1 (non-binding)

> On Jan 25, 2022, at 5:34 PM, Zike Yang  wrote:
> 
> Hi Pulsar Community,
> 
> This is the voting thread for PIP-124. It will stay open for at least
> 48 hours. Pasted below for quoting convenience.
> 
> Here is the issue for this PIP: https://github.com/apache/pulsar/issues/13408
> 
> 
> 
> ## Motivation
> 
> If we enable the DLQ when consuming messages. For some messages that
> can't be processed successfully, the messages will be moved to the
> DLQ, but if we do not specify the data retention for the namespace or
> create a subscription for the DLQ to retain the data, the data of the
> DLQ will be removed automatically. Therefore, we need to create the
> initial subscription before sending messages to the DLQ.
> 
> ## Goal
> 
> Users can set the initial subscription name in the DeadLetterPolicy.
> The consumer will create the initial subscription before sending
> messages to the DLQ. At this point, subsequent messages produced to
> the DLQ are not automatically deleted unexpectedly. If
> `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> subscription won't be created automatically. Otherwise, it will
> confuse the user. Users can explicitly create that subscription to
> handle this case.
> 
> This PIP needs to be compatible with the previous behavior. The
> initial subscription name in the DeadLetterPolicy is optional. The
> default behavior is not to create the initial subscription which is
> consistent with the original behavior.
> 
> ## API Changes
> 
> * Add `initSubscriptionName` to the `DeadLetterPolicy`
> 
> ```java
> /**
> * Name of the initial subscription name of the dead letter topic.
> * If this field is not set, the initial subscription will not be created.
> */
> private String initSubscriptionName;
> ```
> 
> * Add a new field to the `CommandProducer`. The broker will use this
> field to create the initial subscription.
> 
> ```proto
> optional string initial_subscription_name
> ```
> 
> * Add a new config to the Producer Configuration. This allows the
> consumer to specify the initial subscription when creating the
> deadLetterProducer.
> 
> ```java
> /**
> * Use this config to automatically create an initial subscription
> when creating the topic.
> * If this field is not set, the initial subscription will not be created.
> *
> * @param initialSubscriptionName
> *  Name of the initial subscription of the topic.
> * @return the producer builder instance
> */
> ProducerBuilder initialSubscriptionName(String initialSubscriptionName);
> ```
> 
> 
> ## Implementation
> 
> When the deadLetterProducer is initialized, the consumer will set the
> initial subscription of the deadLetterProducer according to the
> DeadLetterPolicy.
> 
> When the broker creates a producer(after receiving the
> CommandProducer), if it finds that the topic does not exist, it will
> not only create the topic(if the topic automatically creation is
> enabled) but also create the initial subscription(if the initial
> subscription name is set). When creating an initial subscription, the
> user needs to have the `canConsume` permission and the
> `allowAutoSubscriptionCreation` needs to be enabled.
> 
> ## Reject Alternatives
> 
> ### Create the initial subscription using the consumer
> 
> Before the deadLetterProducer is initialized, the consumer first tries
> to create a deadLetterConsumer using the initial subscription name in
> the DeadLetterPolicy. When this subscription already exists, the
> ConsumerBusy exception will occur. In this case, we can ignore that
> exception and create the deadLetterProducer.
> 
> This is the original solution for this PIP. But this introduces too
> much workload for the client. In the subsequent discussion, we try to
> add a new command `CommandCreateSubscription` to avoid creating and
> closing the deadLetterConsumer, but again, this introduces workload as
> well as greater complexity.
> 
> ### Use the retention policy to retent the data of the DLQ
> 
> Before creating the deadLetterProducer,  an admin can create a
> retention policy for the topic or namespace. Then, consumers of the
> topic have the duration of the retention policy to discover the topic
> and create a subscription before messages are lost.
> 
> But users are not easy to set a different data retention policy or
> create a new subscription for a lazy created DLQ topic.
> 
> ### Add a new policy to retain data when the topic has no subscriptions
> 
> Adding a new policy for pulsar topics: a namespace or a topic level
> policy that makes it possible to retain messages indefinitely when a
> topic has no subscriptions.
> 
> For the auto-created topic, the subscription can only be determined at
> the time of creation. It may or may not create. If users are able to
> determine which consumers are, and these consumers need to receive any
> message sent by the producer, they should create the topic and
> subscription manually or use the consumer to trigger the topic
> auto-c

[VOTE] Pulsar Release 2.9.2 Candidate 2

[Ran Gao]

Sorry, the 2.9.2 release candidate-1 has a wrong sign certificate.

This is the second release candidate for Apache Pulsar, version 2.9.2.

*** Please download, test, and vote on this release. This vote will stay
open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-2/

SHA-512 checksums:

563f65582c5307b4ef1e0322958ed19d7c181fb8bb8d7b8cab06ab0a6adb5520f7d18b6f97960b93c3318815529a8b8721e00e9cc9484532a2e5ed3221450094
 ./apache-pulsar-2.9.2-bin.tar.gz
60d1049611b938b0ddc769132124d43820728afc8a06813a5ec9efc095c5497c59d9bbcaaf7df5b0c0e97e051d66f59c1f8ee08885d05ca2c635773e0283770a
 ./apache-pulsar-2.9.2-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1136

The tag to be voted upon:
v2.9.2-candidate-2 (8a5d2253b888b3b865a2aedf635d672821c7)
https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

[GitHub] [pulsar-helm-chart] lhotari closed pull request #209: fix(charts): missing eof on bk sts

[GitBox]

lhotari closed pull request #209:
URL: https://github.com/apache/pulsar-helm-chart/pull/209


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari commented on pull request #209: fix(charts): missing eof on bk sts

[GitBox]

lhotari commented on pull request #209:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/209#issuecomment-1020832918


   @ericsyh This problem is already fixed by #208 . I have merged it now. I'll 
close this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari merged pull request #208: Fix ci error caused by wrong block of if clause

[GitBox]

lhotari merged pull request #208:
URL: https://github.com/apache/pulsar-helm-chart/pull/208


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] ericsyh commented on pull request #209: fix(charts): missing eof on bk sts

[GitBox]

ericsyh commented on pull request #209:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/209#issuecomment-1020897852


   > @ericsyh This problem is already fixed by #208 . I have merged it now. 
I'll close this PR.
   
   OK, sure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari merged pull request #206: Upgrade default images for Grafana & Pulsar Manager

[GitBox]

lhotari merged pull request #206:
URL: https://github.com/apache/pulsar-helm-chart/pull/206


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] Matt-Esch commented on issue #99: Authenticating using a Token

[GitBox]

Matt-Esch commented on issue #99:
URL: 
https://github.com/apache/pulsar-client-node/issues/99#issuecomment-1020882307


   Segfault issue tracked here: 
https://github.com/apache/pulsar-client-node/issues/191 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] michaeljmarshall commented on issue #441: Support HerdDB in the standard docker image.

[GitBox]

michaeljmarshall commented on issue #441:
URL: https://github.com/apache/pulsar-manager/issues/441#issuecomment-1020387920


   @mattisonchao - yes, that makes sense to me for docker-compose. Ideally, we 
will also make it configurable in the pulsar helm chart.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Architecture of function authorization for process mode

[Jerry Peng]

Devin,

You can customize how each function authenticates with brokers by creating
a implementation of this interface:

https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/FunctionAuthProvider.java

and setting the class name of your custom function auth provider here:

https://github.com/apache/pulsar/blob/ed10d82dfe7645ddf3faa45df6a6597029fab4cd/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java#L489

However, just as Matteo stated, there is not a great way to restrict access
to credentials/tokens in process mode.  A user can arguably write some code
in his or her function to read out the credentials of other functions
running on the same machine.  Given that you are not worried about
malicious users, this can help prevent bugs / unintentional actions like a
user accidentally submitting a function to read/write from/to a topic it is
not suppose to.

Best,

Jerry

On Mon, Jan 24, 2022 at 11:57 PM Matteo Merli 
wrote:

> The only recommended way to run a multi-tenant Pulsar functions
> clusters is to run it with Kubernetes runtime.
>
> In thread or process runtime, there is no reliable way to restrict the
> access to the credentials of each function instance (since it needs to
> be readable by the same unix user), or for what it matters, to
> restrict the resources that this function has access to (eg: cpu,
> memory, network, disk..).
>
>
> --
> Matteo Merli
> 
>
> On Mon, Jan 24, 2022 at 11:54 PM Niclas Hedhman 
> wrote:
> >
> >
> > This sounds quite important, and I would like to hear from the devs if
> > this is correct or something is misunderstood. I am about to recommend
> > (or not) a critical multi-tenant system, so would like to get the full
> > picture.
> >
> > Thanks
> > Niclas
> >
> > On 2022-01-21 15:51, Devin Bost wrote:
> > > I discovered that all functions running in process mode inherit the
> > > same
> > > role from the token specified in the functions_worker.yml file. So,
> > > that
> > > means that all functions in process mode have the same authorization
> > > scope,
> > > which must be broad enough to ensure that all functions have the
> > > required
> > > permission to operate. Architecturally, this design limits the ability
> > > to
> > > enable self-service/multi-tenant management of process-mode functions
> > > because it means that functions in one tenant automatically have
> > > permission
> > > to access topics in all other tenants.
> > > Even if a custom PulsarAuthorizationProvider is used, there's no
> > > current
> > > way to distinguish roles between functions in process mode because they
> > > all
> > > share the same token. In the code path where authorization is checked (
> > >
> https://github.com/apache/pulsar/blob/adcbe0f118ece0999b8603f37010194b44c241b4/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java#L90
> ),
> > > there's no way to see which function is originating the request.  The
> > > original principal used for the authorization check (
> > >
> https://github.com/apache/pulsar/blob/3e6fedf9b69758c13d92c6ff75a7bd779038543a/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest/TopicsBase.java#L758
> )
> > > is derived from the role (
> > >
> https://github.com/apache/pulsar/blob/cba8800de1013d0e8ac81f43ecb040a55978c358/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java#L369
> ),
> > > so during the authorization check, it doesn't appear that there's any
> > > other
> > > information about who is initiating the request that could be used. (It
> > > appears that there's no way for us to securely know who the requester
> > > is
> > > except via their token.)
> > >
> > > This brings me to wondering what the vision is for functions in process
> > > mode. Are they intended to only be managed by an operator with
> > > administrative access to the cluster? If there's an interest in
> > > supporting
> > > multi-tenant permission scoping for process mode functions, what's the
> > > right way to do it?
> > >
> > > Devin G. Bost
>

Re: Architecture of function authorization for process mode

[Niclas Hedhman]

On 2022-01-25 08:57, Matteo Merli wrote:

The only recommended way to run a multi-tenant Pulsar functions
clusters is to run it with Kubernetes runtime.

In thread or process runtime, there is no reliable way to restrict the
access to the credentials of each function instance (since it needs to
be readable by the same unix user), or for what it matters, to
restrict the resources that this function has access to (eg: cpu,
memory, network, disk..).


Thank you, that helps a lot.

Niclas

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[Jia Zhai]

+1



On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
wrote:

> +1 (non-binding)
>
> > On Jan 25, 2022, at 5:34 PM, Zike Yang 
> wrote:
> >
> > Hi Pulsar Community,
> >
> > This is the voting thread for PIP-124. It will stay open for at least
> > 48 hours. Pasted below for quoting convenience.
> >
> > Here is the issue for this PIP:
> https://github.com/apache/pulsar/issues/13408
> >
> > 
> >
> > ## Motivation
> >
> > If we enable the DLQ when consuming messages. For some messages that
> > can't be processed successfully, the messages will be moved to the
> > DLQ, but if we do not specify the data retention for the namespace or
> > create a subscription for the DLQ to retain the data, the data of the
> > DLQ will be removed automatically. Therefore, we need to create the
> > initial subscription before sending messages to the DLQ.
> >
> > ## Goal
> >
> > Users can set the initial subscription name in the DeadLetterPolicy.
> > The consumer will create the initial subscription before sending
> > messages to the DLQ. At this point, subsequent messages produced to
> > the DLQ are not automatically deleted unexpectedly. If
> > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> > subscription won't be created automatically. Otherwise, it will
> > confuse the user. Users can explicitly create that subscription to
> > handle this case.
> >
> > This PIP needs to be compatible with the previous behavior. The
> > initial subscription name in the DeadLetterPolicy is optional. The
> > default behavior is not to create the initial subscription which is
> > consistent with the original behavior.
> >
> > ## API Changes
> >
> > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> >
> > ```java
> > /**
> > * Name of the initial subscription name of the dead letter topic.
> > * If this field is not set, the initial subscription will not be created.
> > */
> > private String initSubscriptionName;
> > ```
> >
> > * Add a new field to the `CommandProducer`. The broker will use this
> > field to create the initial subscription.
> >
> > ```proto
> > optional string initial_subscription_name
> > ```
> >
> > * Add a new config to the Producer Configuration. This allows the
> > consumer to specify the initial subscription when creating the
> > deadLetterProducer.
> >
> > ```java
> > /**
> > * Use this config to automatically create an initial subscription
> > when creating the topic.
> > * If this field is not set, the initial subscription will not be created.
> > *
> > * @param initialSubscriptionName
> > *  Name of the initial subscription of the topic.
> > * @return the producer builder instance
> > */
> > ProducerBuilder initialSubscriptionName(String
> initialSubscriptionName);
> > ```
> >
> >
> > ## Implementation
> >
> > When the deadLetterProducer is initialized, the consumer will set the
> > initial subscription of the deadLetterProducer according to the
> > DeadLetterPolicy.
> >
> > When the broker creates a producer(after receiving the
> > CommandProducer), if it finds that the topic does not exist, it will
> > not only create the topic(if the topic automatically creation is
> > enabled) but also create the initial subscription(if the initial
> > subscription name is set). When creating an initial subscription, the
> > user needs to have the `canConsume` permission and the
> > `allowAutoSubscriptionCreation` needs to be enabled.
> >
> > ## Reject Alternatives
> >
> > ### Create the initial subscription using the consumer
> >
> > Before the deadLetterProducer is initialized, the consumer first tries
> > to create a deadLetterConsumer using the initial subscription name in
> > the DeadLetterPolicy. When this subscription already exists, the
> > ConsumerBusy exception will occur. In this case, we can ignore that
> > exception and create the deadLetterProducer.
> >
> > This is the original solution for this PIP. But this introduces too
> > much workload for the client. In the subsequent discussion, we try to
> > add a new command `CommandCreateSubscription` to avoid creating and
> > closing the deadLetterConsumer, but again, this introduces workload as
> > well as greater complexity.
> >
> > ### Use the retention policy to retent the data of the DLQ
> >
> > Before creating the deadLetterProducer,  an admin can create a
> > retention policy for the topic or namespace. Then, consumers of the
> > topic have the duration of the retention policy to discover the topic
> > and create a subscription before messages are lost.
> >
> > But users are not easy to set a different data retention policy or
> > create a new subscription for a lazy created DLQ topic.
> >
> > ### Add a new policy to retain data when the topic has no subscriptions
> >
> > Adding a new policy for pulsar topics: a namespace or a topic level
> > policy that makes it possible to retain messages indefinitely when a
> > topic has no subscriptions.
> >
> > For the auto-created topic, the subscription can only be determined at
> 

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[Hang Chen]

+1 (binding)

Thanks,
Hang

Jia Zhai  于2022年1月26日周三 10:17写道：
>
> +1
>
>
>
> On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
> wrote:
>
> > +1 (non-binding)
> >
> > > On Jan 25, 2022, at 5:34 PM, Zike Yang 
> > wrote:
> > >
> > > Hi Pulsar Community,
> > >
> > > This is the voting thread for PIP-124. It will stay open for at least
> > > 48 hours. Pasted below for quoting convenience.
> > >
> > > Here is the issue for this PIP:
> > https://github.com/apache/pulsar/issues/13408
> > >
> > > 
> > >
> > > ## Motivation
> > >
> > > If we enable the DLQ when consuming messages. For some messages that
> > > can't be processed successfully, the messages will be moved to the
> > > DLQ, but if we do not specify the data retention for the namespace or
> > > create a subscription for the DLQ to retain the data, the data of the
> > > DLQ will be removed automatically. Therefore, we need to create the
> > > initial subscription before sending messages to the DLQ.
> > >
> > > ## Goal
> > >
> > > Users can set the initial subscription name in the DeadLetterPolicy.
> > > The consumer will create the initial subscription before sending
> > > messages to the DLQ. At this point, subsequent messages produced to
> > > the DLQ are not automatically deleted unexpectedly. If
> > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> > > subscription won't be created automatically. Otherwise, it will
> > > confuse the user. Users can explicitly create that subscription to
> > > handle this case.
> > >
> > > This PIP needs to be compatible with the previous behavior. The
> > > initial subscription name in the DeadLetterPolicy is optional. The
> > > default behavior is not to create the initial subscription which is
> > > consistent with the original behavior.
> > >
> > > ## API Changes
> > >
> > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > >
> > > ```java
> > > /**
> > > * Name of the initial subscription name of the dead letter topic.
> > > * If this field is not set, the initial subscription will not be created.
> > > */
> > > private String initSubscriptionName;
> > > ```
> > >
> > > * Add a new field to the `CommandProducer`. The broker will use this
> > > field to create the initial subscription.
> > >
> > > ```proto
> > > optional string initial_subscription_name
> > > ```
> > >
> > > * Add a new config to the Producer Configuration. This allows the
> > > consumer to specify the initial subscription when creating the
> > > deadLetterProducer.
> > >
> > > ```java
> > > /**
> > > * Use this config to automatically create an initial subscription
> > > when creating the topic.
> > > * If this field is not set, the initial subscription will not be created.
> > > *
> > > * @param initialSubscriptionName
> > > *  Name of the initial subscription of the topic.
> > > * @return the producer builder instance
> > > */
> > > ProducerBuilder initialSubscriptionName(String
> > initialSubscriptionName);
> > > ```
> > >
> > >
> > > ## Implementation
> > >
> > > When the deadLetterProducer is initialized, the consumer will set the
> > > initial subscription of the deadLetterProducer according to the
> > > DeadLetterPolicy.
> > >
> > > When the broker creates a producer(after receiving the
> > > CommandProducer), if it finds that the topic does not exist, it will
> > > not only create the topic(if the topic automatically creation is
> > > enabled) but also create the initial subscription(if the initial
> > > subscription name is set). When creating an initial subscription, the
> > > user needs to have the `canConsume` permission and the
> > > `allowAutoSubscriptionCreation` needs to be enabled.
> > >
> > > ## Reject Alternatives
> > >
> > > ### Create the initial subscription using the consumer
> > >
> > > Before the deadLetterProducer is initialized, the consumer first tries
> > > to create a deadLetterConsumer using the initial subscription name in
> > > the DeadLetterPolicy. When this subscription already exists, the
> > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > exception and create the deadLetterProducer.
> > >
> > > This is the original solution for this PIP. But this introduces too
> > > much workload for the client. In the subsequent discussion, we try to
> > > add a new command `CommandCreateSubscription` to avoid creating and
> > > closing the deadLetterConsumer, but again, this introduces workload as
> > > well as greater complexity.
> > >
> > > ### Use the retention policy to retent the data of the DLQ
> > >
> > > Before creating the deadLetterProducer,  an admin can create a
> > > retention policy for the topic or namespace. Then, consumers of the
> > > topic have the duration of the retention policy to discover the topic
> > > and create a subscription before messages are lost.
> > >
> > > But users are not easy to set a different data retention policy or
> > > create a new subscription for a lazy created DLQ topic.
> > >
> > > ### Add a new policy to

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[Michael Marshall]

+1 (non binding) - this proposal looks great! Thank you for a good
discussion of this feature!

Thanks,
Michael

On Tue, Jan 25, 2022 at 8:20 PM Hang Chen  wrote:
>
> +1 (binding)
>
> Thanks,
> Hang
>
> Jia Zhai  于2022年1月26日周三 10:17写道：
> >
> > +1
> >
> >
> >
> > On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > > On Jan 25, 2022, at 5:34 PM, Zike Yang 
> > > wrote:
> > > >
> > > > Hi Pulsar Community,
> > > >
> > > > This is the voting thread for PIP-124. It will stay open for at least
> > > > 48 hours. Pasted below for quoting convenience.
> > > >
> > > > Here is the issue for this PIP:
> > > https://github.com/apache/pulsar/issues/13408
> > > >
> > > > 
> > > >
> > > > ## Motivation
> > > >
> > > > If we enable the DLQ when consuming messages. For some messages that
> > > > can't be processed successfully, the messages will be moved to the
> > > > DLQ, but if we do not specify the data retention for the namespace or
> > > > create a subscription for the DLQ to retain the data, the data of the
> > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > initial subscription before sending messages to the DLQ.
> > > >
> > > > ## Goal
> > > >
> > > > Users can set the initial subscription name in the DeadLetterPolicy.
> > > > The consumer will create the initial subscription before sending
> > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > the DLQ are not automatically deleted unexpectedly. If
> > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> > > > subscription won't be created automatically. Otherwise, it will
> > > > confuse the user. Users can explicitly create that subscription to
> > > > handle this case.
> > > >
> > > > This PIP needs to be compatible with the previous behavior. The
> > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > default behavior is not to create the initial subscription which is
> > > > consistent with the original behavior.
> > > >
> > > > ## API Changes
> > > >
> > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > >
> > > > ```java
> > > > /**
> > > > * Name of the initial subscription name of the dead letter topic.
> > > > * If this field is not set, the initial subscription will not be 
> > > > created.
> > > > */
> > > > private String initSubscriptionName;
> > > > ```
> > > >
> > > > * Add a new field to the `CommandProducer`. The broker will use this
> > > > field to create the initial subscription.
> > > >
> > > > ```proto
> > > > optional string initial_subscription_name
> > > > ```
> > > >
> > > > * Add a new config to the Producer Configuration. This allows the
> > > > consumer to specify the initial subscription when creating the
> > > > deadLetterProducer.
> > > >
> > > > ```java
> > > > /**
> > > > * Use this config to automatically create an initial subscription
> > > > when creating the topic.
> > > > * If this field is not set, the initial subscription will not be 
> > > > created.
> > > > *
> > > > * @param initialSubscriptionName
> > > > *  Name of the initial subscription of the topic.
> > > > * @return the producer builder instance
> > > > */
> > > > ProducerBuilder initialSubscriptionName(String
> > > initialSubscriptionName);
> > > > ```
> > > >
> > > >
> > > > ## Implementation
> > > >
> > > > When the deadLetterProducer is initialized, the consumer will set the
> > > > initial subscription of the deadLetterProducer according to the
> > > > DeadLetterPolicy.
> > > >
> > > > When the broker creates a producer(after receiving the
> > > > CommandProducer), if it finds that the topic does not exist, it will
> > > > not only create the topic(if the topic automatically creation is
> > > > enabled) but also create the initial subscription(if the initial
> > > > subscription name is set). When creating an initial subscription, the
> > > > user needs to have the `canConsume` permission and the
> > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > >
> > > > ## Reject Alternatives
> > > >
> > > > ### Create the initial subscription using the consumer
> > > >
> > > > Before the deadLetterProducer is initialized, the consumer first tries
> > > > to create a deadLetterConsumer using the initial subscription name in
> > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > exception and create the deadLetterProducer.
> > > >
> > > > This is the original solution for this PIP. But this introduces too
> > > > much workload for the client. In the subsequent discussion, we try to
> > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > closing the deadLetterConsumer, but again, this introduces workload as
> > > > well as greater complexity.
> > > >
> > > > ### Use the retention policy to retent the data of the DLQ
> > > >
> > > > Before creating 

Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.

[Michael Marshall]

> Currently, we don't have some system topics that users can consume, I think
> it's a good chance to normalise the system topic naming rule.
> but "__" is more common, we need more complex naming rules to reduce
> restrictions on users. Maybe we can use some prefix like "__pulsar_sys_"
> etc.

If "__" is too common, I am fine with something more unique. Ideally,
it won't be too long, though.

> Others, some plugins like "Protocol handler" want to use system topics or
> pulsar-manager can monitor how many system topics we have.

It'd be helpful to enumerate our current use of system topics. I
imagine that we'll continue to add system topics to Pulsar,
which is why I want a comprehensive design and definition for them.

> I think we also need to support registering system topic names, because 
> currently
> we can only rely on system topic names to create system topics. (Maybe 
> changing
> the internal api to support creating system topic would also be a good
> option. But then it's harder to monitor).

Registering individual topics as system topics on the fly will not
allow us to protect against name collisions with user topics and
system topics. For this reason, we already prevent users from creating
transaction system topics here [0]. By having a generic prefix, we'll
be able to trivially protect all system topic names.

Here are some of my current thoughts on system topics:

A system topic is a topic that is completely internal to Pulsar
components. Internally, it is a normal topic. It requires elevated
permission to produce/consume when authorization is enabled, even if
topic level policies are not enabled. Generic calls like
`clearNamespaceBacklog` should not affect system topics. Deleting a
namespace or tenant should delete the system topics within it.

Let me know what you think.

Thanks,
Michael

[0] 
https://github.com/apache/pulsar/blob/35f0e13fc3385b54e88ddd8e62e44146cf3b060d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L1341-L1347


On Sun, Jan 23, 2022 at 7:23 AM mattison chao  wrote:
>
> HI, Michael Marshall
>
> > I am concerned that this design will not scale and that it will be very
> expensive on installations with many topics.
>
> I very much agree with your concern, but currently we already have some
> topic names to use ``endWith()`` and ``startWith()`` to determine.
>
> > It would also improve the speed of operations on namespaces because
> filtering out system topics would only require a quick check for
> `startsWith("__")`.
>
> Currently, we don't have some system topics that users can consume, I think
> it's a good chance to normalise the system topic naming rule.
> but "__" is more common, we need more complex naming rules to reduce
> restrictions on users. Maybe we can use some prefix like "__pulsar_sys_"
> etc.
>
> Others, some plugins like "Protocol handler" want to use system topics or
> pulsar-manager can monitor how many system topics we have. I think we also
> need to support registering system topic names, because currently we can
> only rely on system topic names to create system topics. (Maybe changing
> the internal api to support creating system topic would also be a good
> option. But then it's harder to monitor).
>
> Do you have any great ideas ?
>
> Best,
> Mattison
>
> On Sun, 23 Jan 2022 at 14:46, Michael Marshall  wrote:
>
> > This is a really important thread. I think our system topic design
> > needs more definition, structure, and documentation. Thanks for
> > starting the thread, Bo Cong.
> >
> > > If the user uses it, it means that the user has considered the result of
> > the clear system sub backlog
> >
> > I think we need to discuss authorization for operating on system
> > topics, especially when topic level policies are not enabled.
> >
> > From reading the PIP [0], it looks like we already have many different
> > ways to designate a system topic (prefixes, suffixes, and total string
> > equality). I am concerned that this design will not scale and that it
> > will be very expensive on installations with many topics.
> >
> > Is it possible for us to pivot our implementation so that system
> > topics and only system topics start with two underscores? This could
> > simplify system topic authorization. It would also improve the speed
> > of operations on namespaces because filtering out system topics would
> > only require a quick check for `startsWith("__")`.
> >
> > While it's possible that users already have topics that start with
> > "__", I think it is unlikely, and I think it is essential that we
> > design system topics so that they are highly scalable.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/issues/13794
> >
> >
> > On Thu, Jan 13, 2022 at 10:27 AM PengHui Li  wrote:
> > >
> > > I agree with the point, we should avoid the `clearNamespaceBacklog(String
> > > namespace)` to apply to the internal topic or internal cursor.
> > > It will introduce abnormal behaviors, e.g. clear a repli

[GitHub] [pulsar-helm-chart] michaeljmarshall merged pull request #210: Remove references to tag: 2.6.0 in examples

[GitBox]

michaeljmarshall merged pull request #210:
URL: https://github.com/apache/pulsar-helm-chart/pull/210


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] michaeljmarshall commented on pull request #205: Tiered Storage config

[GitBox]

michaeljmarshall commented on pull request #205:
URL: 
https://github.com/apache/pulsar-helm-chart/pull/205#issuecomment-1021885592


   @sijie - PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari closed issue #85: Change Pulsar Manager docker image version

[GitBox]

lhotari closed issue #85:
URL: https://github.com/apache/pulsar-helm-chart/issues/85


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] lhotari commented on issue #85: Change Pulsar Manager docker image version

[GitBox]

lhotari commented on issue #85:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/85#issuecomment-1021917774


The docker image version was upgraded as part of #206. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ

[PengHui Li]

+1 (binding)

Regards,
Penghui

On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall 
wrote:

> +1 (non binding) - this proposal looks great! Thank you for a good
> discussion of this feature!
>
> Thanks,
> Michael
>
> On Tue, Jan 25, 2022 at 8:20 PM Hang Chen  wrote:
> >
> > +1 (binding)
> >
> > Thanks,
> > Hang
> >
> > Jia Zhai  于2022年1月26日周三 10:17写道：
> > >
> > > +1
> > >
> > >
> > >
> > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao 
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang  .INVALID>
> > > > wrote:
> > > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > This is the voting thread for PIP-124. It will stay open for at
> least
> > > > > 48 hours. Pasted below for quoting convenience.
> > > > >
> > > > > Here is the issue for this PIP:
> > > > https://github.com/apache/pulsar/issues/13408
> > > > >
> > > > > 
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > If we enable the DLQ when consuming messages. For some messages
> that
> > > > > can't be processed successfully, the messages will be moved to the
> > > > > DLQ, but if we do not specify the data retention for the namespace
> or
> > > > > create a subscription for the DLQ to retain the data, the data of
> the
> > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > initial subscription before sending messages to the DLQ.
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Users can set the initial subscription name in the
> DeadLetterPolicy.
> > > > > The consumer will create the initial subscription before sending
> > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> initial
> > > > > subscription won't be created automatically. Otherwise, it will
> > > > > confuse the user. Users can explicitly create that subscription to
> > > > > handle this case.
> > > > >
> > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > default behavior is not to create the initial subscription which is
> > > > > consistent with the original behavior.
> > > > >
> > > > > ## API Changes
> > > > >
> > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > >
> > > > > ```java
> > > > > /**
> > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > * If this field is not set, the initial subscription will not be
> created.
> > > > > */
> > > > > private String initSubscriptionName;
> > > > > ```
> > > > >
> > > > > * Add a new field to the `CommandProducer`. The broker will use
> this
> > > > > field to create the initial subscription.
> > > > >
> > > > > ```proto
> > > > > optional string initial_subscription_name
> > > > > ```
> > > > >
> > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > consumer to specify the initial subscription when creating the
> > > > > deadLetterProducer.
> > > > >
> > > > > ```java
> > > > > /**
> > > > > * Use this config to automatically create an initial subscription
> > > > > when creating the topic.
> > > > > * If this field is not set, the initial subscription will not be
> created.
> > > > > *
> > > > > * @param initialSubscriptionName
> > > > > *  Name of the initial subscription of the topic.
> > > > > * @return the producer builder instance
> > > > > */
> > > > > ProducerBuilder initialSubscriptionName(String
> > > > initialSubscriptionName);
> > > > > ```
> > > > >
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > When the deadLetterProducer is initialized, the consumer will set
> the
> > > > > initial subscription of the deadLetterProducer according to the
> > > > > DeadLetterPolicy.
> > > > >
> > > > > When the broker creates a producer(after receiving the
> > > > > CommandProducer), if it finds that the topic does not exist, it
> will
> > > > > not only create the topic(if the topic automatically creation is
> > > > > enabled) but also create the initial subscription(if the initial
> > > > > subscription name is set). When creating an initial subscription,
> the
> > > > > user needs to have the `canConsume` permission and the
> > > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > > >
> > > > > ## Reject Alternatives
> > > > >
> > > > > ### Create the initial subscription using the consumer
> > > > >
> > > > > Before the deadLetterProducer is initialized, the consumer first
> tries
> > > > > to create a deadLetterConsumer using the initial subscription name
> in
> > > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > > exception and create the deadLetterProducer.
> > > > >
> > > > > This is the original solution for this PIP. But this introduces too
> > > > > much wo