[GitHub] [pulsar-helm-chart] chlyzzo opened a new issue #160: helm install --valtopic lose: ues examples/values-minikube.yaml --set initialize=true --namespace pulsar-st pulsar-minikube ./charts/pulsa

[GitBox]

chlyzzo opened a new issue #160:
URL: https://github.com/apache/pulsar-helm-chart/issues/160


   **Is your feature request related to a problem? Please describe.**
   first, i clone apache/pulsar-helm-chart, on k8s machine, i use the command:
   ---
   helm install --values examples/values-minikube.yaml --set initialize=true 
--namespace pulsar-st pulsar-minikube ./charts/pulsar
   -
   install a pulsar;
   then i get a pulsar, so i use the command into pulsar:
   kubectl exec -it -n pulsar-st pulsar-minikube-toolset-0 -- /bin/bash
   
   and then produce a message:
   ./bin/pulsar-client produce my-topic --messages "dsfs334"
   
   and list the topics:
   ./bin/pulsar-admin topics list public/default
   
   i can see the :
   "persistent://public/default/my-topic"
   
   **Describe the solution you'd like**
   but after a few seconds, the topics lose, and do not consume the message 
from topic,
   use the command:./bin/pulsar-admin topics list public/default
   and get nothing.
   
   **Describe alternatives you've considered**
   so, helm install pussar,the topics lose, 
   
   **Additional context**
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] javahongxi opened a new issue #417: How to build and startup pulsar-manager in IDEA?

[GitBox]

javahongxi opened a new issue #417:
URL: https://github.com/apache/pulsar-manager/issues/417


   I want to build the source code of pulsar-manager, and startup it in IDEA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] Update the default log pattern

[tom lee]

Hello dev list,

As mentioned in this issue(https://github.com/apache/pulsar/issues/8298),
there is no date in pulsar's log at present, only time. Sometimes it's not
easy to track problems.

To facilitate problem tracking, unify the pulsar's format, we plan to
update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
HH:mm:ss.SSS)

However, considering that there may be many systems that parse logs and if
we change the pattern we are going to break things. So the community must
be informed and agree on this change.

The PR is https://github.com/apache/pulsar/pull/11935.

I posted this email here to attract more reviewers.

Regards,
Tom

Re: PIP: Topic policy across multiple clusters

[PengHui Li]

>From the PulsarAdmin Java API perspective, looks like we need to add a new
param `isGlobal` for all topic policy set/get/remove methods.

Example:

setRetention(String topic, RetentionPolicies retention) ===>
setRetention(String topic, RetentionPolicies retention, boolean isGlobal);
void removeRetention(String topic) ===> void removeRetention(String topic,
boolean isGlobal);

For the get policy method, it does not make any sense to put the `applied`
and the `isGlobal` together, so looks like we need a new method
such as `getGlobalRetention(String topic)`.

But after applying the change, the Topics API looks unfriendly to users.

Another option is we can create a separate class for the topic policy such
as `pulsarAdmin.topicPolicy(boolean isGlobal)`. Since all the get applied
policy method
will not apply the `isGlobal` option, so we can create a base class
TopicPolicyBase and 2 implementations LocalTopicPolicy and
GlobalTopicPolicy.

For the usage of the Topic Policy Admin API, it should be:

```
admin.topicPolicy(true).setRetention(String topic, RetentionPolicies
retention) for setting the global retention policy for the topic,
admin.topicPolicy().setRetention(String topic, RetentionPolicies retention)
for setting the local retention policy for the topic,
admin.topicPolicy().getRetention(String topic, boolean applied) for getting
the applied policy
admin.topicPolicy().getRetention(String topic) for getting the local
retention policy
admin.topicPolicy(true).getRetention(String topic) for getting the global
retention policy
```
All the existing topic policy methods in the `admin.topics()` will not be
changed, so no compatibility issues need to be considered.
Since the `admin.topics()` have many methods which not belongs to the topic
policy, so `admin.topics(boolean isGlobal)` will confuse users.

WDYT @lin...@apache.org @Sijie Guo  @mme...@apache.org


Thanks,
Penghui

On Thu, Jul 29, 2021 at 2:37 AM linlin  wrote:

> Hi, All
>
> I have prepared a PIP for the Topic policy across multiple clusters.
> Please take a look and let me know what you think.
>
> I would really appreciate it.
>
> Best Regards,
> Lin Lin
>
>
>
> ===
>
> clusters
>
>
>-
>
>Status:
>-
>
>Authors: Penghui Li、Chen Hang、 Lin Lin
>-
>
>Pull Request:
>-
>
>Mailing List discussion:
>-
>
>Release:
>
> Motivation
>
> When setting the topic policy for a geo-replicated cluster, some policies
> want to affect the whole geo-replicated cluster but some only want to
> affect the local cluster. So the proposal is to support global topic policy
> and local topic policy.
> Approach
>
> Currently, we are using the TopicPolicies construction to store the topic
> policy for a topic. An easy way to achieve different topic policies for
> multiple clusters is to add a global flag to TopicPolicies .Replicator will
> replicate policies with a global flag to other clusters
>
> ```
>
> public class TopicPolicies {
>
>  boolean isGlobal = false
>
> }
>
> ```
>
> We only cache one local Topic Policies in the memory before. Now it will
> become two, one is Local and the other is Global.
>
> After adding the global topic policy, the topic applied priority is:
>
>
>1.
>
>Local cluster topic policy
>2.
>
>Global topic policy
>3.
>
>Namespace policy
>4.
>
>Broker default configuration
>
>
> When setting a global topic policy, we can use the `--global` option, it
> should be:
>
> ```
>
> bin/pulsar-admin topics set-retention -s 1G -t 1d --global my-topic
>
> ```
>
> If the --global option is not added, the behavior is consistent with
> before, and only updates the local policies.
>
> Topic policies are stored in System Topic, we can directly use Replicator
> to replicate data to other Clusters. We need to add a new API to the
> Replicator interface, which can set the Filter Function. Then add a
> Function to filter out the data with isGlobal = false
>
> Delete a cluster?
>
> Deleting a cluster will now delete local topic policies and will not affect
> other clusters, because only global policies will be replicated to other
> clusters
> Changes
>
>-
>
>Every topic policy API adds the `--global` option. Including broker REST
>API, Admin SDK, CMD.
>
>
>
>-
>
>Add API `public void setFilterFunction(Function)`. If
>Function returns false, then filter out the input message.
>
>
> Compatibility
>
> The solution does not introduce any compatibility issues, `isGlobal` in
> TopicPolicies is false by default in existing Policies.
>
> Test Plan
>
>1.
>
>Existing TopicPolicies will not be affected
>2.
>
>Only replicate Global Topic Policies, FilterFunction can run as expected
>3.
>
>Priority of Topic Policies matches our setting
>

Re: [VOTE] Pulsar Release 2.8.1 Candidate 3

[Massimiliano Mirelli]

пн, 6 сент. 2021 г., 19:26 Enrico Olivelli :

> Il Lun 6 Set 2021, 16:37 Massimiliano Mirelli <
> massimilianomirelli...@gmail.com> ha scritto:
>
> > Thank you for the rc.
> >
> > +1 (nonbinding)
> >
> >- Built from source code
> >- Verified checksums and signatures
> >- Run distributed system tests with Fallout.
> >
> > I had to:
> >
> >- chmod +x docker/pulsar/scripts/*
> >
> > in order to have the cluster successfully deployed, otherwise
> > apply-config-from-env.py too scrict permissions to run.
> >
>
>
> Can you please explain more?
>
> Do you mean that the docker image does not work out of the box?
>
> Enrico


I had to run the command below before building the docker image from
source.

`chmod +x docker/pulsar/scripts/*`

I did this in order for the k8s cluster to deploy smoothly. Otherwise,
broker and bookies pods wouldn't have enough permissions to execute
`docker/pulsar/scripts/apply-env.py`.

Massimiliano

>
> > Thank you,
> > Massimiliano
> >
> >
> > On Mon, 6 Sept 2021 at 16:51, PengHui Li  wrote:
> >
> > > Thanks for the great work.
> > >
> > > +1(binding)
> > >
> > > - verified the signature
> > > - start standalone, publish and consume
> > > - verified Cassandra connector
> > > - verified Function
> > > - verified stateful Function
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Sat, Sep 4, 2021 at 2:22 PM Hang Chen  wrote:
> > >
> > > > This is the third release candidate for Apache Pulsar, version 2.8.1.
> > > >
> > > > It fixes the following issues:
> > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.8.1+is%3Aclosed
> > > >
> > > > *** Please download, test and vote on this release. This vote will
> stay
> > > > open
> > > > for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided
> > for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.1-candidate-3/
> > > >
> > > > SHA-512 checksums:
> > > > 1d3ca1d21836626e5f9130400b411ce1cf258edd
> > apache-pulsar-2.8.1-src.tar.gz
> > > > a981925e4c801bb5c0a4a3b683445ab36fd04ca1
> > apache-pulsar-2.8.1-bin.tar.gz
> > > >
> > > > Maven staging repo:
> > > >
> > https://repository.apache.org/content/repositories/orgapachepulsar-1099
> > > >
> > > > The tag to be voted upon:
> > > > v2.8.1-candidate-3 (09a6b588fe7df63c6cd8acb6699cab46f8f57bec)
> > > > https://github.com/apache/pulsar/releases/tag/v2.8.1-candidate-3
> > > >
> > > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > > >
> > > > Please download the source package, and follow the README to build
> > > > and run the Pulsar standalone service.
> > > >
> > >
> >
>

Re: [DISCUSS] Update the default log pattern

[PengHui Li]

LGTM, I have added the tag `release/note-required` to make sure the change
will be highlighted in the release note.

Thanks,
Penghui

On Tue, Sep 7, 2021 at 4:41 PM tom lee  wrote:

> Hello dev list,
>
> As mentioned in this issue(https://github.com/apache/pulsar/issues/8298),
> there is no date in pulsar's log at present, only time. Sometimes it's not
> easy to track problems.
>
> To facilitate problem tracking, unify the pulsar's format, we plan to
> update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
> HH:mm:ss.SSS)
>
> However, considering that there may be many systems that parse logs and if
> we change the pattern we are going to break things. So the community must
> be informed and agree on this change.
>
> The PR is https://github.com/apache/pulsar/pull/11935.
>
> I posted this email here to attract more reviewers.
>
> Regards,
> Tom
>

Re: [DISCUSS] Update the default log pattern

[tom lee]

Thanks @Penghui for your comments.

I added timezone in log pattern. Do other guys have any questions or
concerns about this change? Looking forward to your reply and discussion.

before the change:
[image: before-change.jpg]

after the change:
[image: after-change.jpg]

Regards,
Tom

PengHui Li  于2021年9月7日周二 下午4:56写道：

> LGTM, I have added the tag `release/note-required` to make sure the change
> will be highlighted in the release note.
>
> Thanks,
> Penghui
>
> On Tue, Sep 7, 2021 at 4:41 PM tom lee  wrote:
>
> > Hello dev list,
> >
> > As mentioned in this issue(https://github.com/apache/pulsar/issues/8298
> ),
> > there is no date in pulsar's log at present, only time. Sometimes it's
> not
> > easy to track problems.
> >
> > To facilitate problem tracking, unify the pulsar's format, we plan to
> > update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
> > HH:mm:ss.SSS)
> >
> > However, considering that there may be many systems that parse logs and
> if
> > we change the pattern we are going to break things. So the community must
> > be informed and agree on this change.
> >
> > The PR is https://github.com/apache/pulsar/pull/11935.
> >
> > I posted this email here to attract more reviewers.
> >
> > Regards,
> > Tom
> >
>

Re: [DISCUSS] Update the default log pattern

[Ivan Kelly]

Hi Tom,

The images seem to be stripped by the mailing list. Perhaps post them to
the PR.

Cheers,
Ivan

On Tue, Sep 7, 2021 at 1:30 PM tom lee  wrote:

> Thanks @Penghui for your comments.
>
> I added timezone in log pattern. Do other guys have any questions or
> concerns about this change? Looking forward to your reply and discussion.
>
> before the change:
> [image: before-change.jpg]
>
> after the change:
> [image: after-change.jpg]
>
> Regards,
> Tom
>
> PengHui Li  于2021年9月7日周二 下午4:56写道：
>
>> LGTM, I have added the tag `release/note-required` to make sure the change
>> will be highlighted in the release note.
>>
>> Thanks,
>> Penghui
>>
>> On Tue, Sep 7, 2021 at 4:41 PM tom lee  wrote:
>>
>> > Hello dev list,
>> >
>> > As mentioned in this issue(https://github.com/apache/pulsar/issues/8298
>> ),
>> > there is no date in pulsar's log at present, only time. Sometimes it's
>> not
>> > easy to track problems.
>> >
>> > To facilitate problem tracking, unify the pulsar's format, we plan to
>> > update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
>> > HH:mm:ss.SSS)
>> >
>> > However, considering that there may be many systems that parse logs and
>> if
>> > we change the pattern we are going to break things. So the community
>> must
>> > be informed and agree on this change.
>> >
>> > The PR is https://github.com/apache/pulsar/pull/11935.
>> >
>> > I posted this email here to attract more reviewers.
>> >
>> > Regards,
>> > Tom
>> >
>>
>

Re: [DISCUSS] Update the default log pattern

[tom lee]

Thanks @Ivan for reminding me.

Ivan Kelly  于2021年9月7日周二 下午8:34写道：

> Hi Tom,
>
> The images seem to be stripped by the mailing list. Perhaps post them to
> the PR.
>
> Cheers,
> Ivan
>
> On Tue, Sep 7, 2021 at 1:30 PM tom lee  wrote:
>
> > Thanks @Penghui for your comments.
> >
> > I added timezone in log pattern. Do other guys have any questions or
> > concerns about this change? Looking forward to your reply and discussion.
> >
> > before the change:
> > [image: before-change.jpg]
> >
> > after the change:
> > [image: after-change.jpg]
> >
> > Regards,
> > Tom
> >
> > PengHui Li  于2021年9月7日周二 下午4:56写道：
> >
> >> LGTM, I have added the tag `release/note-required` to make sure the
> change
> >> will be highlighted in the release note.
> >>
> >> Thanks,
> >> Penghui
> >>
> >> On Tue, Sep 7, 2021 at 4:41 PM tom lee  wrote:
> >>
> >> > Hello dev list,
> >> >
> >> > As mentioned in this issue(
> https://github.com/apache/pulsar/issues/8298
> >> ),
> >> > there is no date in pulsar's log at present, only time. Sometimes it's
> >> not
> >> > easy to track problems.
> >> >
> >> > To facilitate problem tracking, unify the pulsar's format, we plan to
> >> > update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
> >> > HH:mm:ss.SSS)
> >> >
> >> > However, considering that there may be many systems that parse logs
> and
> >> if
> >> > we change the pattern we are going to break things. So the community
> >> must
> >> > be informed and agree on this change.
> >> >
> >> > The PR is https://github.com/apache/pulsar/pull/11935.
> >> >
> >> > I posted this email here to attract more reviewers.
> >> >
> >> > Regards,
> >> > Tom
> >> >
> >>
> >
>

[GitHub] [pulsar-helm-chart] frankjkelly commented on a change in pull request #159: Fix #152 Helm chart does not support Istio port naming

[GitBox]

frankjkelly commented on a change in pull request #159:
URL: https://github.com/apache/pulsar-helm-chart/pull/159#discussion_r703497200



##
File path: charts/pulsar/templates/bookkeeper-service.yaml
##
@@ -33,7 +33,7 @@ metadata:
 spec:
   ports:
   - name: bookie
-port: {{ .Values.bookkeeper.ports.bookie }}

Review comment:
   @lhotari thanks for the feedback. @addisonj or @sijie Lari raises a 
great point - is there another way around this? 
   Istio also supports setting the `appProtocol` but requires Kubernetes 1.18+
   
https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] Pulsar Release 2.8.1 Candidate 3

[Hang Chen]

Hi Massimiliano,
 Thanks for your test, we should push a PR to add +x permission for
docker/pulsar/scripts/* files, not just *.sh when build source tarball.

Thanks,
Hang

Massimiliano Mirelli  于2021年9月7日周二
下午4:54写道：

> пн, 6 сент. 2021 г., 19:26 Enrico Olivelli :
>
> > Il Lun 6 Set 2021, 16:37 Massimiliano Mirelli <
> > massimilianomirelli...@gmail.com> ha scritto:
> >
> > > Thank you for the rc.
> > >
> > > +1 (nonbinding)
> > >
> > >- Built from source code
> > >- Verified checksums and signatures
> > >- Run distributed system tests with Fallout.
> > >
> > > I had to:
> > >
> > >- chmod +x docker/pulsar/scripts/*
> > >
> > > in order to have the cluster successfully deployed, otherwise
> > > apply-config-from-env.py too scrict permissions to run.
> > >
> >
> >
> > Can you please explain more?
> >
> > Do you mean that the docker image does not work out of the box?
> >
> > Enrico
>
>
> I had to run the command below before building the docker image from
> source.
>
> `chmod +x docker/pulsar/scripts/*`
>
> I did this in order for the k8s cluster to deploy smoothly. Otherwise,
> broker and bookies pods wouldn't have enough permissions to execute
> `docker/pulsar/scripts/apply-env.py`.
>
> Massimiliano
>
> >
> > > Thank you,
> > > Massimiliano
> > >
> > >
> > > On Mon, 6 Sept 2021 at 16:51, PengHui Li  wrote:
> > >
> > > > Thanks for the great work.
> > > >
> > > > +1(binding)
> > > >
> > > > - verified the signature
> > > > - start standalone, publish and consume
> > > > - verified Cassandra connector
> > > > - verified Function
> > > > - verified stateful Function
> > > >
> > > > Thanks,
> > > > Penghui
> > > >
> > > > On Sat, Sep 4, 2021 at 2:22 PM Hang Chen 
> wrote:
> > > >
> > > > > This is the third release candidate for Apache Pulsar, version
> 2.8.1.
> > > > >
> > > > > It fixes the following issues:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.8.1+is%3Aclosed
> > > > >
> > > > > *** Please download, test and vote on this release. This vote will
> > stay
> > > > > open
> > > > > for at least 72 hours ***
> > > > >
> > > > > Note that we are voting upon the source (tag), binaries are
> provided
> > > for
> > > > > convenience.
> > > > >
> > > > > Source and binary files:
> > > > >
> > >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.1-candidate-3/
> > > > >
> > > > > SHA-512 checksums:
> > > > > 1d3ca1d21836626e5f9130400b411ce1cf258edd
> > > apache-pulsar-2.8.1-src.tar.gz
> > > > > a981925e4c801bb5c0a4a3b683445ab36fd04ca1
> > > apache-pulsar-2.8.1-bin.tar.gz
> > > > >
> > > > > Maven staging repo:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachepulsar-1099
> > > > >
> > > > > The tag to be voted upon:
> > > > > v2.8.1-candidate-3 (09a6b588fe7df63c6cd8acb6699cab46f8f57bec)
> > > > > https://github.com/apache/pulsar/releases/tag/v2.8.1-candidate-3
> > > > >
> > > > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > > > >
> > > > > Please download the source package, and follow the README to build
> > > > > and run the Pulsar standalone service.
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Pulsar Release 2.8.1 Candidate 3

[Hang Chen]

Thanks for the validation.

Close this vote with

3 +1(binding),
2 +1(non-binding)

I will start the subsequent release work soon.

Thanks,
Hang

Hang Chen  于2021年9月7日周二 下午9:14写道：

> Hi Massimiliano,
>  Thanks for your test, we should push a PR to add +x permission for
> docker/pulsar/scripts/* files, not just *.sh when build source tarball.
>
> Thanks,
> Hang
>
> Massimiliano Mirelli  于2021年9月7日周二
> 下午4:54写道：
>
>> пн, 6 сент. 2021 г., 19:26 Enrico Olivelli :
>>
>> > Il Lun 6 Set 2021, 16:37 Massimiliano Mirelli <
>> > massimilianomirelli...@gmail.com> ha scritto:
>> >
>> > > Thank you for the rc.
>> > >
>> > > +1 (nonbinding)
>> > >
>> > >- Built from source code
>> > >- Verified checksums and signatures
>> > >- Run distributed system tests with Fallout.
>> > >
>> > > I had to:
>> > >
>> > >- chmod +x docker/pulsar/scripts/*
>> > >
>> > > in order to have the cluster successfully deployed, otherwise
>> > > apply-config-from-env.py too scrict permissions to run.
>> > >
>> >
>> >
>> > Can you please explain more?
>> >
>> > Do you mean that the docker image does not work out of the box?
>> >
>> > Enrico
>>
>>
>> I had to run the command below before building the docker image from
>> source.
>>
>> `chmod +x docker/pulsar/scripts/*`
>>
>> I did this in order for the k8s cluster to deploy smoothly. Otherwise,
>> broker and bookies pods wouldn't have enough permissions to execute
>> `docker/pulsar/scripts/apply-env.py`.
>>
>> Massimiliano
>>
>> >
>> > > Thank you,
>> > > Massimiliano
>> > >
>> > >
>> > > On Mon, 6 Sept 2021 at 16:51, PengHui Li  wrote:
>> > >
>> > > > Thanks for the great work.
>> > > >
>> > > > +1(binding)
>> > > >
>> > > > - verified the signature
>> > > > - start standalone, publish and consume
>> > > > - verified Cassandra connector
>> > > > - verified Function
>> > > > - verified stateful Function
>> > > >
>> > > > Thanks,
>> > > > Penghui
>> > > >
>> > > > On Sat, Sep 4, 2021 at 2:22 PM Hang Chen 
>> wrote:
>> > > >
>> > > > > This is the third release candidate for Apache Pulsar, version
>> 2.8.1.
>> > > > >
>> > > > > It fixes the following issues:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.8.1+is%3Aclosed
>> > > > >
>> > > > > *** Please download, test and vote on this release. This vote will
>> > stay
>> > > > > open
>> > > > > for at least 72 hours ***
>> > > > >
>> > > > > Note that we are voting upon the source (tag), binaries are
>> provided
>> > > for
>> > > > > convenience.
>> > > > >
>> > > > > Source and binary files:
>> > > > >
>> > >
>> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.1-candidate-3/
>> > > > >
>> > > > > SHA-512 checksums:
>> > > > > 1d3ca1d21836626e5f9130400b411ce1cf258edd
>> > > apache-pulsar-2.8.1-src.tar.gz
>> > > > > a981925e4c801bb5c0a4a3b683445ab36fd04ca1
>> > > apache-pulsar-2.8.1-bin.tar.gz
>> > > > >
>> > > > > Maven staging repo:
>> > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachepulsar-1099
>> > > > >
>> > > > > The tag to be voted upon:
>> > > > > v2.8.1-candidate-3 (09a6b588fe7df63c6cd8acb6699cab46f8f57bec)
>> > > > > https://github.com/apache/pulsar/releases/tag/v2.8.1-candidate-3
>> > > > >
>> > > > > Pulsar's KEYS file containing PGP keys we use to sign the release:
>> > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>> > > > >
>> > > > > Please download the source package, and follow the README to build
>> > > > > and run the Pulsar standalone service.
>> > > > >
>> > > >
>> > >
>> >
>>
>

[DISCUSS] Update the default log pattern

[tom lee]

Hello dev list,

As mentioned in this issue(https://github.com/apache/pulsar/issues/8298),
there is no date in pulsar's log at present, only time. Sometimes it's not
easy to track problems.

To facilitate problem tracking, unify the pulsar's format, we plan to
update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
HH:mm:ss.SSS)

However, considering that there may be many systems that parse logs and if
we change the pattern we are going to break things. So the community must
be informed and agree on this change.

The PR is https://github.com/apache/pulsar/pull/11935.

I posted this email here to attract more reviewers.

Regards,
Tom

Re: [DISCUSS] Update the default log pattern

[Michael Marshall]

Thanks for posting this on the dev mailing list, Tom. Would you also post
it to the users ML (us...@pulsar.apache.org)?

Thanks,
Michael

On Tue, Sep 7, 2021 at 7:36 AM tom lee  wrote:

> Thanks @Ivan for reminding me.
>
> Ivan Kelly  于2021年9月7日周二 下午8:34写道：
>
> > Hi Tom,
> >
> > The images seem to be stripped by the mailing list. Perhaps post them to
> > the PR.
> >
> > Cheers,
> > Ivan
> >
> > On Tue, Sep 7, 2021 at 1:30 PM tom lee  wrote:
> >
> > > Thanks @Penghui for your comments.
> > >
> > > I added timezone in log pattern. Do other guys have any questions or
> > > concerns about this change? Looking forward to your reply and
> discussion.
> > >
> > > before the change:
> > > [image: before-change.jpg]
> > >
> > > after the change:
> > > [image: after-change.jpg]
> > >
> > > Regards,
> > > Tom
> > >
> > > PengHui Li  于2021年9月7日周二 下午4:56写道：
> > >
> > >> LGTM, I have added the tag `release/note-required` to make sure the
> > change
> > >> will be highlighted in the release note.
> > >>
> > >> Thanks,
> > >> Penghui
> > >>
> > >> On Tue, Sep 7, 2021 at 4:41 PM tom lee  wrote:
> > >>
> > >> > Hello dev list,
> > >> >
> > >> > As mentioned in this issue(
> > https://github.com/apache/pulsar/issues/8298
> > >> ),
> > >> > there is no date in pulsar's log at present, only time. Sometimes
> it's
> > >> not
> > >> > easy to track problems.
> > >> >
> > >> > To facilitate problem tracking, unify the pulsar's format, we plan
> to
> > >> > update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
> > >> > HH:mm:ss.SSS)
> > >> >
> > >> > However, considering that there may be many systems that parse logs
> > and
> > >> if
> > >> > we change the pattern we are going to break things. So the community
> > >> must
> > >> > be informed and agree on this change.
> > >> >
> > >> > The PR is https://github.com/apache/pulsar/pull/11935.
> > >> >
> > >> > I posted this email here to attract more reviewers.
> > >> >
> > >> > Regards,
> > >> > Tom
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Update the default log pattern

[tom lee]

Thanks @Michael for your comments and suggestions. I just sent it to the
users ML.

Michael Marshall  于2021年9月8日周三 上午5:15写道：

> Thanks for posting this on the dev mailing list, Tom. Would you also post
> it to the users ML (us...@pulsar.apache.org)?
>
> Thanks,
> Michael
>
> On Tue, Sep 7, 2021 at 7:36 AM tom lee  wrote:
>
> > Thanks @Ivan for reminding me.
> >
> > Ivan Kelly  于2021年9月7日周二 下午8:34写道：
> >
> > > Hi Tom,
> > >
> > > The images seem to be stripped by the mailing list. Perhaps post them
> to
> > > the PR.
> > >
> > > Cheers,
> > > Ivan
> > >
> > > On Tue, Sep 7, 2021 at 1:30 PM tom lee  wrote:
> > >
> > > > Thanks @Penghui for your comments.
> > > >
> > > > I added timezone in log pattern. Do other guys have any questions or
> > > > concerns about this change? Looking forward to your reply and
> > discussion.
> > > >
> > > > before the change:
> > > > [image: before-change.jpg]
> > > >
> > > > after the change:
> > > > [image: after-change.jpg]
> > > >
> > > > Regards,
> > > > Tom
> > > >
> > > > PengHui Li  于2021年9月7日周二 下午4:56写道：
> > > >
> > > >> LGTM, I have added the tag `release/note-required` to make sure the
> > > change
> > > >> will be highlighted in the release note.
> > > >>
> > > >> Thanks,
> > > >> Penghui
> > > >>
> > > >> On Tue, Sep 7, 2021 at 4:41 PM tom lee 
> wrote:
> > > >>
> > > >> > Hello dev list,
> > > >> >
> > > >> > As mentioned in this issue(
> > > https://github.com/apache/pulsar/issues/8298
> > > >> ),
> > > >> > there is no date in pulsar's log at present, only time. Sometimes
> > it's
> > > >> not
> > > >> > easy to track problems.
> > > >> >
> > > >> > To facilitate problem tracking, unify the pulsar's format, we plan
> > to
> > > >> > update pulsar's default log pattern. (HH:mm:ss.SSS -> -MM-dd
> > > >> > HH:mm:ss.SSS)
> > > >> >
> > > >> > However, considering that there may be many systems that parse
> logs
> > > and
> > > >> if
> > > >> > we change the pattern we are going to break things. So the
> community
> > > >> must
> > > >> > be informed and agree on this change.
> > > >> >
> > > >> > The PR is https://github.com/apache/pulsar/pull/11935.
> > > >> >
> > > >> > I posted this email here to attract more reviewers.
> > > >> >
> > > >> > Regards,
> > > >> > Tom
> > > >> >
> > > >>
> > > >
> > >
> >
>

[PIP 94] Message converter at broker level

[Yunze Xu]

Hi, folks

I’ve created PIP 94, see https://github.com/apache/pulsar/issues/11962 
 for details.
PTAL and give your suggestions if you have any concern.

Thanks,
Yunze

[Proposal] Make Dispatcher pluggable

[linlin]

# Motivation

There are many users who need to use tag messages. The implementation of
this part has also been discussed before：

https://lists.apache.org/list.html?*@pulsar.apache.org:lte=2y:Proposal%20for%20Consumer%20Filtering%20in%20Pulsar%20brokers

I suggest to plug-in the Dispatcher, so that users can not only develop tag
messages, but also customize their own delayed messages and other features.

Plug-in Dispatcher has no compatibility impact on existing Brokers.

# Modifications

   1.

   Add a configuration item `dispatcherProviderClassName`，the creation of
   an existing Dispatcher will be moved to DefaultDispatcherProvider, as the
   default class.
   2.

   Add a new Interface DispatcherProvider


public interface DispatcherProvider {



// Use `DispatcherProvider` to create `Dispatcher`

Dispatcher createDispatcher(Consumer consumer, Subscription
subscription);

static DispatcherProvider createDispatcherProvider(ServiceConfiguration
serviceConfiguration) {

   // According to `dispatcherProviderClassName`, create Provider through
reflection

}

}

We can get BrokerService from Subscription, and then Pulsar from
BrokerService, so the parameters of this interface must be sufficient.

DispatcherProvider will be created when the subscription is initialized:

```

DispatcherProvider.createDispatcherProvider(config);

```

When there is a Consumer subscription, we will create a Dispatcher:

dispatcher = dispatcherProvider.createDispatcher(consumer, this);

I opened a new PR：

https://github.com/apache/pulsar/pull/11948

Re: [Proposal] Make Dispatcher pluggable

[Enrico Olivelli]

Lin,

Il giorno mer 8 set 2021 alle ore 07:34 linlin  ha
scritto:

> # Motivation
>
> There are many users who need to use tag messages. The implementation of
> this part has also been discussed before：
>
>
> https://lists.apache.org/list.html?*@pulsar.apache.org:lte=2y:Proposal%20for%20Consumer%20Filtering%20in%20Pulsar%20brokers
>
> I suggest to plug-in the Dispatcher, so that users can not only develop tag
> messages, but also customize their own delayed messages and other features.
>

I also share this problem, because if you want to efficiently implement
message filtering you need to do it in the broker side.

I am not sure that making the full Dispatcher pluggable is a good idea,
because the code is too complex and also
it really depends on the internals of the Broker.

If we make this pluggable that we must define a limited private but
"stable" API.

My suggestion is to define particular needs and then add features to make
pluggable single specific parts
of the dispatcher.

For instance I would add some support for "Message filtering", leaving the
implementation of the "filter" to a plugin.
This way you could implement filtering using JMS rules, or using other
metadata or security related information

Regards

Enrico


>
> Plug-in Dispatcher has no compatibility impact on existing Brokers.
>
> # Modifications
>
>1.
>
>Add a configuration item `dispatcherProviderClassName`，the creation of
>an existing Dispatcher will be moved to DefaultDispatcherProvider, as
> the
>default class.
>2.
>
>Add a new Interface DispatcherProvider
>
>
> public interface DispatcherProvider {
>
>
>
> // Use `DispatcherProvider` to create `Dispatcher`
>
> Dispatcher createDispatcher(Consumer consumer, Subscription
> subscription);
>
> static DispatcherProvider createDispatcherProvider(ServiceConfiguration
> serviceConfiguration) {
>
>// According to `dispatcherProviderClassName`, create Provider through
> reflection
>
> }
>
> }
>
> We can get BrokerService from Subscription, and then Pulsar from
> BrokerService, so the parameters of this interface must be sufficient.
>
> DispatcherProvider will be created when the subscription is initialized:
>
> ```
>
> DispatcherProvider.createDispatcherProvider(config);
>
> ```
>
> When there is a Consumer subscription, we will create a Dispatcher:
>
> dispatcher = dispatcherProvider.createDispatcher(consumer, this);
>
> I opened a new PR：
>
> https://github.com/apache/pulsar/pull/11948
>

Re: PIP-93 Pulsar Proxy Protocol Handlers

[Enrico Olivelli]

(ping)


Il giorno ven 3 set 2021 alle ore 14:06 Enrico Olivelli 
ha scritto:

> Sijie,
> Thanks for your questions, answers inline below.
>
> Il giorno gio 2 set 2021 alle ore 02:23 Sijie Guo  ha
> scritto:
>
>> I would like to see the clarification between the broker protocol handlers
>> and proxy protocol handlers before moving it to a vote thread.
>>
>
> A PH in the broker is very useful as it allows you to directly access the
> ManagedLedger and implement high performance adapters for
> other wire protocols.
> The bigger limitation is that you can access efficiently only the topics
> owned by the local broker.
> If you try to forward/proxy the request to another broker (you can do it,
> and this was Matteo's suggestion at the latest Video Community meeting)
> you have the downside that the broker has to waste resources to do the
> "proxy work"
> and you generally want a broker machine to be used only to deal with the
> local traffic.
>
> The load balancing mechanism of the brokers is not meant to deal with
> additional work due to proxying requests related to the topics for which
> the broker is not owner.
>
> A PH in the proxy is useful to add new protocols that are running in front
> of the whole cluster and not only of one single broker.
> This is a very different use case in respect to having the PH in broker.
>
> The work of the proxy usually is to forward requests to the internal
> services of the cluster, and in case of new protocols in the proxy
> you need some logic to fill in the gaps in the original wireprotocol.
>
> System architects expect a different kind of load on the proxy and other
> kinds of load on the brokers.
> For instance you usually can run very few proxies to cover a big cluster
> with many brokers.
> So adding a PH on all the brokers is sometimes overkilling.
>
>
>>
>> I can see how it will cause confusion for protocol developers.
>>
>
> Protocol developers are very advanced users that do need to understand
> clearly the internals of Pulsar.
> In fact this request of having PHs in the Proxy layer came from myself and
> from other colleagues of mine who are working heavily in implementing
> new protocol handlers in Pulsar.
>
> And we faced the limitation of the need to create a new proxy service for
> each new protocol, but all of these "proxy services" have in common
> most of the features of the Pulsar proxy.
> When we also came to deal with System Architects it was clear the
> requirement to have only one single "place" to put all of the interactions
> at "cluster level" with Pulsar.
>
> I think this is a good picture of what I mean:
> - PH in the Broker -> add protocols inside the Broker, work for owned
> topics
> - PH in the Proxy -> add protocols in front of the whole Cluster
>
>
>> Yunze brought a good idea on KoP.
>
>
> I also have good ideas and working solutions for a Pulsar-proxy like KOP
> Proxy.
> I will be happy to discuss this in a separate thread or at a separate
> table with Yunze.
>
> A smart KOP proxy can work if you run inside the Pulsar proxy process or
> you can copy/paste the Pulsar Proxy code and create another service.
>
>
>> But I don't think that's the right
>> direction. If you can give an example of the usage of a proxy handler and
>> how it is different from using a broker handler, that would help me
>> understand this PIP.
>>
>
> For some protocols you have to execute some non trivial work for mapping
> the wireprotocol and the concepts of the protocol to the Pulsar model.
> For instance some protocols do not have the concept of "lookup", and the
> proxy does the lookup and forwards the request to the internal broker.
>
> For some protocols you can just use the PulsarClient to connect to the
> internal brokers, you do not need and you do not want to access the
> ManagedLedgers:
> in this case adding the execution inside the broker is only complicating
> the overall design of the system and putting load on the brokers.
>
> There is a good amount of processing that should be executed on the proxy,
> and it is not good to run it on a broker.
> If you do not put the "custom code" in the Proxy and you can only write a
> Broker PH you end up in adding it to the Broker.
>
> If you expose directly (with some LoadBalancer or whatever) your brokers
> in which you run the PH code that you would put in the proxy
> you end up in putting on the broker some load that is not expected:
> - the broker will have to work even for topics for which it is not the
> owner
> - the broker will have to do things that cannot be dealt correctly by the
> Pulsar load balancer (because it expects that the load it proportional to
> the owned bundles)
>
>
>>
>> The reason why Pulsar proxy is built is to have a "smart" proxy that is
>> aware of Pulsar protocol. The Pulsar proxy can be replaced with other
>> mature proxy software with SNI routing or multiple advertised listeners
>> now. Hence I am afraid that we are taking the wrong direction here. Here
>> are various reasons.
>>