Re: [DISCUSS] Close the connection quickly after authorization failure

[Haiting Jiang]

Hi Zixuan,

Just want to make sure if we disconnect the connection, what's the
behavior of the client?
will it throw exceptions and fail, or do retries?
Is the client possibly doing too many retries and crashing the server?

Thanks,
Haiting

On Thu, Nov 24, 2022 at 3:41 PM Zixuan Liu  wrote:
>
> Hi all,
>
> I would like to discuss whether to disconnect the broker and the client
> after authorization failure.
>
> Right now the Pulsar doesn't disconnect the client on the broker side after
> authorization failure, then this client can still do other operations.
>
> For an unauthorized user, we should disconnect this connection to avoid
> hogging resources.
>
> Please let me know what you think.
>
> PR: https://github.com/apache/pulsar/pull/18578
>
> Thanks,
> Zixuan

Re: [VOTE] PIP-222: Add CommandPartitionedTopicMetadata metrics

[Haiting Jiang]

+1 binding

Thanks,
Haiting

On Thu, Nov 24, 2022 at 1:05 AM Jiuming Tao
 wrote:
>
>
> Bump
>
>
> > 2022年11月22日 下午4:03，Ran Gao  写道：
> >
> > +1 (unbinding)
> >
> > Thanks,
> > Ran
> >
> > On 2022/11/17 09:06:50 Jiuming Tao wrote:
> >> Dear Pulsar Community,
> >>
> >> Please review and vote on this PIP.
> >>
> >> PIP link: https://github.com/apache/pulsar/issues/18319 
> >> 
> >>
> >> Discuss thread: 
> >> https://lists.apache.org/thread/sybl4nno4503w42hzt7b5lsyk6m2rbo6 
> >> 
> >>
> >> Thanks,
> >> Tao Jiuming
>

[VOTE] Pulsar Client C++ Release 3.1.0 Candidate 1

[Zike Yang]

This is the first release candidate for Apache Pulsar Client C++, version 3.1.0.

It fixes the following issues:
https://github.com/apache/pulsar-client-cpp/milestone/2?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-cpp/pulsar-client-cpp-3.1.0-candidate-1/

SHA-512 checksums:
617035c5a419fd058262f7e66f42cb8b4da711e2e191dc6d4428110804f7528c35d96b86f668e56072c2ef88f24a0ad078d578320e78ff63cd1209f44a191aa1
 ./apache-pulsar-client-cpp-3.1.0.tar.gz


The tag to be voted upon:
v3.1.0-candidate-1 (3b5aff7f7630295de6749bacfff53c2ebc63449c)
https://github.com/apache/pulsar-client-cpp/releases/tag/v3.1.0-candidate-1

Pulsar's KEYS file containing PGP keys you use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to compile and test.

Thanks
Zike Yang

Re: [DISCUSS] PIP-223: Add metrics for all Rest Endpoints

[Haiting Jiang]

Hi Jiuming,

> I’m not sure that if observe failed requests latency is meaningful.

1. Normally for failed requests, the latency sometimes will help in debugging.
2. Add httpcode label to latency, we can analyze the latency between
different successful results too, like 200 and 307.

As for `pulsar_broker_rest_endpoint_failed`, after a second thought. I
think it would be useful for operators to set up system alarms. So +1
on this.

Thanks,
Haiting

On Thu, Nov 24, 2022 at 12:52 AM Jiuming Tao
 wrote:
>
> Hi Haiting,
>
> I’m not sure that if observe failed requests latency is meaningful, so that I 
> was added `pulsar_broker_rest_endpoint_failed` to record these requests which 
> is failed.
>
> Thanks,
> Tao Jiuming
>
> > 2022年11月23日 下午12:55，Haiting Jiang  写道：
> >
> > Hi Jiuming,
> >
> > I am not sure why we need a new metric for failed requests.
> > Can we just put the `httpcode` tag also in the
> > `pulsar_broker_rest_endpoint_latency_ms`?
> > So that user can also see the latency of failed requests (like timeout
> > requests).
> > And it can easily cover the case of failed metrics.
> >
> > Thanks,
> > Haiting
> >
> > On Wed, Nov 23, 2022 at 12:00 PM Jiuming Tao
> >  wrote:
> >>
> >> Hi Penghui,
> >>
> >>> We'd better add an option to enable or disable the endpoint-level metrics.
> >>> And keep it as disabled as default.
> >>
> >> Agreed, I also considered if we need a configuration to enable/disable the 
> >> feature, seems add an option is better
> >>
> >>> So that we can
> >>> get the request path from the handle method
> >>> `public void handle(String path, Request baseRequest, HttpServletRequest
> >>> request, HttpServletResponse response)`
> >>
> >> I’ve tried this, but the `path` is the request path like 
> >> `/user/111/query`, and we cannot get the path configured by @Path by using 
> >> `Handler`.
> >> Jetty is a Servlet container, and we are using Jersey to provide Rest API 
> >> services, I think use Jersey’s mechanism is better
> >>
> >> Thanks,
> >> Tao Jiuming
> >>
> >>
> >>
> >>> 2022年11月23日 上午9:37，PengHui Li  写道：
> >>>
> >>> Hi, Jiuming
> >>>
> >>> Thanks for starting the proposal.
> >>>
> >>> We'd better add an option to enable or disable the endpoint-level metrics.
> >>> And keep it as disabled as default.
> >>>
> >>> I noticed the existing jetty metrics are based on
> >>> `org.eclipse.jetty.server.handler.StatisticsHandler`.
> >>> Can we just have a new StatisticsHandler? e.g. EndpointStatisticsHandler.
> >>> So that we can
> >>> get the request path from the handle method
> >>> `public void handle(String path, Request baseRequest, HttpServletRequest
> >>> request, HttpServletResponse response)`
> >>>
> >>> Thanks,
> >>> Penghui
> >>>
> >>> On Tue, Nov 22, 2022 at 5:20 PM Jiuming Tao 
> >>> 
> >>> wrote:
> >>>
>  Hi Haiting,
> 
>  I'm sorry I didn't explain FAILED, the FAILED means the HTTP response 
>  code
> > = 400, and I’ll update the PIP later.
> 
>  Thanks,
>  Tao Jiuming
> 
> > 2022年11月22日 下午5:15，Haiting Jiang  写道：
> >
> > Hi Jiuming,
> >
> > Overall, this PIP makes sense to me.
> > About the metric "pulsar_broker_rest_endpoint_failed", please provide
> > a more clear definition of "fail". Are redirects like 403 included?
> >
> > Thanks,
> > Haiting
> >
> > On Tue, Nov 22, 2022 at 3:17 AM Jiuming Tao
> >  wrote:
> >>
> >> Hi pulsar community,
> >>
> >> I’ve opened a PIP to discuss: PIP-223: Add metrics for all Rest
>  Endpoints
> >>
> >> The PIP link: https://github.com/apache/pulsar/issues/18560 <
>  https://github.com/apache/pulsar/issues/18560>
> >>
> >> Thanks,
> >> Tao Jiuming
> 
> 
> >>
>

Re: [DISCUSS] Close the connection quickly after authorization failure

[Zixuan Liu]

Hi Haiting,

>Just want to make sure if we disconnect the connection, what's the
behavior of the client?

The client reconnects with the broker multiple times. When the maximum
number of reconnections is reached, the client shutdown completely.

Thanks,
Zixuan

Haiting Jiang  于2022年11月24日周四 16:04写道：

> Hi Zixuan,
>
> Just want to make sure if we disconnect the connection, what's the
> behavior of the client?
> will it throw exceptions and fail, or do retries?
> Is the client possibly doing too many retries and crashing the server?
>
> Thanks,
> Haiting
>
> On Thu, Nov 24, 2022 at 3:41 PM Zixuan Liu  wrote:
> >
> > Hi all,
> >
> > I would like to discuss whether to disconnect the broker and the client
> > after authorization failure.
> >
> > Right now the Pulsar doesn't disconnect the client on the broker side
> after
> > authorization failure, then this client can still do other operations.
> >
> > For an unauthorized user, we should disconnect this connection to avoid
> > hogging resources.
> >
> > Please let me know what you think.
> >
> > PR: https://github.com/apache/pulsar/pull/18578
> >
> > Thanks,
> > Zixuan
>

[Result][Vote] PIP-222: Add CommandPartitionedTopicMetadata metrics

[Jiuming Tao]

Thanks for vote this PIP, closing the vote

3 bindings:

Enrico Olivelli
Haiting Jiang
Penghui Li

1 non-bindings:

Ran Gao

Thanks,
Tao Jiuming

Re: [VOTE] Pulsar Release 2.11.0 Candidate 1

[guo jiwei]

Hi
  We found a case that may break the old users. could we discuss if we need
to resolve that and then continue to validate this release?

  https://github.com/apache/pulsar/issues/18598


Regards
Tboy


On Thu, Nov 24, 2022 at 3:56 PM Haiting Jiang 
wrote:

> +1 binding
>
> - Checksum and signatures
> - Built from sources using `mvn clean install -DskipTests` using JDK
> 17 and maven 3.6.1
> - Run Pulsar standalone and produce/consume case
> - Validate Pub/Sub and Java Functions
> - Validate Connectors
> - Validate Stateful Functions
> - Run a simple performance check
>
>
> BTW, there seems to be some significant performance improvement with
> 2.11.0, compared with 2.10.2. Great work!
> The command used for non-batching small payloads is `bin/pulsar-perf
> produce -b 0 -bm 0 -r 100 -s 16 -o 500 --warmup-time 30 -time 120
> persistent://public/default/my-topic`
> - Result for 2.11.0-RC1: throughput 140305.091 msg/s, Latency: mean
> 3.374 ms - med 3.221
> - Result for 2.10.2: throughput 62072.785 msg/s, Latency: mean: 7.458
> ms - med:6.566
> Note: the perf result is only meaningful when compared with other
> versions. It's run on a local laptop with pulsar standalone.
>
> Thanks,
> Haiting
>
> On Thu, Nov 24, 2022 at 1:00 PM guo jiwei  wrote:
> >
> > Hi  Dave
> >All of the artifacts are signed by me.
> >
> >
> >
> > Regards
> > Tboy
> >
> >
> > On Thu, Nov 24, 2022 at 11:10 AM Dave Fisher 
> wrote:
> >
> > > I haven’t checked and it’s ok to have more than one release manager
> for a
> > > release. But are all of the artifacts signed by one individual or both
> TBoy
> > > and Nicolo?
> > >
> > > The rule is who builds it signs it.
> > >
> > > Best,
> > > Dave
> > >
> > > Sent from my iPhone
> > >
> > > > On Nov 23, 2022, at 6:41 PM, guo jiwei  wrote:
> > > >
> > > > Hi Nicolo
> > > >
> > > >Added to the repo.
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > > >
> > > >
> > > >> On Thu, Nov 24, 2022 at 1:03 AM Nicolò Boschi  >
> > > wrote:
> > > >>
> > > >> Hi, thanks for driving the release
> > > >>
> > > >> I found out that I forgot to add Pulsar Shell (PIP-181) artifacts
> to the
> > > >> staging script, so I created these two pull requests:
> > > >> - https://github.com/apache/pulsar/pull/18583
> > > >> - https://github.com/apache/pulsar-site/pull/302
> > > >>
> > > >> Since you built all the repo locally, I believe you can just push
> the
> > > >> additional artifacts without invalidating the rc.
> > > >>
> > > >> Thanks,
> > > >> Nicolò Boschi
> > > >>
> > > >>
> > > >> Il giorno mer 23 nov 2022 alle ore 10:51 guo jiwei <
> > > techno...@apache.org>
> > > >> ha scritto:
> > > >>
> > > >>> Hi Haiting
> > > >>>
> > >  Is this expected or a bug?
> > > >>>  Yes, it's expected. we can ignore this step. I will correct the
> doc
> > > >>> later.
> > > >>>
> > >  I think you mean RocksDB, it's the default for standalone.
> > > >>> Thanks for correcting it.
> > > >>>
> > > >>>
> > > >>> Regards
> > > >>> Jiwei Guo (Tboy)
> > > >>>
> > > >>>
> > > >>> On Wed, Nov 23, 2022 at 3:29 PM Enrico Olivelli <
> eolive...@gmail.com>
> > > >>> wrote:
> > > >>>
> > >  Il giorno mer 23 nov 2022 alle ore 05:43 Haiting Jiang
> > >   ha scritto:
> > > >
> > > > Hi Jiwei,
> > > >
> > > > Thank you for cutting this RC.
> > > >
> > > > When following the processes in "Validate Release Candidate",
> > > > command `nc -vz4 localhost 4181` fails. which means table
> service is
> > > > not started in standalone mode.
> > > > Is this expected or a bug?
> > > 
> > >  This is the new behaviour of Pulsar Standalone, we disable useless
> > >  stuff by default
> > > 
> > >  Enrico
> > > 
> > > >
> > > >> Since the metadata store is changed from ZK to etcd,
> > > >
> > > > I think you mean RocksDB, it's the default for standalone.
> > > >
> > > >
> > > > Thanks,
> > > > Haiting
> > > >
> > > > On Mon, Nov 21, 2022 at 11:12 PM guo jiwei  >
> > > >>> wrote:
> > > >>
> > > >> This is the third release candidate for Apache Pulsar, version
> > > >>> 2.11.0.
> > > >>
> > > >> This release contains 1563 commits by 64 contributors.
> > > >>
> > > >>>
> https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-1
> > > >>
> > > >> CI for this release candidate
> > > >> https://github.com/Technoboy-/pulsar/pull/14
> > > >>
> > > >> *** Please download, test and vote on this release. This vote
> will
> > >  stay open
> > > >> for at least 72 hours ***
> > > >>
> > > >> Note that we are voting upon the source (tag), binaries are
> > > >> provided
> > >  for
> > > >> convenience.
> > > >>
> > > >> Source and binary files:
> > > >>
> > > 
> > > >>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-1
> > > >>
> > > >> SHA-512 checksums:
> > > >>
> > > >>
> > > 
> > > >>>
> > > >>
> > >
> 90afcb23

Re: [DISCUSS] The use of consumer redeliverUnacknowledgedMessages method

[丛搏]

Hi, Baodi

> After, when the user calls `redeliverUnacknowledgedMessages` under failover 
> and exclusive subType,
> It gets a warning reminder that the messages may be out of order, right?
>
yes, in failover and exclusive subType, we should suggest user to use
a new API to redeliver the messages and reminder that the messages may
be out of order

Re: [DISCUSS] PIP-223: Add metrics for all Rest Endpoints

[Jiuming Tao]

Hi Haiting,

I’ll update the PIP as we discussed:

1. `pulsar_broker_rest_endpoint_failed`
Labels:
   a. path: The HTTP request path
   b. method: The HTTP request method
   c. code: The HTTP response code
This metric only records HTTP requests which response code >= 400

2. `pulsar_broker_rest_endpoint_latency`
Labels:
   a. path: The HTTP request path
   b. method: The HTTP request method
   c. code: The HTTP response code
This metric records all the HTTP requests, including failed requests.


Thanks,
Tao Jiuming

Haiting Jiang  于2022年11月24日周四 16:35写道：

> Hi Jiuming,
>
> > I’m not sure that if observe failed requests latency is meaningful.
>
> 1. Normally for failed requests, the latency sometimes will help in
> debugging.
> 2. Add httpcode label to latency, we can analyze the latency between
> different successful results too, like 200 and 307.
>
> As for `pulsar_broker_rest_endpoint_failed`, after a second thought. I
> think it would be useful for operators to set up system alarms. So +1
> on this.
>
> Thanks,
> Haiting
>
> On Thu, Nov 24, 2022 at 12:52 AM Jiuming Tao
>  wrote:
> >
> > Hi Haiting,
> >
> > I’m not sure that if observe failed requests latency is meaningful, so
> that I was added `pulsar_broker_rest_endpoint_failed` to record these
> requests which is failed.
> >
> > Thanks,
> > Tao Jiuming
> >
> > > 2022年11月23日 下午12:55，Haiting Jiang  写道：
> > >
> > > Hi Jiuming,
> > >
> > > I am not sure why we need a new metric for failed requests.
> > > Can we just put the `httpcode` tag also in the
> > > `pulsar_broker_rest_endpoint_latency_ms`?
> > > So that user can also see the latency of failed requests (like timeout
> > > requests).
> > > And it can easily cover the case of failed metrics.
> > >
> > > Thanks,
> > > Haiting
> > >
> > > On Wed, Nov 23, 2022 at 12:00 PM Jiuming Tao
> > >  wrote:
> > >>
> > >> Hi Penghui,
> > >>
> > >>> We'd better add an option to enable or disable the endpoint-level
> metrics.
> > >>> And keep it as disabled as default.
> > >>
> > >> Agreed, I also considered if we need a configuration to
> enable/disable the feature, seems add an option is better
> > >>
> > >>> So that we can
> > >>> get the request path from the handle method
> > >>> `public void handle(String path, Request baseRequest,
> HttpServletRequest
> > >>> request, HttpServletResponse response)`
> > >>
> > >> I’ve tried this, but the `path` is the request path like
> `/user/111/query`, and we cannot get the path configured by @Path by using
> `Handler`.
> > >> Jetty is a Servlet container, and we are using Jersey to provide Rest
> API services, I think use Jersey’s mechanism is better
> > >>
> > >> Thanks,
> > >> Tao Jiuming
> > >>
> > >>
> > >>
> > >>> 2022年11月23日 上午9:37，PengHui Li  写道：
> > >>>
> > >>> Hi, Jiuming
> > >>>
> > >>> Thanks for starting the proposal.
> > >>>
> > >>> We'd better add an option to enable or disable the endpoint-level
> metrics.
> > >>> And keep it as disabled as default.
> > >>>
> > >>> I noticed the existing jetty metrics are based on
> > >>> `org.eclipse.jetty.server.handler.StatisticsHandler`.
> > >>> Can we just have a new StatisticsHandler? e.g.
> EndpointStatisticsHandler.
> > >>> So that we can
> > >>> get the request path from the handle method
> > >>> `public void handle(String path, Request baseRequest,
> HttpServletRequest
> > >>> request, HttpServletResponse response)`
> > >>>
> > >>> Thanks,
> > >>> Penghui
> > >>>
> > >>> On Tue, Nov 22, 2022 at 5:20 PM Jiuming Tao
> 
> > >>> wrote:
> > >>>
> >  Hi Haiting,
> > 
> >  I'm sorry I didn't explain FAILED, the FAILED means the HTTP
> response code
> > > = 400, and I’ll update the PIP later.
> > 
> >  Thanks,
> >  Tao Jiuming
> > 
> > > 2022年11月22日 下午5:15，Haiting Jiang  写道：
> > >
> > > Hi Jiuming,
> > >
> > > Overall, this PIP makes sense to me.
> > > About the metric "pulsar_broker_rest_endpoint_failed", please
> provide
> > > a more clear definition of "fail". Are redirects like 403 included?
> > >
> > > Thanks,
> > > Haiting
> > >
> > > On Tue, Nov 22, 2022 at 3:17 AM Jiuming Tao
> > >  wrote:
> > >>
> > >> Hi pulsar community,
> > >>
> > >> I’ve opened a PIP to discuss: PIP-223: Add metrics for all Rest
> >  Endpoints
> > >>
> > >> The PIP link: https://github.com/apache/pulsar/issues/18560 <
> >  https://github.com/apache/pulsar/issues/18560>
> > >>
> > >> Thanks,
> > >> Tao Jiuming
> > 
> > 
> > >>
> >
>

Re: [DISCUSS] The use of consumer redeliverUnacknowledgedMessages method

[丛搏]

Hi, Joe:

> This "brokenness" is not clear to me.
https://github.com/apache/pulsar/pull/10478 This PIP solves some
problems of "brokenness",
>The sequence 3,4,5,6,7,8,9,10,11 12,13,14,15, 16 ,9,10,11,12,13,14,15,16,17, 
>18, 19, 20 ...does not break
> the ordering guarantees of Pulsar
If don't use transaction ack, this order is fine. but when we use
transaction ack, in this case, message 9 and message 10 will be
handled twice. Therefore, we need redeliver and receive to be
synchronized to ensure that messages received before redeliver will
not be repeated and ordered, and will not be repeatedly consumed after
redeliver. To achieve these goals, we need to redeliver to be a
synchronous method instead of async and need to retry automatically.

Happy Thanksgiving!

[Yu]

Hi team,

Today is the Thanksgiving Day. At this wonderful time of the year, I have
much to be immensely thankful for.

Everyone here has made this a truly global community and a terrific place
to collaborate.

Wish Pulsar continues to grow faster and stronger next year with our joint
efforts.

Happy Thanksgiving and have a great holiday season, wherever you are
located!

Yu

PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

[Yunze Xu]

Hi Pulsar community,

I've opened a PIP to discuss: https://github.com/apache/pulsar/issues/18616.

PTAL.

Thanks,
Yunze

Re: [Vote] PIP-215: Configurable TopicCompactionStrategy for StrategicTwoPhaseCompactor and TableView

[guo jiwei]

+1 (binding)


Regards
Jiwei Guo (Tboy)


On Tue, Nov 22, 2022 at 8:57 AM Baodi Shi 
wrote:

> +1 (non-binding)
>
>
> Thanks,
> Baodi Shi
>
> > 2022年11月18日 17:00，Enrico Olivelli  写道：
> >
> > +1 (binding)
> >
> > Enrico
> >
> > Il giorno mar 15 nov 2022 alle ore 02:41 PengHui Li
> >  ha scritto:
> >>
> >> +1
> >>
> >> Thanks,
> >> Penghui
> >>
> >> On Wed, Nov 9, 2022 at 12:52 AM Heesung Sohn
> >>  wrote:
> >>
> >>> Dear Pulsar Community,
> >>>
> >>> Please review and vote on this PIP.
> >>>
> >>> PIP link: https://github.com/apache/pulsar/issues/18099
> >>>
> >>> Thank you,
> >>> -Heesung
> >>>
>
>

[DISCUSS] Enable branch-2.9 force push and frozen branch-2.9

[丛搏]

Hi, pulsar community,

I am sorry for that, there is no commit to execute cherry-pick
--continue, resulting in many cherry-pick commit messages not being
correct[1]. So re-cherry-pick, now requires enabling branch-2.9 force
push and frozen branch-2.9.

reset commit 5ed247de3a.

now branch-2.9 has checkout a new branch[2]

sorry again, I will re-cherry-pick it in these two days

Thanks,
Bo

[1]  
https://github.com/apache/pulsar/commit/f1882ee4f0367c781144435a0eee93c1e3121fed
[2] https://github.com/apache/pulsar/tree/branch-2.9-backup

Re: [VOTE] Pulsar Release 2.9.4 Candidate 1

[丛搏]

Hi, all:

this candidate cancel ,
reason:https://lists.apache.org/thread/jcrq9q0k6kh9rvb71dwb9s3mvo5c5dk5

丛搏  于2022年11月24日周四 11:22写道：
>
> Docker images:
> https://hub.docker.com/layers/congbobo184/pulsar-all/2.9.4/images/sha256-53ef82d20fa7d9ee4809a3480f98cacea9b608cbb3a428881bcad3b8c77a9a32
> https://hub.docker.com/layers/congbobo184/pulsar/2.9.4/images/sha256-cdeaece1241ee3685f62dcf426944dd5fdd04863defad5762bd81aebdc31c251
>
> Please download the source package, and follow the Release Candidate
> Validation[1]
> to validate the release
> https://pulsar.apache.org/contribute/validate-release-candidate/
>
> 丛搏  于2022年11月24日周四 11:11写道：
> >
> > This is the first release candidate for Apache Pulsar, version 2.9.4.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.9.4+
> >
> > *** Please download, test and vote on this release. This vote will stay open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.4-candidate-1/
> >
> > SHA-512 checksums:
> >
> > 6338a5e3602bca4a0cdcb4e255c5a2cd457369d17fb31cc3efefd5d929f952da7bab23bfdf8e7aee9a78c0337170a9f33c10c65f1e83133f8f04802f55d8d084
> >  apache-pulsar-2.9.4-bin.tar.gz
> >
> > 3a8175dced0f69b1952c7d95e316a7d8cef193aa4b0635f468c3ba3e6911e477235f26cdc3d71bc4b8540d8d94fa9536e286b1485805d04abadacc4959818ad3
> >  apache-pulsar-2.9.4-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1191/
> >
> > The tag to be voted upon:
> > v2.9.4-candidate-1 (56a5bace17694c788d759c6e9fbb1da5ac58c3f1)
> > https://github.com/apache/pulsar/releases/tag/v2.9.4-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the Release Candidate
> > Validation[1]
> > to validate the release
> >
> > [1] https://github.com/apache/pulsar/wiki/Release-Candidate-Validation

Re: [DISCUSS] Enable branch-2.9 force push and frozen branch-2.9

[PengHui Li]

Hi Bo,

Thanks for raising the discussion.

+1 for re-cherry-pick the commits.
The misleading commit logs will confuse users.

Regards,
Penghui

On Fri, Nov 25, 2022 at 12:29 PM 丛搏  wrote:

> Hi, pulsar community,
>
> I am sorry for that, there is no commit to execute cherry-pick
> --continue, resulting in many cherry-pick commit messages not being
> correct[1]. So re-cherry-pick, now requires enabling branch-2.9 force
> push and frozen branch-2.9.
>
> reset commit 5ed247de3a.
>
> now branch-2.9 has checkout a new branch[2]
>
> sorry again, I will re-cherry-pick it in these two days
>
> Thanks,
> Bo
>
> [1]
> https://github.com/apache/pulsar/commit/f1882ee4f0367c781144435a0eee93c1e3121fed
> [2] https://github.com/apache/pulsar/tree/branch-2.9-backup
>

Re: [DISCUSS] PIP-223: Add metrics for all Rest Endpoints

[Haiting Jiang]

+1, Great work.


Haiting

On Thu, Nov 24, 2022 at 6:25 PM Jiuming Tao
 wrote:
>
> Hi Haiting,
>
> I’ll update the PIP as we discussed:
>
> 1. `pulsar_broker_rest_endpoint_failed`
> Labels:
>a. path: The HTTP request path
>b. method: The HTTP request method
>c. code: The HTTP response code
> This metric only records HTTP requests which response code >= 400
>
> 2. `pulsar_broker_rest_endpoint_latency`
> Labels:
>a. path: The HTTP request path
>b. method: The HTTP request method
>c. code: The HTTP response code
> This metric records all the HTTP requests, including failed requests.
>
>
> Thanks,
> Tao Jiuming
>
> Haiting Jiang  于2022年11月24日周四 16:35写道：
>
> > Hi Jiuming,
> >
> > > I’m not sure that if observe failed requests latency is meaningful.
> >
> > 1. Normally for failed requests, the latency sometimes will help in
> > debugging.
> > 2. Add httpcode label to latency, we can analyze the latency between
> > different successful results too, like 200 and 307.
> >
> > As for `pulsar_broker_rest_endpoint_failed`, after a second thought. I
> > think it would be useful for operators to set up system alarms. So +1
> > on this.
> >
> > Thanks,
> > Haiting
> >
> > On Thu, Nov 24, 2022 at 12:52 AM Jiuming Tao
> >  wrote:
> > >
> > > Hi Haiting,
> > >
> > > I’m not sure that if observe failed requests latency is meaningful, so
> > that I was added `pulsar_broker_rest_endpoint_failed` to record these
> > requests which is failed.
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > > > 2022年11月23日 下午12:55，Haiting Jiang  写道：
> > > >
> > > > Hi Jiuming,
> > > >
> > > > I am not sure why we need a new metric for failed requests.
> > > > Can we just put the `httpcode` tag also in the
> > > > `pulsar_broker_rest_endpoint_latency_ms`?
> > > > So that user can also see the latency of failed requests (like timeout
> > > > requests).
> > > > And it can easily cover the case of failed metrics.
> > > >
> > > > Thanks,
> > > > Haiting
> > > >
> > > > On Wed, Nov 23, 2022 at 12:00 PM Jiuming Tao
> > > >  wrote:
> > > >>
> > > >> Hi Penghui,
> > > >>
> > > >>> We'd better add an option to enable or disable the endpoint-level
> > metrics.
> > > >>> And keep it as disabled as default.
> > > >>
> > > >> Agreed, I also considered if we need a configuration to
> > enable/disable the feature, seems add an option is better
> > > >>
> > > >>> So that we can
> > > >>> get the request path from the handle method
> > > >>> `public void handle(String path, Request baseRequest,
> > HttpServletRequest
> > > >>> request, HttpServletResponse response)`
> > > >>
> > > >> I’ve tried this, but the `path` is the request path like
> > `/user/111/query`, and we cannot get the path configured by @Path by using
> > `Handler`.
> > > >> Jetty is a Servlet container, and we are using Jersey to provide Rest
> > API services, I think use Jersey’s mechanism is better
> > > >>
> > > >> Thanks,
> > > >> Tao Jiuming
> > > >>
> > > >>
> > > >>
> > > >>> 2022年11月23日 上午9:37，PengHui Li  写道：
> > > >>>
> > > >>> Hi, Jiuming
> > > >>>
> > > >>> Thanks for starting the proposal.
> > > >>>
> > > >>> We'd better add an option to enable or disable the endpoint-level
> > metrics.
> > > >>> And keep it as disabled as default.
> > > >>>
> > > >>> I noticed the existing jetty metrics are based on
> > > >>> `org.eclipse.jetty.server.handler.StatisticsHandler`.
> > > >>> Can we just have a new StatisticsHandler? e.g.
> > EndpointStatisticsHandler.
> > > >>> So that we can
> > > >>> get the request path from the handle method
> > > >>> `public void handle(String path, Request baseRequest,
> > HttpServletRequest
> > > >>> request, HttpServletResponse response)`
> > > >>>
> > > >>> Thanks,
> > > >>> Penghui
> > > >>>
> > > >>> On Tue, Nov 22, 2022 at 5:20 PM Jiuming Tao
> > 
> > > >>> wrote:
> > > >>>
> > >  Hi Haiting,
> > > 
> > >  I'm sorry I didn't explain FAILED, the FAILED means the HTTP
> > response code
> > > > = 400, and I’ll update the PIP later.
> > > 
> > >  Thanks,
> > >  Tao Jiuming
> > > 
> > > > 2022年11月22日 下午5:15，Haiting Jiang  写道：
> > > >
> > > > Hi Jiuming,
> > > >
> > > > Overall, this PIP makes sense to me.
> > > > About the metric "pulsar_broker_rest_endpoint_failed", please
> > provide
> > > > a more clear definition of "fail". Are redirects like 403 included?
> > > >
> > > > Thanks,
> > > > Haiting
> > > >
> > > > On Tue, Nov 22, 2022 at 3:17 AM Jiuming Tao
> > > >  wrote:
> > > >>
> > > >> Hi pulsar community,
> > > >>
> > > >> I’ve opened a PIP to discuss: PIP-223: Add metrics for all Rest
> > >  Endpoints
> > > >>
> > > >> The PIP link: https://github.com/apache/pulsar/issues/18560 <
> > >  https://github.com/apache/pulsar/issues/18560>
> > > >>
> > > >> Thanks,
> > > >> Tao Jiuming
> > > 
> > > 
> > > >>
> > >
> >

Re: [DISCUSS] Enable branch-2.9 force push and frozen branch-2.9

[guo jiwei]

+1 for re-cherry-pick the commits.


Regards
Jiwei Guo (Tboy)


On Fri, Nov 25, 2022 at 1:10 PM PengHui Li  wrote:

> Hi Bo,
>
> Thanks for raising the discussion.
>
> +1 for re-cherry-pick the commits.
> The misleading commit logs will confuse users.
>
> Regards,
> Penghui
>
> On Fri, Nov 25, 2022 at 12:29 PM 丛搏  wrote:
>
> > Hi, pulsar community,
> >
> > I am sorry for that, there is no commit to execute cherry-pick
> > --continue, resulting in many cherry-pick commit messages not being
> > correct[1]. So re-cherry-pick, now requires enabling branch-2.9 force
> > push and frozen branch-2.9.
> >
> > reset commit 5ed247de3a.
> >
> > now branch-2.9 has checkout a new branch[2]
> >
> > sorry again, I will re-cherry-pick it in these two days
> >
> > Thanks,
> > Bo
> >
> > [1]
> >
> https://github.com/apache/pulsar/commit/f1882ee4f0367c781144435a0eee93c1e3121fed
> > [2] https://github.com/apache/pulsar/tree/branch-2.9-backup
> >
>

Re: [DISCUSS] Enable branch-2.9 force push and frozen branch-2.9

[Yubiao Feng]

+1 for re-cherry-pick the commits.

On Fri, Nov 25, 2022 at 12:29 PM 丛搏  wrote:

> Hi, pulsar community,
>
> I am sorry for that, there is no commit to execute cherry-pick
> --continue, resulting in many cherry-pick commit messages not being
> correct[1]. So re-cherry-pick, now requires enabling branch-2.9 force
> push and frozen branch-2.9.
>
> reset commit 5ed247de3a.
>
> now branch-2.9 has checkout a new branch[2]
>
> sorry again, I will re-cherry-pick it in these two days
>
> Thanks,
> Bo
>
> [1]
> https://github.com/apache/pulsar/commit/f1882ee4f0367c781144435a0eee93c1e3121fed
> [2] https://github.com/apache/pulsar/tree/branch-2.9-backup
>

Re: [DISCUSS] Enable branch-2.9 force push and frozen branch-2.9

[Hang Chen]

+1 for re-cherry-picking the commits.

Thanks,
Hang

Yubiao Feng  于2022年11月25日周五 13:58写道：
>
> +1 for re-cherry-pick the commits.
>
> On Fri, Nov 25, 2022 at 12:29 PM 丛搏  wrote:
>
> > Hi, pulsar community,
> >
> > I am sorry for that, there is no commit to execute cherry-pick
> > --continue, resulting in many cherry-pick commit messages not being
> > correct[1]. So re-cherry-pick, now requires enabling branch-2.9 force
> > push and frozen branch-2.9.
> >
> > reset commit 5ed247de3a.
> >
> > now branch-2.9 has checkout a new branch[2]
> >
> > sorry again, I will re-cherry-pick it in these two days
> >
> > Thanks,
> > Bo
> >
> > [1]
> > https://github.com/apache/pulsar/commit/f1882ee4f0367c781144435a0eee93c1e3121fed
> > [2] https://github.com/apache/pulsar/tree/branch-2.9-backup
> >

Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

[Nozomi Kurihara]

Hi Heesung,

Thank you for opening this discussion.

IMO backward-compatibility should be kept at least across minor releases.

Although the performance issue you mentioned (e.g. memory burst, high GC
and OOM) looks problematic, backward-compatibility is also important.
I think which has higher priority depends on the case.

Your current change seems to remove the index-based aggregation completely.
However I think we should keep room for choice.

In order to allow users (here Pulsar server-side admins in particular) to
choose the performance or backward-compatibility, how about introducing a
"force" setting, e.g. "forceAggregatePublisherStatsByProducerName"?

Those who place more importance on the performance than
backward-compatibility can set this flag to true.
Others, those who want to keep backward-compatibility, set this flag to
false.

By the way, I'm not sure, is the producer name generation logic already
implemented in C++, Go and other clients?
If not so, first we should implement it before switching the
producer-name-based aggregation.

Best Regards,
Nozomi

2022年11月17日(木) 8:51 Heesung Sohn :

> Hi,
>
> To add more about the backward incompatibility issue
> ,
>
> Before fix:
> % ./bin/pulsar-admin topics partitioned-stats
> persistent://public/default/pt
> ...
>   "publishers" : [ {
> "msgRateIn" : 0.0,
> "msgThroughputIn" : 0.0,
> "averageMsgSize" : 0.0,
> "chunkedMessageRate" : 0.0,
> "producerId" : 0,
> "supportsPartialProducer" : false
>   } ],
>
> After fix:
> % ./bin/pulsar-admin topics partitioned-stats
> persistent://public/default/pt
> ...
>   "publishers" : [ {
> "msgRateIn" : 0.0,
> "msgThroughputIn" : 0.0,
> "averageMsgSize" : 0.0,
> "chunkedMessageRate" : 0.0,
> "producerId" : 0,
> "supportsPartialProducer" : true,
> "producerName" : "standalone-0-1"
>   }, {
> "msgRateIn" : 0.0,
> "msgThroughputIn" : 0.0,
> "averageMsgSize" : 0.0,
> "chunkedMessageRate" : 0.0,
> "producerId" : 0,
> "supportsPartialProducer" : true,
> "producerName" : "standalone-0-0"
>   } ],
> ...
>
>
> The broker side's producer name generation has been there since this PR
> (1.22-incubating).
> ProducerName was automatically generated in this format
> {clusterName}-{brokerInstanceId}-{producerNameGenerationCounter} until
> 2.10.
> So, a producer to a partitioned topic(2) results in the two producer names,
> like the following.
> standalone-0-0
> standalone-0-1
>
> And since 2.10, by default, partitioned producers have the same producer
> name between partitions by this PR
> (generated on the client
> side).
> standalone-0-0
>
> Hence, the impacted versions(backward incompatibility issue
> ) by
> the proposed fix(Option 1 below) are < 2.10.
>
> When we aggregate stats between partitions, the default(
> aggregatePublisherStatsByProducerName=false) aggregates the producer stats
> by the index of the producer stat list from each partition. So, when lucky,
> it could output a single producer stat. However, this method can be buggy,
> as each partition could return a different size and index of the
> producer stat list.
>
>
> To fix the original issue(described here
> ), I think we have the
> following options.
>
> Option 1(proposed): Deprecate aggregatePublisherStatsByProducerName (the
> current PR here ) in the next
> release and live with behavior change where we get `topics
> partitioned-stats` per-producer-and-partition from old clients(Ver. <2.10),
> instead of stats per-producer.
>
> Option 2: Defer the Option 1 fix and push it to the next major
> version(3.0.0), as this is a breaking change.
>
> Option 3: Keep aggregatePublisherStatsByProducerName config but change the
> default to aggregatePublisherStatsByProducerName=true.
>
> Option 4: As a long-term fix, create separate Admin-APIs for publisher and
> subscription stats and drop their stats from `topics partitioned-stats` as
> it is expensive to aggregate them on the fly. (for thousands of publishers
> and subscriptions). Push this change to the next major version.
>
> Or other suggestions?
>
> Regards,
> Heesung
>
>
>
>
>
>
> On Mon, Nov 14, 2022 at 7:56 PM Heesung Sohn  >
> wrote:
>
> > Dear Pulsar Community,
> >
> > We recently found a bug in `pulsar-admin topics partitioned-stats api`
> that
> > could incur a memory burst, high GC time, or OOM.
> >
> > For this issue, I proposed a fix
> >  by deprecating the
> aggregatePublisherStatsByProducerName
> > config and always aggregating the publishers' stats by publisherName,
> > instead of the list index(aggregatePublisherStatsByProducerName=false,
> > default).
> >
> >
> > 

Re: [ANNOUNCE] New Committer: Cong Zhao

[Qiang Huang]

Congrats!

Nicolò Boschi  于2022年11月22日周二 15:22写道：

> Congrats!
> Nicolò Boschi
>
>
> Il giorno mar 22 nov 2022 alle ore 06:23 Hang Chen 
> ha
> scritto:
>
> > Congrats!
> >
> > Best,
> > Hang
> >
> > tison  于2022年11月22日周二 13:15写道：
> > >
> > > Congrats!
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Max Xu  于2022年11月22日周二 13:14写道：
> > >
> > > > Congratulations! Cong
> > > >
> > > > Best,
> > > > Max Xu
> > > >
> > > >
> > > > On Mon, Nov 21, 2022 at 12:10 PM Haiting Jiang <
> > jianghait...@apache.org>
> > > > wrote:
> > > >
> > > > > The Project Management Committee (PMC) for Apache Pulsar has
> invited
> > > > > Cong Zhao (https://github.com/coderzc)
> > > > > to become a committer and we are pleased to announce that he has
> > > > accepted.
> > > > >
> > > > > Being a committer enables easier contribution to the
> > > > > project since there is no need to go via the patch
> > > > > submission process. This should enable better productivity.
> > > > >
> > > > > Welcome and congratulations, Cong Zhao!
> > > > >
> > > > > Please join us in congratulating and welcoming Cong Zhao onboard!
> > > > >
> > > > > Best Regards,
> > > > > Haiting on behalf of the Pulsar PMC
> > > > >
> > > >
> >
>


-- 
BR,
Qiang Huang