Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi - > On Sep 24, 2024, at 11:01 AM, Heesung Sohn wrote: > > Hi all, > > I realized that sampling outlier topic/subscription/consumer/producer > metrics can be useful to operate Pulsar. > > I noticed that Pulsar's performance issues are mostly caused by > outlier topics/subscription/consumers/

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi all, I realized that sampling outlier topic/subscription/consumer/producer metrics can be useful to operate Pulsar. I noticed that Pulsar's performance issues are mostly caused by outlier topics/subscription/consumers/producers. For example, disk usage can grow a lot when consumers are very s

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

I've relocated the PIP content from the issue ( https://github.com/apache/pulsar/issues/20197) to a PR ( https://github.com/apache/pulsar/pull/21080) so I could add TOC and also be inlined with the new process. On Mon, Aug 28, 2023 at 5:46 PM Asaf Mesika wrote: > Thanks for taking the time to

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Thanks for taking the time to review the document - *highly appreciated*. I'm inlined my comments below. On Mon, Aug 21, 2023 at 12:19 PM Hang Chen wrote: > Hi Asaf, > Thanks for bring up the great proposal. > > After reading this proposal, I have the following questions. > 1. This proposal

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi Asaf, Thanks for bring up the great proposal. After reading this proposal, I have the following questions. 1. This proposal will introduce a big break change in Pulsar, especially in code perspective. I’m interested in how to support both old and new implementation at the same time step by

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

> Proposing a large breaking change (even if it's crucial) is the single fastest way to motivate your users to migrate to a different platform. I wish it wasn't the case, but it's the cold reality. If you read the proposal, there is no real breaking change. There will be a switch to choose the exi

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

> Thanks for the details, Devin. Curios - 'We' stands for which company? What do you mean? I was quoting Rajan when I said, "we." Devin G. Bost On Wed, Jun 14, 2023 at 10:02 AM Asaf Mesika wrote: > Thanks for the details, Devin. Curios - 'We' stands for which company? > > Can you take a look

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Thanks for the details, Devin. Curios - 'We' stands for which company? Can you take a look at my previous response to see if it answers the concern you raised? Thanks! On Wed, Jun 14, 2023 at 1:49 PM Devin Bost wrote: > > Hi, > > > > Are we proposing a change to break existing metrics compati

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

On Wed, Jun 14, 2023 at 11:28 AM Enrico Olivelli wrote: > Il Mer 14 Giu 2023, 04:33 Rajan Dhabalia ha > scritto: > > > Hi, > > > > Are we proposing a change to break existing metrics compatibility > > (prometheus)? If that is the case then it's a big red flag as it will be > a > > pain for any c

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

In my proposal, I do suggest deleting the existing metric system. You still have the same Prometheus /metrics endpoint exposing metrics exactly as you have today. I plan to add an additional metric system based on OTel. You will be able to consume via its native exporters; 1. Prometheus - there wil

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

> Hi, > > Are we proposing a change to break existing metrics compatibility > (prometheus)? If that is the case then it's a big red flag as it will be a > pain for any company to upgrade Pulsar as monitoring is THE most important > part of the system and we don't even want to break compatibility fo

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Il Mer 14 Giu 2023, 04:33 Rajan Dhabalia ha scritto: > Hi, > > Are we proposing a change to break existing metrics compatibility > (prometheus)? If that is the case then it's a big red flag as it will be a > pain for any company to upgrade Pulsar as monitoring is THE most important > part of the

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi, Are we proposing a change to break existing metrics compatibility (prometheus)? If that is the case then it's a big red flag as it will be a pain for any company to upgrade Pulsar as monitoring is THE most important part of the system and we don't even want to break compatibility for any small

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

I'm super happy the video helped convey the long PIP in 30 minutes. On Tue, Jun 6, 2023 at 8:00 PM Dave Fisher wrote: > Hi Asaf, > > > > Just to be clear to all the phrase you used “we decided” along with the > timeline is not a Pulsar community we, but a StreamNative corporate we. > This discu

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

(2) Have a discussion in the Pulsar community about notions like Topic Group and Bundles so that we have the best and most operational logic around reducing cardinality and controlling load and performance. As long as we have bundles and orthogonally add groups we introduce complexity. Topic Groups

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi Asaf, I just watched your presentation from Pulsar Summit. https://www.youtube.com/watch?v=NN4KNK2r4mo Just to be clear to all the phrase you used “we decided” along with the timeline is not a Pulsar community we, but a StreamNative corporate we

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Thanks for the reply, Enrico. Completely agree. This made me realize my TL;DR wasn't talking about export. I added this to it: --- Pulsar OTel Metrics will support exporting as Prometheus HTTP endpoint (`/metrics` but different port) for backward compatibility and also OLTP, so you can push the me

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Asaf, thanks for contributing in this area. Metrics are a fundamental feature of Pulsar. Currently I find it very awkward to maintain metrics, and also I see it as a problem to support only Prometheus. Regarding your proposal, IIRC in the past someone else proposed to support other metrics system

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

> On May 10, 2023, at 1:01 AM, Asaf Mesika wrote: > > On Tue, May 9, 2023 at 11:29 PM Dave Fisher wrote: > >> >> On May 8, 2023, at 2:49 AM, Asaf Mesika wrote: >>> >>> Your feedback made me realized I need to add "TL;DR" section, which I >> just >>> added. >>> >>> I'm quoting it h

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Bumping, as I have received only 1 partial feedback so far on the PIP. Any feedback is highly appreciated! Thanks! Asaf On Wed, May 10, 2023 at 11:00 AM Asaf Mesika wrote: > > > On Tue, May 9, 2023 at 11:29 PM Dave Fisher wrote: > >> >> >> > On May 8, 2023, at 2:49 AM, Asaf Mesika wrote: >>

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

On Tue, May 9, 2023 at 11:29 PM Dave Fisher wrote: > > > > On May 8, 2023, at 2:49 AM, Asaf Mesika wrote: > > > > Your feedback made me realized I need to add "TL;DR" section, which I > just > > added. > > > > I'm quoting it here. It gives a brief summary of the proposal, which > > requires up t

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

On Tue, May 9, 2023 at 9:33 PM Yunze Xu wrote: > > Basically you have a full fledged metrics library objects: Meter, Gauge, > Histogram, Counter. > > It sounds good, but not so attractive. Currently KoP implements its > own metrics library objects. So after that, we need to leverage the > similar

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

> On May 8, 2023, at 2:49 AM, Asaf Mesika wrote: > > Your feedback made me realized I need to add "TL;DR" section, which I just > added. > > I'm quoting it here. It gives a brief summary of the proposal, which > requires up to 5 min of read time, helping you get a high level picture > before

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

> Basically you have a full fledged metrics library objects: Meter, Gauge, > Histogram, Counter. It sounds good, but not so attractive. Currently KoP implements its own metrics library objects. So after that, we need to leverage the similar classes from OTel. I want to talk a little more beyond

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

Your feedback made me realized I need to add "TL;DR" section, which I just added. I'm quoting it here. It gives a brief summary of the proposal, which requires up to 5 min of read time, helping you get a high level picture before you dive into the background/motivation/solution. -

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

On Sun, May 7, 2023 at 4:23 PM Yunze Xu wrote: > I'm excited to learn much more about metrics when I started reading > this proposal. But I became more and more frustrated when I found > there is still too much content left even if I've already spent much > time reading this proposal. I'm wonderi

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

I'm excited to learn much more about metrics when I started reading this proposal. But I became more and more frustrated when I found there is still too much content left even if I've already spent much time reading this proposal. I'm wondering how much time did you expect reviewers to read through

Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system

I'm very appreciative for feedback from multiple pulsar users and devs on this PIP, since it has dramatic changes suggested and quite extensive positive change for the users. On Thu, Apr 27, 2023 at 7:32 PM Asaf Mesika wrote: > Hi all, > > I'm very excited to release a PIP I've been working on

[DISCUSS] PIP-264: Enhanced OTel-based metric system

Hi all, I'm very excited to release a PIP I've been working on in the past 11 months, which I think will be immensely valuable to Pulsar, which I like so much. PIP: https://github.com/apache/pulsar/issues/20197 I'm quoting here the preface: === QUOTE START === Roughly 11 months ago, I started