I'm very appreciative for feedback from multiple pulsar users and devs on this PIP, since it has dramatic changes suggested and quite extensive positive change for the users.
On Thu, Apr 27, 2023 at 7:32 PM Asaf Mesika <asaf.mes...@gmail.com> wrote: > Hi all, > > I'm very excited to release a PIP I've been working on in the past 11 > months, which I think will be immensely valuable to Pulsar, which I like so > much. > > PIP: https://github.com/apache/pulsar/issues/20197 > > I'm quoting here the preface: > > === QUOTE START === > > Roughly 11 months ago, I started working on solving the biggest issue with > Pulsar metrics: the lack of ability to monitor a pulsar broker with a large > topic count: 10k, 100k, and future support of 1M. This started by mapping > the existing functionality and then enumerating all the problems I saw (all > documented in this doc > <https://docs.google.com/document/d/1vke4w1nt7EEgOvEerPEUS-Al3aqLTm9cl2wTBkKNXUA/edit?usp=sharing> > ). > > This PIP is a parent PIP. It aims to gradually solve (using sub-PIPs) all > the current metric system's problems and provide the ability to monitor a > broker with a large topic count, which is currently lacking. As a parent > PIP, it will describe each problem and its solution at a high level, > leaving fine-grained details to the sub-PIPs. The parent PIP ensures all > solutions align and does not contradict each other. > > The basic building block to solve the monitoring ability of large topic > count is aggregating internally (to topic groups) and adding fine-grained > filtering. We could have shoe-horned it into the existing metric system, > but we thought adding that to a system already ingrained with many problems > would be wrong and hard to do gradually, as so many things will break. This > is why the second-biggest design decision presented here is consolidating > all existing metric libraries into a single one - OpenTelemetry > <https://opentelemetry.io/>. The parent PIP will explain why > OpenTelemetry was chosen out of existing solutions and why it far exceeds > all other options. I’ve been working closely with the OpenTelemetry > community in the past eight months: brain-storming this integration, and > raising issues, in an effort to remove serious blockers to make this > migration successful. > > I made every effort to summarize this document so that it can be concise > yet clear. I understand it is an effort to read it and, more so, provide > meaningful feedback on such a large document; hence I’m very grateful for > each individual who does so. > > I think this design will help improve the user experience immensely, so it > is worth the time spent reading it. > > > === QUOTE END === > > > Thanks! > > Asaf Mesika >
