Hi all, I'm very excited to release a PIP I've been working on in the past 11 months, which I think will be immensely valuable to Pulsar, which I like so much.
PIP: https://github.com/apache/pulsar/issues/20197 I'm quoting here the preface: === QUOTE START === Roughly 11 months ago, I started working on solving the biggest issue with Pulsar metrics: the lack of ability to monitor a pulsar broker with a large topic count: 10k, 100k, and future support of 1M. This started by mapping the existing functionality and then enumerating all the problems I saw (all documented in this doc <https://docs.google.com/document/d/1vke4w1nt7EEgOvEerPEUS-Al3aqLTm9cl2wTBkKNXUA/edit?usp=sharing> ). This PIP is a parent PIP. It aims to gradually solve (using sub-PIPs) all the current metric system's problems and provide the ability to monitor a broker with a large topic count, which is currently lacking. As a parent PIP, it will describe each problem and its solution at a high level, leaving fine-grained details to the sub-PIPs. The parent PIP ensures all solutions align and does not contradict each other. The basic building block to solve the monitoring ability of large topic count is aggregating internally (to topic groups) and adding fine-grained filtering. We could have shoe-horned it into the existing metric system, but we thought adding that to a system already ingrained with many problems would be wrong and hard to do gradually, as so many things will break. This is why the second-biggest design decision presented here is consolidating all existing metric libraries into a single one - OpenTelemetry <https://opentelemetry.io/>. The parent PIP will explain why OpenTelemetry was chosen out of existing solutions and why it far exceeds all other options. I’ve been working closely with the OpenTelemetry community in the past eight months: brain-storming this integration, and raising issues, in an effort to remove serious blockers to make this migration successful. I made every effort to summarize this document so that it can be concise yet clear. I understand it is an effort to read it and, more so, provide meaningful feedback on such a large document; hence I’m very grateful for each individual who does so. I think this design will help improve the user experience immensely, so it is worth the time spent reading it. === QUOTE END === Thanks! Asaf Mesika
