> I am working on bumping Prometheus client to 0.12.0 What is the motivation for the update? If it is security related, that might help us make a decision here.
> If we want to be compatible with Open Metrics, I suggest adding metrics > named `_total` in a release version like 2.10.0, and removing the origin > metric in the next release like 2.11.0. My main concern with this solution is that it would increase the size of the metrics payload by double (for the counters). Pulsar already produces a lot of metrics. (Tangentially, I think we should make the number of buckets configurable to cut down on the metrics payload size, as proposed here [0].) I think this is the kind of change that we can make during a minor version bump as long as we give users proper notice. The main drawback is that users will lose historical context for prometheus data from one version of Pulsar to the next, but since Prometheus is pushing us to make this change, it's already a change we'll have to make. Thanks, Michael [0] https://github.com/apache/pulsar/issues/12069 On Wed, Feb 23, 2022 at 9:41 AM ZhangJian He <shoot...@gmail.com> wrote: > > ping @enrico @matteo > Please take a look when you have time. > > Thanks > ZhangJian He > > ZhangJian He <shoot...@gmail.com> 于2022年2月13日周日 09:47写道： > > > ping @enrico @matteo > > Please take a look when you have time. > > > > Thanks > > ZhangJian He > > > > ZhangJian He <shoot...@gmail.com> 于2022年2月11日周五 14:09写道： > > > >> ping @enrico @matteo > >> > >> ZhangJian He <shoot...@gmail.com> 于2022年2月8日周二 16:07写道： > >> > >>> Sorry for missing the information. > >>> Before I upgrade the prom client, pulsar metrics is > >>> ``` > >>> > >>> - pulsar_connection_closed_total_count > >>> > >>> - pulsar_connection_created_total_count > >>> > >>> - pulsar_source_received_total_1min > >>> > >>> - system_exceptions_total_1min > >>> > >>> ``` > >>> > >>> After > >>> > >>> ``` > >>> > >>> - pulsar_connection_closed_total_count_total > >>> > >>> - pulsar_connection_created_total_count_total > >>> > >>> - pulsar_source_received_total_1min_total > >>> > >>> - system_exceptions_total_1min_total > >>> > >>> ``` > >>> > >>> Prometheus client adds a `_total` suffix in pulsar metrics, because they > >>> require all counters to have `_total` suffix, if your metric name is > >>> not ended with `_total`, they will add it. > >>> > >>> I believe that the right name which satisfies `OpenMetrics` should be > >>> ``` > >>> > >>> - pulsar_connection_closed_total > >>> > >>> - pulsar_connection_created_total > >>> > >>> - pulsar_source_received_1min_total > >>> > >>> - system_exceptions_1min_total > >>> > >>> ``` > >>> > >>> Summary, upgrade prometheus client introduces breaking change for these > >>> metrics names which did not end with `_total`. > >>> > >>> > >>> PS: If you let the prometheus client add `_total` in the previous > >>> version, these metrics are not impacted. > >>> > >>> Enrico Olivelli <eolive...@gmail.com> 于2022年2月8日周二 15:54写道： > >>> > >>>> What happens when you upgrade the Prometheus client ? > >>>> > >>>> Can you share some examples of "before" and "after" ? > >>>> My understanding is that you posted how it looks like "after" the > >>>> upgrade > >>>> > >>>> Thanks for working on this > >>>> > >>>> Enrico > >>>> > >>>> Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He > >>>> <shoot...@gmail.com> ha scritto: > >>>> > > >>>> > Before, I am working on bumping Prometheus client to 0.12.0, but they > >>>> > introduce a breaking change, > >>>> > https://github.com/prometheus/client_java/pull/615, adopt the > >>>> `OpenMetrics > >>>> > format`, which acquired all counters have `_total` suffix, > >>>> > > >>>> > but our metrics now have these metrics, there are not satisfied with > >>>> the > >>>> > OpenMetrics format, for example: > >>>> > > >>>> > - pulsar_connection_closed_total_count > >>>> > > >>>> > - pulsar_connection_created_total_count > >>>> > > >>>> > - pulsar_source_received_total_1min > >>>> > > >>>> > - system_exceptions_total_1min > >>>> > > >>>> > > >>>> > I want to discuss, Should we adapt the `OpenMetrics format`? > >>>> > > >>>> > If we want to be compatible with Open Metrics, I suggest adding > >>>> metrics > >>>> > named `_total` in a release version like 2.10.0, and removing the > >>>> origin > >>>> > metric in the next release like 2.11.0. > >>>> > >>>
