Hi, Yes, it also sounds generally speaking good to me.
I'm just not sure whether we should make the existing `Counter` monotonic, or introduce the new `MonotonicCounter`. If we do the former, how would you propose Efrat to do that? Just deprecating `Counter#dec` in let's say Flink 2.4, will not make `Counter` monotonic. That will happen only once we drop `Counter#dec` (in? 2.5? 2.6? 3.0?). Or am I missing something? Best, Piotrek czw., 2 lip 2026 o 23:26 Ryan van Huuksloot via dev <[email protected]> napisał(a): > Hi Efrat, > > Sorry for the delay. > > Overall this sounds good to me. We haven't encountered this too often but I > do know what you are referring to; it's annoying to know if the counter was > decremented versus reset. > > I'm not well versed in all of the touchpoints to give the best guidance but > it does make sense and is a really nice quality of life improvement for > observability. > > Ryan van Huuksloot > Staff Engineer, Infrastructure | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Thu, Jun 25, 2026 at 1:24 PM Efrat Levitan <[email protected]> > wrote: > > > Hey Ryan, > > Thank you for reading through the description. > > I published the FLIP in hope it will provides a better understanding > > of the proposal [1] > > > > > Is the issue you are trying to resolve that we don't know if a Counter > > is monotonic? > > Yes this is the main concern of the FLIP, distinguishing monotonicity > > so backend systems can tell that there was integer overflow or system > > restart during a given time and are able to smooth it out in graphs. > > If Counter.dec was widely used in flink we were having a different > > discussion, but since it was only historically introduced (Dropwizard > > compatibility?) and never used within our codebase, I believe > > declaring monotonicity is worth deprecating Counter.dec (+ providing > > the UpDownCounter alternative) > > > > > Prometheus casts it to a Gauge because that code path hasn't been > > upgraded in years > > Prometheus' latest java client still throws[2] if you try to decrease > > it. This is by design, and the reason flink counters are mapped to > > prom gauges. > > You didn't share more about the internal implementation but if it uses > > prometheus counters, I'd assume negative deltas are ignored as in OTel > > reporter today > > This FLIP can actually pave the way for it to be contributed- > > with true monotonicity, Flink counter will correctly map to prometheus > > Counter (please check out the "proposed change" section of the FLIP) > > > > Let me know what you think. > > Efrat > > > > [1] https://cwiki.apache.org/confluence/x/twDuGQ > > [2] > > > https://github.com/prometheus/client_java/blob/main/prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Counter.java#L189 > > > > On Mon, 22 Jun 2026 at 17:23, Ryan van Huuksloot via dev > > <[email protected]> wrote: > > > > > > Hi Efrat, > > > > > > I'd like to clarify the intention of the FLIP, as it seems like we are > > > introducing multiple issues. > > > > > > Is the issue you are trying to resolve that we don't know if a Counter > is > > > monotonic? > > > > > > After reading the links and the FLIP description, it sounds like we > need > > to > > > fix the OTEL reporter. I don't immediately see any major reason to > > overhaul > > > the entire counter system. The current Counter can go up and down, but > > the > > > OTEL reporter didn't respect that. > > > > > > I also wanted to point out that Prometheus casts it to a Gauge because > > that > > > code path hasn't been upgraded in years. It should use the Native > > Counters > > > that now exist in Prometheus, we just haven't done the work to > > > upgrade/migrate. We have an internal implementation to move Flink > > Counters > > > to Prometheus Native Counters. > > > > > > Is there something I am missing? > > > > > > Ryan van Huuksloot > > > Staff Engineer, Infrastructure | Streaming Platform > > > [image: Shopify] > > > < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > > > > > On Mon, Jun 15, 2026 at 2:07 PM Efrat Levitan <[email protected]> > > wrote: > > > > > > > Hi all, > > > > I would like to start a discussion about how to differentiate > > > > monotonic and non monotonic counters in flink metrics. > > > > > > > > Monotonic (ever-increasing) Counters can benefit from automatic reset > > > > detection on the monitoring system side (when the value drops we can > > > > safely assume the process was reset and synthetically adjust the > > > > value) > > > > > > > > Historically, flink Counter can be decremented / incremented by a > > > > non-positive value, but this has almost never been used intentionally > > > > across the flink codebase, i.e if a counter got decremented this is > > > > usually a bug [1]. > > > > So though system-emitted counters are effectively monotonic, > exporters > > > > must respect org.apache.flink.metrics.Counter contract and assume > > > > non-monotonicity. > > > > > > > > I'd like to propose deprecating org.apache.flink.metrics.Counter#dec > > > > in favor of a new UpDownCounter implementation. This matches modern > > > > metric APIs like OTel, where the regular Counter is monotonic[2] and > > > > an additional UpDownCounter supports[3] non-positive additions. > > > > > > > > While it seems to be the cleanest approach, we could still avoid the > > > > deprecation by introducing a MonotonicCounter and have all flink > > > > counters migrated, or expand the Counter interface to declare > > > > monotonicity (based on the implementation). > > > > > > > > Recognising monotonicity will also align counters reporting across > > > > monitoring systems. Today, for instance, Otel reporter drops[4] > > > > non-incremental data points with a warning, while Prometheus reporter > > > > casts[5] them as Gauges. > > > > > > > > I'm looking forward to your feedback > > > > Efrat > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-39892 > > > > [2] > > > > > > > https://github.com/open-telemetry/opentelemetry-java/blob/main/api/all/src/main/java/io/opentelemetry/api/metrics/LongCounter.java#L40 > > > > [3] > > > > > > > https://github.com/open-telemetry/opentelemetry-java/blob/main/api/all/src/main/java/io/opentelemetry/api/metrics/LongUpDownCounter.java > > > > [4] https://issues.apache.org/jira/browse/FLINK-39893 > > > > [5] > > > > > > > https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L177-L184 > > > > > > >
