I was using $_interval since delta is for gauge. I now changed to $_rate_interval and it makes no difference whatsoever. I also really made sure to set an interval where I know I have a lag I want to catch. Am I really understanding delta correctly.
I expect to get the difference between first value in the series and the last value of the series. If I start with 0, lots happens in between, lag goes down to 0 by the end of my interval. I expect to get 0. If I start on 0, lag increases to 10, I expect to get 10 If I start on 10 and goes down to 2, I expect to -8 On Monday, July 10, 2023 at 4:24:26 PM UTC+2 Ben Kochie wrote: > Use `$__rate_interval` to make sure you get enough samples for `delta()` > to work. (delta(), rate(), increase(), function similarly). > > On Mon, Jul 10, 2023 at 4:17 PM Helena Lundell <[email protected]> wrote: > >> Thanks for the fast reply and sorry for my late response. >> >> We have a scraping on 15 seconds. My query interval is 30 min. So I >> should have data. And the Grafana dashboard below where I add >> difference(changed from delta) on legend values I see what I want. >> This difference from the Grafana panel is exactely what I am looking for >> and thought delta would give me. >> >> Is there an alternative to delta? Like first-last? >> On Saturday, July 1, 2023 at 3:07:22 PM UTC+2 Ben Kochie wrote: >> >>> Grafana defaults to a "minimum step" of 15 seconds. I bet your scrape >>> interval is longer than 15 seconds, so "$__interval" is matching no data. >>> >>> You can either adjust the datasource minimum step to your shortest >>> scrape interval, or set the "min step" in the query options to match the >>> scrape interval of your metric. >>> >>> On Sat, Jul 1, 2023 at 3:04 PM Helena Lundell <[email protected]> wrote: >>> >>>> I am trying to identify an alert for an increasing Kafka Lag. The >>>> metrics are in prometheus and I use Grafana 9.4 for visualisation. >>>> >>>> In Grafana I have a dasboard where I can see that the lag is increasing >>>> or decreasing. I have choosen to display the delta in the legend and it >>>> shows a number that does not really make sense, but it shows something. >>>> >>>> This is the base promql: >>>> avg by(consumergroup, topic, cluster) >>>> (kafka_consumergroup_lag{namespace="ns-kafka-int", >>>> consumergroup=~".*$container", cluster="$cluster"}) >>>> >>>> When I add delta to this query I get no data: >>>> avg by(consumergroup, topic, cluster) >>>> (delta(kafka_consumergroup_lag{namespace="ns-kafka-int", >>>> consumergroup=~".*$container", cluster="$cluster"}[$__interval])) >>>> >>>> So adding delta gives me no result even though when I exclude delta I >>>> can see that the metric has been changing. >>>> >>>> The metric is a gaugue. >>>> >>>> Question: >>>> Am I doing something wrong? >>>> Can I trust the delta from the Grafana dashboard(someone might know)? >>>> Can I expect a positive or negative value to identify and increase? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/0f37aa72-dd90-4329-8e19-5587460b80c4n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/0f37aa72-dd90-4329-8e19-5587460b80c4n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/63cf733e-87bd-4530-9531-5a3a29556d8an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/63cf733e-87bd-4530-9531-5a3a29556d8an%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ae65d8bc-f7d2-4961-a38c-8df791f7373en%40googlegroups.com.
