Use `$__rate_interval` to make sure you get enough samples for `delta()` to work. (delta(), rate(), increase(), function similarly).
On Mon, Jul 10, 2023 at 4:17 PM Helena Lundell <[email protected]> wrote: > Thanks for the fast reply and sorry for my late response. > > We have a scraping on 15 seconds. My query interval is 30 min. So I should > have data. And the Grafana dashboard below where I add difference(changed > from delta) on legend values I see what I want. > This difference from the Grafana panel is exactely what I am looking for > and thought delta would give me. > > Is there an alternative to delta? Like first-last? > On Saturday, July 1, 2023 at 3:07:22 PM UTC+2 Ben Kochie wrote: > >> Grafana defaults to a "minimum step" of 15 seconds. I bet your scrape >> interval is longer than 15 seconds, so "$__interval" is matching no data. >> >> You can either adjust the datasource minimum step to your shortest scrape >> interval, or set the "min step" in the query options to match the scrape >> interval of your metric. >> >> On Sat, Jul 1, 2023 at 3:04 PM Helena Lundell <[email protected]> wrote: >> >>> I am trying to identify an alert for an increasing Kafka Lag. The >>> metrics are in prometheus and I use Grafana 9.4 for visualisation. >>> >>> In Grafana I have a dasboard where I can see that the lag is increasing >>> or decreasing. I have choosen to display the delta in the legend and it >>> shows a number that does not really make sense, but it shows something. >>> >>> This is the base promql: >>> avg by(consumergroup, topic, cluster) >>> (kafka_consumergroup_lag{namespace="ns-kafka-int", >>> consumergroup=~".*$container", cluster="$cluster"}) >>> >>> When I add delta to this query I get no data: >>> avg by(consumergroup, topic, cluster) >>> (delta(kafka_consumergroup_lag{namespace="ns-kafka-int", >>> consumergroup=~".*$container", cluster="$cluster"}[$__interval])) >>> >>> So adding delta gives me no result even though when I exclude delta I >>> can see that the metric has been changing. >>> >>> The metric is a gaugue. >>> >>> Question: >>> Am I doing something wrong? >>> Can I trust the delta from the Grafana dashboard(someone might know)? >>> Can I expect a positive or negative value to identify and increase? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/0f37aa72-dd90-4329-8e19-5587460b80c4n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/0f37aa72-dd90-4329-8e19-5587460b80c4n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/63cf733e-87bd-4530-9531-5a3a29556d8an%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/63cf733e-87bd-4530-9531-5a3a29556d8an%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoW6S71ege9UVLiqvh-q8DxQ2VP_kjmSuVe0FqQ6ej_MA%40mail.gmail.com.
