On Wed, Sep 20, 2023 at 5:25 AM David Leibovic <[email protected]>
wrote:
> Thanks very much to you both for the suggestions! Changing the regexp
> matching to exact string matching didn't help noticeably in my case,
> perhaps because I have less than 5 instances over which it has to do a
> regexp match. But removing the unnecessary lower bounds checks from my
> inequality checks reduced loading time by about 3/5.
>
> It's too bad that prometheus doesn't support something like variables that
> could be repeatedly referenced in promql - that would probably speed things
> up even more.
>
This has been discussed in the past. But it was possibly complicated to
implement.
Another way to improve this. I think what we could do is a query
optimization that automatically detects identical expressions and only
evaluates them once. This would make it transparent to the user without
having to change the language.
>
> On Monday, September 18, 2023 at 3:23:43 AM UTC-4 Brian Candler wrote:
>
>> One possibility is to use a recording rule for the expensive repeated
>> query.
>>
>> If I
>> rewrite avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval])
>> to X, I get:
>>
>> ((50 - 0) / (12 - 0) * ((X <= 12) - 0) + 0) or
>> ((100 - 51) / (35.4 - 12.1) * ((X > 12 and X <= 35.4) - 12.1) + 51) or
>> ((150 - 101) / (55.4 - 35.5) * ((X > 35.4 and X <= 55.4) - 35.5) + 101) or
>> ((200 - 151) / (150.4 - 55.5) * ((X > 55.4 and X <= 150.4) - 55.5) + 151)
>> or
>> ((300 - 201) / (250.4 - 150.5) * ((X > 150.4 and X <= 250.4) - 150.5) +
>> 201) or
>> ((400 - 301) / (350.4 - 250.5) * ((X > 250.4 and X <= 350.4) - 250.5) +
>> 301) or
>> ((500 - 401) / (500.4 - 350.5) * ((X > 350.4 and X <= 500.4) - 350.5) +
>> 401) or
>> clamp_max(X, 600)
>>
>> I guess you're trying to apply different scaling for different ranges of
>> X:
>> - if X is between 0 and 12 (or negative) then rescale to 0 to 50
>> - if X is between 12 and 35.4 then rescale to 50(?) to 100
>> - if X is between 35.4 and 55.4 then rescale to 100(?) to 150
>> etc (except there seem to be some small discontinuities at the
>> boundaries, e.g. 12 versus 12.1, 50 versus 51)
>>
>> "A or B" will suppress elements in the B vector where the A vector has a
>> value (i.e. with a matching label set). That means it's unnecessary to test
>> the lower bounds, and I think your expression could simplify to something
>> like this:
>>
>> (X <= 12) * k1 + o1 or
>> (X <= 35.4) * k2 + o2 or
>> (X <= 55.4) * k3 + o3 or
>> (X <= 150.4) * k4 + o4 or
>> (X <= 250.4) * k5 + o5 or
>> (X <= 350.4) * k6 + o6 or
>> (X <= 500.4) * k7 + o7 or
>> clamp_max(X, 600)
>>
>> That would roughly halve the number of the subexpressions X.
>>
>> On Monday, 18 September 2023 at 07:01:50 UTC+1 Ben Kochie wrote:
>>
>>> One thing you can do to speed things up is to eliminate the `=~` in your
>>> query. Using regexp matching means it has to do a string search over every
>>> instance in your Prometheus for each metric. Using exact matching (`=`)
>>> will speed things up a lot. Although you won't be able to do multiple
>>> matching if you want that in your dashboard variables.
>>>
>>> On Mon, Sep 18, 2023 at 7:58 AM David Leibovic <[email protected]>
>>> wrote:
>>>
>>>> Hi there, I'm trying to optimize a slow query of this form:
>>>>
>>>> (1 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 10) or
>>>> (2 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 20) or
>>>> (3 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 30) or
>>>> (10 * avg_over_time(foo{instance=~"$i"}[$interval]))
>>>>
>>>> I suspect it's slow because of the many duplicate calls to
>>>> avg_over_time(foo{instance=~"$i"}[$interval])
>>>>
>>>> Is there some way to only call the avg_over_time function once and
>>>> re-use the results subsequently? I'm using Prometheus in conjunction with
>>>> Grafana, in case it's relevant.
>>>>
>>>> The full query I'm trying to optimize is much more complicated, but I
>>>> figured the above would be enough to understand the problem. But in case
>>>> it's helpful, here is the full query I am trying to optimize (it's an Air
>>>> Quality Index computation):
>>>>
>>>> ((50 - 0) / (12 - 0) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 12) - 0) + 0) or
>>>> ((100 - 51) / (35.4 - 12.1) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 12
>>>> and avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 35.4) - 12.1) + 51) or
>>>> ((150 - 101) / (55.4 - 35.5) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>>>> 35.4 and
>>>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 55.4) - 35.5) + 101) or
>>>> ((200 - 151) / (150.4 - 55.5) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>>>> 55.4 and
>>>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 150.4) - 55.5) + 151) or
>>>> ((300 - 201) / (250.4 - 150.5) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>>>> 150.4 and
>>>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 250.4) - 150.5) + 201) or
>>>> ((400 - 301) / (350.4 - 250.5) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>>>> 250.4 and
>>>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 350.4) - 250.5) + 301) or
>>>> ((500 - 401) / (500.4 - 350.5) *
>>>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>>>> 350.4 and
>>>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>>>> 500.4) - 350.5) + 401) or
>>>> clamp_max(avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]),
>>>> 600)
>>>>
>>>> Thanks for any help you can provide!
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Prometheus Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/prometheus-users/664689b3-9f45-4b05-9438-4225e2dce773n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/prometheus-users/664689b3-9f45-4b05-9438-4225e2dce773n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/971f3928-d0df-412c-abeb-ae2713ddab89n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/971f3928-d0df-412c-abeb-ae2713ddab89n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CABbyFmp3%2BumxN2fnCOx2%3Dwj5uvRkK2wx1VdSRWuoY0DkWoLTgw%40mail.gmail.com.
