If you refactor the rules a bit, you may find them easier to maintain:
alert1:
expr: probe_success{somelabel="XYZ"} == 0
labels:
someswitch: foo
alert2:
expr: probe_success{somelabel="ABC"} == 0
labels:
someswitch: bar
alert3:
expr: |
probe_success == 0
unless probe_success{somelabel="XYZ"} == 0
unless probe_success{somelabel="ABC"} == 0
The 'special cases' alert1 and alert2 have particular rules; alert3 has the
generic catch-all rule with 'unless' blocks to suppress the alert1 and
alert2 cases, but using identical expressions.
I think this approach is easier to reason about than having to generate new
expressions with inverted logic. The expressions do have to return the
same label sets (which, in the case of the same metric, should be true)
On Wednesday, 18 January 2023 at 13:48:59 UTC [email protected] wrote:
> If your special cases have a *longer* "for" duration than the general
> ones, then I guess they won't be useful for inhibiting the general ones,
> since the special cases will start firing too late relative to the general
> ones to inhibit them. I guess you could introduce a copy of each special
> case alert without any "for" duration (or a shorter one) that you don't
> route anywhere and that is only used for inhibitions. And then you have a
> second version of it that's actually routed, with a longer "for" duration?
>
> Whether that's more maintainable than going for !~ and =~ regex matchers
> as you described is a good question though. Maybe rather than
> distinguishing each special case in the alerting rules themselves, maybe
> you can attach a special new (single) label to your targets that
> differentiate the general ones from the longer "for" duration ones, so you
> can just use that one label for filtering in the rules?
>
> On Wed, Jan 18, 2023 at 2:00 PM Mario Cornaccini <[email protected]>
> wrote:
>
>> hi,
>>
>> for the same metric, i want to have multiple rules in alert manager, to
>> have longer FOR: times for some special cases.
>>
>> the way i do this now is:
>> alert1 # general
>> probe_succes{ somelabel !~"specialcase1|specialcase2"}
>> alert2 # special
>> probe_success{somelabel =~"specialcase1|specialcase2"}
>> .. which is obviously badly maintainable and ugly and won't scale..
>>
>> i've seen this,
>> https://www.robustperception.io/using-time-series-as-alert-thresholds/..
>> but it looks a bit, well, hard to maintain..
>>
>>
>>
>> so i got this idea, what would happen if i did this:
>>
>> in prometheus rules :
>> alert1 # handles the special case
>> ie. probe_success{somelabel="XYZ"}
>> labels:
>> someswitch: true
>>
>> alert 2 #handles the general case
>> probe_success{}
>>
>> and in alert manager:
>> define an inhibit rule which mutes the general alert, if there is also an
>> special case one, based on the someswitch label, would that work?
>>
>> any help/pointers/comments greatly appreciated,
>> cheers,
>> mario
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/5fe28eec-a91c-4619-af41-128966ad08d9n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/5fe28eec-a91c-4619-af41-128966ad08d9n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> Julius Volz
> PromLabs - promlabs.com
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d33055a4-2b6d-4dca-8d3e-e4a52ef4b18an%40googlegroups.com.
