If your special cases have a *longer* "for" duration than the general ones, then I guess they won't be useful for inhibiting the general ones, since the special cases will start firing too late relative to the general ones to inhibit them. I guess you could introduce a copy of each special case alert without any "for" duration (or a shorter one) that you don't route anywhere and that is only used for inhibitions. And then you have a second version of it that's actually routed, with a longer "for" duration?
Whether that's more maintainable than going for !~ and =~ regex matchers as you described is a good question though. Maybe rather than distinguishing each special case in the alerting rules themselves, maybe you can attach a special new (single) label to your targets that differentiate the general ones from the longer "for" duration ones, so you can just use that one label for filtering in the rules? On Wed, Jan 18, 2023 at 2:00 PM Mario Cornaccini <[email protected]> wrote: > hi, > > for the same metric, i want to have multiple rules in alert manager, to > have longer FOR: times for some special cases. > > the way i do this now is: > alert1 # general > probe_succes{ somelabel !~"specialcase1|specialcase2"} > alert2 # special > probe_success{somelabel =~"specialcase1|specialcase2"} > .. which is obviously badly maintainable and ugly and won't scale.. > > i've seen this, > https://www.robustperception.io/using-time-series-as-alert-thresholds/.. > but it looks a bit, well, hard to maintain.. > > > > so i got this idea, what would happen if i did this: > > in prometheus rules : > alert1 # handles the special case > ie. probe_success{somelabel="XYZ"} > labels: > someswitch: true > > alert 2 #handles the general case > probe_success{} > > and in alert manager: > define an inhibit rule which mutes the general alert, if there is also an > special case one, based on the someswitch label, would that work? > > any help/pointers/comments greatly appreciated, > cheers, > mario > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/5fe28eec-a91c-4619-af41-128966ad08d9n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/5fe28eec-a91c-4619-af41-128966ad08d9n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5zLPC9zmUenJyiEGH%2B2tLd9f-PNRctnC3pOGagmLgFJUA%40mail.gmail.com.
