Hi,
I guess the nature of the alert is such that the various notification
throttling intervals in the Alertmanager ("repeat_interval" etc.) are not
sufficient for what you want?
Inhibit rules can only refer to source alerts that are currently firing,
and while you could *theoretically* write something like this:
$MY_ALERTING_EXPRESSION
unless ignoring(alertstate, alertname, $OTHER_ALERTING_RULE_LABELS)
present_over_time(ALERTS{alertstate="firing",alertname="$MY_ALERT_NAME"}[24h])
...the problem with that is that it would really only generate the alert on
a single rule evaluation cycle and then immediately resolve it (because in
the second evaluation it sees that the alert already fired in the last 24h,
so the expression returns nothing anymore and the alert resolves). That
will not yield anything stable for you, and you might even miss the alert
completely in case the communication between Prometheus and Alertmanager is
interrupted exactly at that initial time. So it could be interesting to
learn more about what your actual use case is to understand the underlying
requirements for this better.
Caveat: Prometheus 2.42.2 (
https://github.com/prometheus/prometheus/releases/tag/v2.42.0) introduced a
"keep_firing_for" field for alerting rules that could be used to keep the
alert firing for X amount of minutes or hours after that single first
firing cycle. Still, it would auto-resolve after that time window, even if
no underlying problem has been addressed/fixed.
Regards,
Julius
On Wed, Mar 15, 2023 at 4:08 PM Atanas Filyanov <[email protected]>
wrote:
> Hi,
>
> I am trying to have an alert that could only be fired once per 24 hours.
> That mean if the alert fires now, then resolves after, e.g., 10 mins and
> after another 10 gets into a firing state again, it won't send the second
> alert as it already fired in the last 24 hours.
>
> I thought to use inhibit_rules and ALERTS_FOR_STATE[24h] but I do not
> think these can be used together.
>
> Does anyone have an idea of how this could be achieved?
>
> Thanks,
> Atanas
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/e4443270-ede8-4eee-a993-ec42295bed86n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/e4443270-ede8-4eee-a993-ec42295bed86n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Julius Volz
PromLabs - promlabs.com
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAObpH5wiN%3DRN%3DGVBx6dffLxMzkuR_kYn%3DV36%3DvwBTmxFeoua3g%40mail.gmail.com.
