I was also thinking of adding a label in the alert, such as:
labels:
severity: warning
mute: |
"
{{ `changes(ALERTS_FOR_STATE{alertname="MYALERTNAME", name="%s"}[24h])`
$labels.name | query | first }}
"
and then only forward to the receiver if the "mute" value is 0. The
expression is not quite right though, I need to fix it and check if it does
the job correctly.
On Tue, Mar 21, 2023 at 1:43 PM Atanas Filyanov <[email protected]>
wrote:
> Hi Julius,
>
> Thanks for the reply and the suggestions. So my use case is as follows - I
> am monitoring the price movements of certains coins, if the price drops,
> e.g., more than 5%, I would fire an alert. But once the alert is fired, I
> don't want it to fire for the next 24 hours. I already use the
> repeat_interval which works to some extent, but the problem with that is
> that as the price of the coin changes, it could rise again and resolve the
> alert but then after a few hours could drop under the threshold again and
> the alert will fire, which I don't want to. I have defined rules for other
> threshold levels, such as 10%, 15%, etc. and want the same behavior for
> these too.
>
> Does it make sense?
>
> I understand what you are saying regarding the problems with the above
> expression. It will fire and resolve after my evaluation interval in the
> next cycle. And not when the value actually rises above the threshold,
> which would be the expected behavior.
>
> Thanks,
> Atanas
>
> On Tue, Mar 21, 2023 at 12:21 PM Julius Volz <[email protected]>
> wrote:
>
>> Hi,
>>
>> I guess the nature of the alert is such that the various notification
>> throttling intervals in the Alertmanager ("repeat_interval" etc.) are not
>> sufficient for what you want?
>>
>> Inhibit rules can only refer to source alerts that are currently firing,
>> and while you could *theoretically* write something like this:
>>
>> $MY_ALERTING_EXPRESSION
>> unless ignoring(alertstate, alertname, $OTHER_ALERTING_RULE_LABELS)
>>
>> present_over_time(ALERTS{alertstate="firing",alertname="$MY_ALERT_NAME"}[24h])
>>
>> ...the problem with that is that it would really only generate the alert
>> on a single rule evaluation cycle and then immediately resolve it (because
>> in the second evaluation it sees that the alert already fired in the last
>> 24h, so the expression returns nothing anymore and the alert resolves).
>> That will not yield anything stable for you, and you might even miss the
>> alert completely in case the communication between Prometheus and
>> Alertmanager is interrupted exactly at that initial time. So it could be
>> interesting to learn more about what your actual use case is to understand
>> the underlying requirements for this better.
>>
>> Caveat: Prometheus 2.42.2 (
>> https://github.com/prometheus/prometheus/releases/tag/v2.42.0)
>> introduced a "keep_firing_for" field for alerting rules that could be used
>> to keep the alert firing for X amount of minutes or hours after that single
>> first firing cycle. Still, it would auto-resolve after that time window,
>> even if no underlying problem has been addressed/fixed.
>>
>> Regards,
>> Julius
>>
>> On Wed, Mar 15, 2023 at 4:08 PM Atanas Filyanov <[email protected]>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to have an alert that could only be fired once per 24 hours.
>>> That mean if the alert fires now, then resolves after, e.g., 10 mins and
>>> after another 10 gets into a firing state again, it won't send the second
>>> alert as it already fired in the last 24 hours.
>>>
>>> I thought to use inhibit_rules and ALERTS_FOR_STATE[24h] but I do not
>>> think these can be used together.
>>>
>>> Does anyone have an idea of how this could be achieved?
>>>
>>> Thanks,
>>> Atanas
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/e4443270-ede8-4eee-a993-ec42295bed86n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/e4443270-ede8-4eee-a993-ec42295bed86n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> --
>> Julius Volz
>> PromLabs - promlabs.com
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAExYgLztTeOggjtqRE5wi%3DnNN1hOsBZ2owjSn6nrFrAS4DjfSg%40mail.gmail.com.
