I have a service that pushes forecast metrics every 1 minute to Prometheus
using a push gateway and configured an alert rule in Prometheus.
Requirements:
- 5 minutes is the stability filter.
- If forecasts are not published over the last 5 minutes, then trigger a
firing alert.
- If forecasts are published over the last 5 minutes, then resolve the
alert.
The reason we need a stability filter is: sometimes the service is not able
to push metrics due to the push gateway service being down for 1 minute and
the push gateway service being recovered within 2 minutes, so we do not
want to send firing alerts in this scenario.
*Prometheus configurations:*
evaluation_interval: 1m
scrape_interval: 30s
*Alert rule:*
- alert: forecaster expr:
rate(forecasts_published_counter{job=\"metrics_job\",
module_name=\"forecaster\"}[5m]) <= 0 for: 5m
I experimented stability filter using *FOR clause*, it works for firing
alerts but it does not work for resolving alerts.
*The service is not publishing forecasts for over the 5 minutes:*
- The time taken to change the alert state from *inactive* to *pending* is
1m i.e evaluation interval.
- The time taken to change the alert state from *pending* to *firing* is
between *(FOR clause interval)* and *(evaluation_interval +
scrape_interval + FOR clause interval)* i.e 5m and (1m + 30s + 5m = 6m
30s).
*The service publishing forecasts over the 5 minutes:*
- The time taken to change the alert state from firing to inactive
(Resolved) is 1m i.e evaluation interval.
I can change the evaluation interval to 5m but it affects other services.
So I do not want to change it.
*Is there any other way to set a stability filter (5m) in Prometheus for
changing the alert state from firing to inactive(Resolved)?*
Thanks,
Shivakumar Sajjan
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/e62bf7cd-4865-441a-ab88-7c6761885924n%40googlegroups.com.
