> I can change the evaluation interval to 5m but it affects other services. So I do not want to change it.
Evaluation intervals can be set at the level of each Rule Group, so you just put your rule into its own Rule Group and you can give it whatever evaluation interval you like. See: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule_group > I experimented stability filter using *FOR clause*, it works for firing alerts but it does not work for resolving alerts. There's a new feature in Prometheus v2.42.0 <https://github.com/prometheus/prometheus/releases/tag/v2.42.0> to give the equivalent behaviour for resolving alerts: - [FEATURE] Add 'keep_firing_for' field to alerting rules. #11827 <https://github.com/prometheus/prometheus/pull/11827> On Wednesday, 15 March 2023 at 07:56:36 UTC shivakumar sajjan wrote: > I have a service that pushes forecast metrics every 1 minute to Prometheus > using a push gateway and configured an alert rule in Prometheus. > > Requirements: > > - 5 minutes is the stability filter. > - If forecasts are not published over the last 5 minutes, then trigger > a firing alert. > - If forecasts are published over the last 5 minutes, then resolve the > alert. > > The reason we need a stability filter is: sometimes the service is not > able to push metrics due to the push gateway service being down for 1 > minute and the push gateway service being recovered within 2 minutes, so we > do not want to send firing alerts in this scenario. > > *Prometheus configurations:* > > evaluation_interval: 1m > > scrape_interval: 30s > > *Alert rule:* > - alert: forecaster expr: > rate(forecasts_published_counter{job=\"metrics_job\", > module_name=\"forecaster\"}[5m]) <= 0 for: 5m > > I experimented stability filter using *FOR clause*, it works for firing > alerts but it does not work for resolving alerts. > > *The service is not publishing forecasts for over the 5 minutes:* > > - The time taken to change the alert state from *inactive* to *pending* is > 1m i.e evaluation interval. > - The time taken to change the alert state from *pending* to *firing* is > between *(FOR clause interval)* and *(evaluation_interval + > scrape_interval + FOR clause interval)* i.e 5m and (1m + 30s + 5m = 6m > 30s). > > *The service publishing forecasts over the 5 minutes:* > > - The time taken to change the alert state from firing to inactive > (Resolved) is 1m i.e evaluation interval. > > I can change the evaluation interval to 5m but it affects other services. > So I do not want to change it. > > *Is there any other way to set a stability filter (5m) in Prometheus for > changing the alert state from firing to inactive(Resolved)?* > > > > Thanks, > > Shivakumar Sajjan > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/881906f6-f6d1-4c48-8b24-ae5b02a16f19n%40googlegroups.com.
