Enter your alerting expression exactly as-is into the PromQL query browser in the prometheus web interface: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
If you see any gaps between 8:00AM and 8:05AM then this is where the alert was resolved. Or you can look at the negated expression to see more clearly if there are values which exceed 10% between 8:00 and 8:05: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 >= 10 The important point here is, if the alerting expression doesn't return a value for even a single rule evaluation interval, then the alert will be immediately resolved. However, Prometheus 2.42 <https://github.com/prometheus/prometheus/releases/tag/v2.42.0> has added a new feature which will should help you: - [FEATURE] Add 'keep_firing_for' field to alerting rules. #11827 <https://github.com/prometheus/prometheus/pull/11827> For example: if you set "keep_firing_for: 5m" then your alert won't resolve until the alerting value has been absent for 5 minutes continuously (that is, on *every* evaluation cycle over 5 minutes). This is complementary to the existing feature "for: 5m" which means that the alert won't fire until the alerting value has been present for 5 minutes continuously. On Sunday, 2 April 2023 at 09:54:57 UTC+1 nihra l wrote: > Hi, > > we are using following expression for node memory alert. > > node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10 > > we are getting alert when we have less than 10% memory available. But when > the value changes in next evaluation cycle which is still less than 10%, it > is resolving previous alert and firing fresh alert instead of following > repeat inverval. > Configuration details: > For condition in alert rule: 1min > group_wait: 30s, group_interval: 5m, repeat_interval: 24h > > Example: > 8:00AM memory available is 9.5% alert1 triggered > 8:05 AM memory available is 9.8% instead of following repeat cycle it is > resolving alert1 and firing fresh alert. > > Thanks > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3a2e20ed-f98d-4481-9fc1-f7af691e5cf8n%40googlegroups.com.
