Hi Nihra, * Is your rule evaluation interval in Prometheus really 5 minutes, as your examples show? In that case it would explain why the Alertmanager auto-resolves your alerts in between cycles, because the default resolve timeout for alerts received by the Alertmanager is 5 minutes. So Alertmanager expects to receive the same firing alert from Prometheus every <5m in order for it to not auto-resolve. See https://github.com/prometheus/alertmanager/blob/747430cd42e1aad5a5f0a25737e9a398b0ef371e/config/config.go#L621
* If that's not the case, are you 100% sure that no evaluation cycle ever goes above 10% in between? What happens if you manually evaluate your expression at a high resolution, do you see it constantly under 10%? Regards, Julius On Sun, Apr 2, 2023 at 10:54 AM nihra l <[email protected]> wrote: > Hi, > > we are using following expression for node memory alert. > > node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10 > > we are getting alert when we have less than 10% memory available. But when > the value changes in next evaluation cycle which is still less than 10%, it > is resolving previous alert and firing fresh alert instead of following > repeat inverval. > Configuration details: > For condition in alert rule: 1min > group_wait: 30s, group_interval: 5m, repeat_interval: 24h > > Example: > 8:00AM memory available is 9.5% alert1 triggered > 8:05 AM memory available is 9.8% instead of following repeat cycle it is > resolving alert1 and firing fresh alert. > > Thanks > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/933e1f53-6dae-4427-b1b0-9303bec44d4fn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/933e1f53-6dae-4427-b1b0-9303bec44d4fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5x7sy71wEvN%3DCkqaaueWnVYFA1rbViLsjnEAxk98MPETg%40mail.gmail.com.
