To expand on that slightly: - if the "up" metric were fluctuating between 0 and 1, then that would be a problem with with prometheus talking to consul_exporter - but what you're seeing is the "consul_up" metric fluctuating between 0 and 1. "consul_up" is a metric returned by console_exporter. This means that consul_exporter is working just fine, and prometheus is also able to talk to it just fine; but the testing done by console_exporter is giving 0 and 1 results.
So you need to focus your attention on consul_exporter itself, and how it communicates with the target consul cluster. Looking at logs from the consul_exporter container would be a good start. Looking at tcpdump of traffic between the consul_exporter container and the consul cluster it's talking to would be another thing to try. We don't know how consul_exporter is locating the consul cluster, but if it's using DNS, then that's something else to check too. On Thursday, 22 December 2022 at 10:44:46 UTC [email protected] wrote: > This means your consul_exporter is having problems talking to your consul > server. Only you can debug this communication problem. > > On Thu, Dec 22, 2022 at 11:38 AM Nikash Narula <[email protected]> wrote: > >> We have an alert set to trigger when consul_up is failing to be above 90% >> in past 5 min: (avg_over_time(consul_up{job="consul-exporter"}[5m]) * 100) >> < 90. >> We see continuous fluctuations between consul_up being 1 (up) and 0 >> (down). Thus, our alert is getting triggered often even when all Consul >> health checks are spotless (we had a Consul support engineer verify this). >> *Environment *(Prod): >> >> - >> >> System information: Linux 5.8.0-1041-aws x86_64 >> - >> >> consul_exporter version: 0.7.1 >> - >> >> Consul version: Consul v1.8.0 >> - >> >> Prometheus version: prometheus, version 2.28.1 (branch: HEAD, >> revision: b0944590a1c9a6b35dc5a696869f75f422b107a1) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/26d5df3a-0635-4c69-ab47-303ee7327dd2n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/26d5df3a-0635-4c69-ab47-303ee7327dd2n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/00ecbb95-8c00-4456-b8aa-e6421818385cn%40googlegroups.com.
