On Mon, Sep 4, 2023 at 5:00 PM Brian Candler <[email protected]> wrote:
> On Monday, 4 September 2023 at 15:49:25 UTC+1 Анастасия Зель wrote: > > Hello, we use HA prometheus with two servers. > > You mean, two Prometheus servers with the same config, both scraping the > same targets? > > > > The problem is we get different metrics in dashboards from this two > servers. > > Small differences are to be expected. That's because the two servers > won't be scraping the targets at the same points in time. If you see more > significant differences, then please provide some examples. > > > > And we also scrape metrics from k8s, and some pods are not scraping > because of error context deadline exceeded > > That basically means "scrape timed out". The scrape hadn't completed > within the "scrape_timeout:" value that you've set. You'll need to look at > your individual exporters and the failing scrape URLs: either the target is > not reachable at all (e.g. firewalling or network configuration issue), or > the target is taking too long to respond. > > > Its differents pods on each server. In prometheus logs we dont see any of > errors. > > Where *do* you see the "context deadline exceeded" errors then? > Usually on the `/targets` page. Prometheus does not log scrape errors by default. I would love this to be a configuration option, or even better, a per-job `scrape_configs` option. > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/34cf1354-9e58-4517-8c3d-3301d4fc0236n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/34cf1354-9e58-4517-8c3d-3301d4fc0236n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrsM%3DDjSu2Mjvkmhzo%3D5XNJbmNvDFPN3fScuVRBOkzs%3Dg%40mail.gmail.com.
