Hello, we use HA prometheus with two servers. The problem is we get different metrics in dashboards from this two servers. And we also scrape metrics from k8s, and some pods are not scraping because of error context deadline exceeded Its differents pods on each server. In prometheus logs we dont see any of errors. How is that possible? What we can do for debug this? prometheus, version 2.40.7 (branch: HEAD, revision: ab239ac5d43f6c1068f0d05283a0544576aaecf8) build user: root@afba4a8bd7cc build date: 20221214-08:49:43 go version: go1.19.4 platform: linux/amd64
prometheus config file # This file is managed by ansible. Please don't edit it by hand or your changes would be overwritten. # # http://prometheus.io/docs/operating/configuration/ global: evaluation_interval: 30s scrape_interval: 30s scrape_timeout: 15s external_labels: null rule_files: - /etc/prometheus/rules/*.rules - job_name: 'k8s_pods' scrape_interval: 5m scrape_timeout: 1m kubernetes_sd_configs: - role: pod api_server: https://x.x.x.x:6443 tls_config: insecure_skip_verify: true bearer_token_file: "/etc/prometheus/kubernetes_bearer_token" relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: (.+):(?:\d+);(\d+) replacement: ${1}:${2} target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: kubernetes_pod_node_name -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5562ad53-4827-458d-885b-a206ca19c4a2n%40googlegroups.com.
