This indeed should not happen. Maybe can throw more than the address
label into the hash and also include the port?
On 23 Jan 02:36, Vikas Budhvant wrote:
>
>
> We have 4 prom-agents having scrape config as below
> regrex is different for each agent(0, 1, 2, and 3)
>
> ` - job_name: 'kube-pods'
> kubernetes_sd_configs: - role: pod relabel_configs: - source_labels:
> [__address__] modulus: 4 target_label: __tmp_hash action: hashmod -
> source_labels: [__tmp_hash] regex: ^0$ action: keep - source_labels:
> [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex:
> true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
> action: replace target_label: __metrics_path__ regex: (.+) - source_labels:
> [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action:
> replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label:
> __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) -
> source_labels: [__meta_kubernetes_namespace] action: replace target_label:
> kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action:
> replace target_label: kubernetes_pod_name`
>
> our expectations was same metrics should not be scraped by 2 prometheus
> agent, which is not working and we are seeing same metrics being scraped by
> more than 1 agents.
> I verified this using query
> count(count({job="kube-pods"}) by (prometheus_agent,kubernetes_pod_name,
> *name*,instance)) by (kubernetes_pod_name,*name*,instance) > 1
>
> there are some metrics with exactly 1 count as well which means issue is
> not consistent for all metrics. not sure if its something to do with pods
> or their config or somethings else.
>
> Interestingly we don't see such duplicated for below scrape job
> ` - job_name: 'hlo-pods'
> kubernetes_sd_configs: - role: pod namespaces: names: - hlo -
> relabel_configs: - source_labels: [__address__] modulus: 4 target_label:
> __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ^2$ action:
> keep - action: labelmap regex: __meta_kubernetes_pod_label_(.+) -
> source_labels: [__meta_kubernetes_namespace] action: replace target_label:
> kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action:
> replace target_label: kubernetes_pod_name - source_labels: [
> __meta_kubernetes_pod_container_name ] action: replace target_label:
> kubernetes_container_name `
>
> Any clue what should I check? what could be possible cause?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/4b460268-5bd0-41af-88ef-d42ba7315785n%40googlegroups.com.
--
Julien Pivotto
@roidelapluie
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/Y8%2B1LQgW5GjhHLeg%40nixos.
