Thanks for the clarification Julius! I will read through the links you have shared.
On Fri, Jan 20, 2023, 5:16 AM Julius Volz <[email protected]> wrote: > Hi Murali, > > So I guess both of those metrics are coming from the same > kube-state-metrics target? While I'm not sure why those two metrics would > be behaving differently, I'm not sure you'd want put the > "__meta_kubernetes_pod_node_name" service discovery meta label into the > "node" target label, because this discovery meta label contains only the > node name of the discovered kube-state-metrics target and not of the > individual specific pods that the metric from kube-state-metrics is about. > So I would actually expect both metrics to only show one single "node" > value ever after your relabeling rule (or as many values as you have k-s-m > instances). > > One thing I did notice on > https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md > is that "kube_pod_info" already has a "node" label to begin with (coming > from kube-state-metrics), while "kube_pod_container_info" does not. But > your relabeling rule sets a "node" label on the target itself, which will > conflict during the scrape with the "node" label on the "kube_pod_info" > metric. That should cause the original "node" label on "kube_pod_info" > to be moved to an "exported_node" label to resolve the label collision. > That is, unless you are setting the "honor_labels: true" option in the > scrape config (see > https://github.com/prometheus/prometheus/blob/64842f137e1ae6e041e12a2707d99d6da4ba885b/docs/configuration/configuration.md?plain=1#L135-L155), > which would 100% explain why you'd see the behavior you're describing, > since that would keep the original label from kube-state-metrics and throw > away the conflicting target label. > > Anyway, for diagnosing discovery and relabeling issues in general, it's > always a good idea to check out the /service-discovery page on your > Prometheus server. There you can inspect the labels of any target both > before and after relabeling ("Discovered Labels" column vs. "Target Labels" > column). You can also use the relabeling simulator / explainer tool at > https://relabeler.promlabs.com/ (shameless plug) to enter your relabeling > rules and the target labels before relabeling (both in YAML format) to see > what effect the rules have on the labels. > > Regards, > Julius > > On Thu, Jan 19, 2023 at 2:36 PM Murali Krishna Kanagala < > [email protected]> wrote: > >> Hi All, >> >> While working on some alert rules I have noticed that the metric* >> kube_pod_container_Info* has wrong values under the node label. This >> label uses the below scrape/relabel config which looks right. Whereas the >> other metric from the same job *kube_pod_info *has the correct node >> label value. The node value in the *kube_pod_container_info *is giving >> the name of the node where Prometheus is running (only returns 1 node value >> for all the pods in the cluster). >> >> - source_labels: [__meta_kubernetes_pod_node_name] >> separator: ; >> regex: (.*) >> target_label: node >> replacement: $1 >> action: replace >> >> Another observation is that a bunch of metrics under this job >> *kubernetes-service-endpoints* (like >> *kube_pod_container_status_restarts_total) >> *are also reporting only one node value for all the pods on the cluster. >> >> This is deployed using helm chart version 19.3.1 with the default scrape >> config. >> >> Any suggestions/ recommendations? >> -- >> Regards, >> Murali Krishna Kanagala >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAKimyZv4MODLT47YSn0dOrN1GO7DbG6Ty%3Dicaa%3DUuTv0uuXXng%40mail.gmail.com.
