Hi Murali, So I guess both of those metrics are coming from the same kube-state-metrics target? While I'm not sure why those two metrics would be behaving differently, I'm not sure you'd want put the "__meta_kubernetes_pod_node_name" service discovery meta label into the "node" target label, because this discovery meta label contains only the node name of the discovered kube-state-metrics target and not of the individual specific pods that the metric from kube-state-metrics is about. So I would actually expect both metrics to only show one single "node" value ever after your relabeling rule (or as many values as you have k-s-m instances).
One thing I did notice on https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md is that "kube_pod_info" already has a "node" label to begin with (coming from kube-state-metrics), while "kube_pod_container_info" does not. But your relabeling rule sets a "node" label on the target itself, which will conflict during the scrape with the "node" label on the "kube_pod_info" metric. That should cause the original "node" label on "kube_pod_info" to be moved to an "exported_node" label to resolve the label collision. That is, unless you are setting the "honor_labels: true" option in the scrape config (see https://github.com/prometheus/prometheus/blob/64842f137e1ae6e041e12a2707d99d6da4ba885b/docs/configuration/configuration.md?plain=1#L135-L155), which would 100% explain why you'd see the behavior you're describing, since that would keep the original label from kube-state-metrics and throw away the conflicting target label. Anyway, for diagnosing discovery and relabeling issues in general, it's always a good idea to check out the /service-discovery page on your Prometheus server. There you can inspect the labels of any target both before and after relabeling ("Discovered Labels" column vs. "Target Labels" column). You can also use the relabeling simulator / explainer tool at https://relabeler.promlabs.com/ (shameless plug) to enter your relabeling rules and the target labels before relabeling (both in YAML format) to see what effect the rules have on the labels. Regards, Julius On Thu, Jan 19, 2023 at 2:36 PM Murali Krishna Kanagala < [email protected]> wrote: > Hi All, > > While working on some alert rules I have noticed that the metric* > kube_pod_container_Info* has wrong values under the node label. This > label uses the below scrape/relabel config which looks right. Whereas the > other metric from the same job *kube_pod_info *has the correct node label > value. The node value in the *kube_pod_container_info *is giving the name > of the node where Prometheus is running (only returns 1 node value for all > the pods in the cluster). > > - source_labels: [__meta_kubernetes_pod_node_name] > separator: ; > regex: (.*) > target_label: node > replacement: $1 > action: replace > > Another observation is that a bunch of metrics under this job > *kubernetes-service-endpoints* (like > *kube_pod_container_status_restarts_total) > *are also reporting only one node value for all the pods on the cluster. > > This is deployed using helm chart version 19.3.1 with the default scrape > config. > > Any suggestions/ recommendations? > -- > Regards, > Murali Krishna Kanagala > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5z1-9qTiK6wXYBz2FXs35pOLhH_PMif_MRh392JVWyu0g%40mail.gmail.com.
