[prometheus-users] job label missing from discoveredLabels (prometheus v2.42.0)

[Vu Nguyen]

Hi all,

We have a test code that reads target metadata info and job label name from 
`discoveredLabels` list. That list is included in the response we get from 
'/api/v1/targets' endpoint.

During the test, we noticed that the response from target endpoint is 
inconsistent: the job label sometimes is missing from `discoveredLabels` 
for a few discovered targets. 

Below output is what I extracted from our deployment: the first target have 
the job label in its discovered labels but missing in the second target, on 
the same config job.

{
  "status" : "success",
  "data" : {
    "activeTargets" : [ {
      "discoveredLabels" : {
        "__meta_kubernetes_pod_phase" : "Running",
        "__meta_kubernetes_pod_ready" : "true",
        "__meta_kubernetes_pod_uid" : 
"a7b4cce2-1be7-4df9-a032-c7a51bb655db",
        "__metrics_path__" : "/metrics",
        "__scheme__" : "http",
        "__scrape_interval__" : "15s",
        "__scrape_timeout__" : "10s",
        "job" : "kubernetes-pods"
      },
      "labels" : {
        "app" : "testapp",
        "job" : "kubernetes-pods",
        "kubernetes_namespace" : "spider1"
      },
      "health" : "down",
      "scrapeInterval" : "15s",
      "scrapeTimeout" : "10s"
    }, {
      "discoveredLabels" : {
         "__meta_kubernetes_pod_phase" : "Running",
        "__meta_kubernetes_pod_ready" : "true",
        "__meta_kubernetes_pod_uid" : 
"85dfeac6-985d-479e-8459-fc20ae8dcec3",
        "__metrics_path__" : "/metrics",
        "__scheme__" : "http",
        "__scrape_interval__" : "15s",
        "__scrape_timeout__" : "10s",
        "app" : "another-testapp"
      },
      "labels" : {
        "app" : "another-testapp",
        "job" : "kubernetes-pods",
        "kubernetes_namespace" : "spider1"
      },
      "scrapePool" : "kubernetes-pods",
      "health" : "down",
      "scrapeInterval" : "15s",
      "scrapeTimeout" : "10s"
    } ]
  }
}

Could you please help us understand why we have this inconsistency? Is that 
correct way to get job level value from `discoveredLabels` set?

Thanks,
Vu

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/eadc7ee9-e501-46b7-8e22-a39ed3672f33n%40googlegroups.com.