Inside a scrape job, either you are using static_configs: ...
or file_sd_configs: ... or kubernetes_sd_configs: ... (or others). These are all forms of service discovery. The result of all of these is the same: a list of targets to scrape as __address__, and other labels to associate with those targets. I will say this for the final time: if it works with static_configs but doesn't work with kubernetes_sd_configs, it's because you're generating a different __address__ and/or labels. And I've already told you twice how to view the list of targets and labels which the scrape job is using. On Tuesday, 21 March 2023 at 04:53:50 UTC dikshant rai wrote: > Hi Brian, > I hope there is some confusion for static config, we are not using any SD, > instead we are adding targets manually that is exposed to internet and that > is working fine but when we want to monitor pods or services using > sd_config, we are able to list out those with their name along with port > number assigned. The only issue is that they are down. > > On Tuesday, 21 March 2023 at 00:51:10 UTC+5:30 Brian Candler wrote: > >> If it's the same blackbox exporter you're talking to, and it is returning >> "success" to one probe request (originating from your static_sd_configs) >> but "fail" to another probe request (from your kubernetes_sd_configs), then >> ISTM that the requests that you are sending to blackbox_exporter are >> different in those two cases. >> >> On Monday, 20 March 2023 at 17:06:29 UTC dikshant rai wrote: >> >>> Hi Brian, >>> Thanks for responding. I am using grafana-agent because we are using >>> mimir on top of prometheus. so at first, I thought the same so I went to >>> grafanaa-agent pod using exec command and tried to connect other running >>> services using telnet command. For ex. telnet >>> develop.static-service.dev.svc.cluster.local:8080, it throws 400 error and >>> the same error we are getting from sd-config jobs. Either we are missing >>> something or there may be one more reason that we are using istio for >>> communication that is listening on 15020 port but we tried to connect using >>> that port also but it throws error. >>> >>> On Sunday, 19 March 2023 at 19:45:13 UTC+5:30 Brian Candler wrote: >>> >>>> What you need to do, as I suggested before, is go to the PromQL web >>>> interface and look at Status > Targets to see exactly the scrape URL and >>>> its query parameters, e.g. >>>> >>>> [image: img1.png] >>>> >>>> Then you'll see what's different between the static_configs job and the >>>> kubernetes_sd_configs job, in terms of the generated scrape URL and >>>> params. (And the total number of endpoints being probed, of course). >>>> >>>> Looking at your config, my guess is that the __address__ that you're >>>> getting from kubernetes SD is either a plain address or an address:port >>>> pair, but the target for your http_2xx module requires a complete URL like >>>> http://address:port or https://address:port (maybe with an additional path >>>> too). You'll be able to see if the http:// is missing under Status > >>>> Targets. >>>> >>>> If that's true, then you need to modify your rewriting rules to build >>>> the target appropriately, for example: >>>> >>>> - source_labels: [__address__] >>>> target_label: __param_target >>>> replacement: "http://${1}"; >>>> >>>> That's just a starting point. You need to work on it as appropriate >>>> until your kubernetes SD config gives "target" params which match your >>>> static SD config. >>>> >>>> On Sunday, 19 March 2023 at 13:35:43 UTC dikshant rai wrote: >>>> >>>>> Hi Brian, >>>>> Thanks for the response. >>>>> I have configured static and sd both as different job and when I am >>>>> hitting SD job, I am getting targets as all the services inside the >>>>> cluster >>>>> but they are throwing 400 or 404 error but when I check the logs, the >>>>> status is 200. >>>>> I am attaching job for your reference. Let me know if I am missing >>>>> something. I also tried to check connection by going into >>>>> blackbox-exporter >>>>> pod but it is still throwing same error. I have put all under same >>>>> namespace to avoid any communication failure. >>>>> - job_name: "blackbox-kubernetes-services" >>>>> metrics_path: /probe >>>>> params: >>>>> module: [http_2xx] >>>>> kubernetes_sd_configs: >>>>> - role: service >>>>> relabel_configs: >>>>> # Example relabel to probe only some services that have " >>>>> example.io/should_be_probed = true" annotation >>>>> # - source_labels: >>>>> [__meta_kubernetes_service_annotation_example_io_should_be_probed] >>>>> # action: keep >>>>> # regex: true >>>>> - source_labels: [__address__] >>>>> target_label: __param_target >>>>> - target_label: __address__ >>>>> replacement: prometheus-blackbox-exporter:9115 >>>>> - source_labels: [__param_target] >>>>> target_label: instance >>>>> - action: labelmap >>>>> regex: __meta_kubernetes_service_label_(.+) >>>>> - source_labels: [__meta_kubernetes_namespace] >>>>> target_label: kubernetes_namespace >>>>> - source_labels: [__meta_kubernetes_service_name] >>>>> target_label: kubernetes_service_name >>>>> >>>>> On Friday, 17 March 2023 at 21:04:29 UTC+5:30 Brian Candler wrote: >>>>> >>>>>> > When I am using static config then it shows fine but when I am >>>>>> using service discovery for status of service or pod some are coming up >>>>>> some are down >>>>>> >>>>>> Then clearly you're not scraping/probing the same endpoints as you >>>>>> did with the static config. >>>>>> >>>>>> The easiest way to check is to go to the Prometheus web interface and >>>>>> look at the "Status > Targets" page and see what *actual* targets you're >>>>>> probing: first with the static config, and then secondly with the SD >>>>>> config >>>>>> (actually you can do both at the same time, but configure them as >>>>>> separate >>>>>> jobs so you can distinguish them by the "job" label). The first column >>>>>> shows the URL being scraped and the URL params being passed ("module" >>>>>> and >>>>>> "target" are the ones that need to be set for blackbox_exporter, of >>>>>> course). >>>>>> >>>>>> If you can show your static config, and the service discovery config, >>>>>> the problem might be clearer. My guess is the SD discovery isn't >>>>>> returning >>>>>> data in the form that you expect, or that your rewrite rules aren't >>>>>> working >>>>>> in the way that you expect. >>>>>> >>>>>> On Friday, 17 March 2023 at 15:23:28 UTC dikshant rai wrote: >>>>>> >>>>>>> Hi, >>>>>>> I tried to configure blackbox exporter for checking api status with >>>>>>> prometheus on kubernetes. When I am using static config then it shows >>>>>>> fine >>>>>>> but when I am using service discovery for status of service or pod some >>>>>>> are >>>>>>> coming up some are down. I used telnet to check connectivity as well >>>>>>> that >>>>>>> is fine but still don't know where it is failing. >>>>>>> Please help. >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/77f2e3d7-ad17-4d2e-8993-613a1454c580n%40googlegroups.com.
