If it's the same blackbox exporter you're talking to, and it is returning "success" to one probe request (originating from your static_sd_configs) but "fail" to another probe request (from your kubernetes_sd_configs), then ISTM that the requests that you are sending to blackbox_exporter are different in those two cases.
On Monday, 20 March 2023 at 17:06:29 UTC dikshant rai wrote: > Hi Brian, > Thanks for responding. I am using grafana-agent because we are using mimir > on top of prometheus. so at first, I thought the same so I went to > grafanaa-agent pod using exec command and tried to connect other running > services using telnet command. For ex. telnet > develop.static-service.dev.svc.cluster.local:8080, it throws 400 error and > the same error we are getting from sd-config jobs. Either we are missing > something or there may be one more reason that we are using istio for > communication that is listening on 15020 port but we tried to connect using > that port also but it throws error. > > On Sunday, 19 March 2023 at 19:45:13 UTC+5:30 Brian Candler wrote: > >> What you need to do, as I suggested before, is go to the PromQL web >> interface and look at Status > Targets to see exactly the scrape URL and >> its query parameters, e.g. >> >> [image: img1.png] >> >> Then you'll see what's different between the static_configs job and the >> kubernetes_sd_configs job, in terms of the generated scrape URL and >> params. (And the total number of endpoints being probed, of course). >> >> Looking at your config, my guess is that the __address__ that you're >> getting from kubernetes SD is either a plain address or an address:port >> pair, but the target for your http_2xx module requires a complete URL like >> http://address:port or https://address:port (maybe with an additional path >> too). You'll be able to see if the http:// is missing under Status > >> Targets. >> >> If that's true, then you need to modify your rewriting rules to build the >> target appropriately, for example: >> >> - source_labels: [__address__] >> target_label: __param_target >> replacement: "http://${1}"; >> >> That's just a starting point. You need to work on it as appropriate until >> your kubernetes SD config gives "target" params which match your static SD >> config. >> >> On Sunday, 19 March 2023 at 13:35:43 UTC dikshant rai wrote: >> >>> Hi Brian, >>> Thanks for the response. >>> I have configured static and sd both as different job and when I am >>> hitting SD job, I am getting targets as all the services inside the cluster >>> but they are throwing 400 or 404 error but when I check the logs, the >>> status is 200. >>> I am attaching job for your reference. Let me know if I am missing >>> something. I also tried to check connection by going into blackbox-exporter >>> pod but it is still throwing same error. I have put all under same >>> namespace to avoid any communication failure. >>> - job_name: "blackbox-kubernetes-services" >>> metrics_path: /probe >>> params: >>> module: [http_2xx] >>> kubernetes_sd_configs: >>> - role: service >>> relabel_configs: >>> # Example relabel to probe only some services that have " >>> example.io/should_be_probed = true" annotation >>> # - source_labels: >>> [__meta_kubernetes_service_annotation_example_io_should_be_probed] >>> # action: keep >>> # regex: true >>> - source_labels: [__address__] >>> target_label: __param_target >>> - target_label: __address__ >>> replacement: prometheus-blackbox-exporter:9115 >>> - source_labels: [__param_target] >>> target_label: instance >>> - action: labelmap >>> regex: __meta_kubernetes_service_label_(.+) >>> - source_labels: [__meta_kubernetes_namespace] >>> target_label: kubernetes_namespace >>> - source_labels: [__meta_kubernetes_service_name] >>> target_label: kubernetes_service_name >>> >>> On Friday, 17 March 2023 at 21:04:29 UTC+5:30 Brian Candler wrote: >>> >>>> > When I am using static config then it shows fine but when I am using >>>> service discovery for status of service or pod some are coming up some are >>>> down >>>> >>>> Then clearly you're not scraping/probing the same endpoints as you did >>>> with the static config. >>>> >>>> The easiest way to check is to go to the Prometheus web interface and >>>> look at the "Status > Targets" page and see what *actual* targets you're >>>> probing: first with the static config, and then secondly with the SD >>>> config >>>> (actually you can do both at the same time, but configure them as separate >>>> jobs so you can distinguish them by the "job" label). The first column >>>> shows the URL being scraped and the URL params being passed ("module" and >>>> "target" are the ones that need to be set for blackbox_exporter, of >>>> course). >>>> >>>> If you can show your static config, and the service discovery config, >>>> the problem might be clearer. My guess is the SD discovery isn't >>>> returning >>>> data in the form that you expect, or that your rewrite rules aren't >>>> working >>>> in the way that you expect. >>>> >>>> On Friday, 17 March 2023 at 15:23:28 UTC dikshant rai wrote: >>>> >>>>> Hi, >>>>> I tried to configure blackbox exporter for checking api status with >>>>> prometheus on kubernetes. When I am using static config then it shows >>>>> fine >>>>> but when I am using service discovery for status of service or pod some >>>>> are >>>>> coming up some are down. I used telnet to check connectivity as well that >>>>> is fine but still don't know where it is failing. >>>>> Please help. >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2523defc-d2c8-4288-b3e7-b1c7f4d089adn%40googlegroups.com.
