For context and maybe upcomming errors here is my prom.yml so far:
# my global config
global:
scrape_interval: 1m # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 30s # Evaluate rules every 15 seconds. The default
is every 1 minute.
scrape_timeout: 30s
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
# Load rules once and periodically evaluate them according to the global
'evaluation_interval'.
rule_files:
- rules.yml
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries
scraped from this config.
- job_name: 'Windows-Exporter Host-Server'
file_sd_configs:
- files:
- C:\Prometheus\targets\hosts.yml
relabel_configs:
# When __address__ is of the form "name address", extract
# name to "instance" label and address to "__address__"
- source_labels: [__address__]
regex: '(.+) (.+)'
target_label: instance
replacement: '${1}'
- source_labels: [__address__]
regex: '(.+) (.+)'
target_label: __address__
replacement: '${2}'
[...]
- job_name: 'Ping (ICMP)'
metrics_path: /probe
params:
module: [icmp_ttl5]
file_sd_configs:
- files:
- C:\Prometheus\targets\hosts.yml
- C:\Prometheus\targets\netcomponents.yml
- C:\Prometheus\targets\vmhost-05.yml
- C:\Prometheus\targets\vmhost-06.yml
- C:\Prometheus\targets\vmhost-08.yml
- C:\Prometheus\targets\vmhost-09.yml
- C:\Prometheus\targets\vmhost-10.yml
- C:\Prometheus\targets\vmopheo-01.yml
- C:\Prometheus\targets\vmopheo-02.yml
- C:\Prometheus\targets\vmopheo-04.yml
- C:\Prometheus\targets\vmopheo-06.yml
- C:\Prometheus\targets\vmopheo-07.yml
- C:\Prometheus\targets\vmopheo-08.yml
relabel_configs:
- source_labels: [__address__]
regex: '([^ ]+)' # name or address only
target_label: instance
- source_labels: [__address__]
regex: '([^ ]+)' # name or address only
target_label: __param_target
- source_labels: [__address__]
regex: '(.+) (.+)' # name address
target_label: instance
replacement: '${1}'
- source_labels: [__address__]
regex: '(.+) (.+)' # name address
target_label: __param_target
replacement: '${2}'
- source_labels: [module]
target_label: __param_module
- target_label: __address__
replacement: 127.0.0.1:9115 # Blackbox exporter
Kolja Krückmann schrieb am Freitag, 28. April 2023 um 14:10:00 UTC+2:
> Hi Brian, thanks for your quick response.
>
> Actually I just pasted the relabeling you attached and it immediately
> worked :)
>
> Brian Candler schrieb am Freitag, 28. April 2023 um 13:37:33 UTC+2:
>
>> Do you want to show what you've done so far?
>>
>> You can read these:
>>
>>
>> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-blackbox-exporter.html
>>
>> # simple config
>>
>> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-relabeling.html
>>
>> # node_exporter with relabelling
>>
>> If you combine the two, you could end up with something like this:
>>
>> - job_name: blackbox
>> file_sd_configs:
>> - files:
>> - /etc/prometheus/blackbox.d/*.yml
>> metrics_path: /probe
>> relabel_configs:
>> - source_labels: [__address__]
>> regex: '([^ ]+)' # name or address only
>> target_label: instance
>> - source_labels: [__address__]
>> regex: '([^ ]+)' # name or address only
>> target_label: __param_target
>> - source_labels: [__address__]
>> regex: '(.+) (.+)' # name address
>>
>> target_label: instance
>> replacement: '${1}'
>> - source_labels: [__address__]
>> regex: '(.+) (.+)' # name address
>> target_label: __param_target
>> replacement: '${2}'
>> - source_labels: [module]
>> target_label: __param_module
>> - target_label: __address__
>> replacement: 127.0.0.1:9115 # Blackbox exporter
>>
>> Where /etc/prometheus/blackbox.d/ping.yml contains, for example:
>>
>> - labels:
>> module: icmp
>> targets:
>> - google-pri 8.8.8.8
>> - google-sec 8.8.4.4
>>
>> Then you should get metrics like this:
>>
>> probe_success{job="blackbox",instance="google-pri",module="icmp"} 1
>>
>> On Friday, 28 April 2023 at 09:49:21 UTC+1 Kolja Krückmann wrote:
>>
>>> Hi y'all, now i got my metrics running (Thanks Brian)
>>>
>>> For the next step I want to use the Blackbox_exporter with the same
>>> targets (where the target.yml is build like "- targetname targetIP")
>>> I want to have the same labeling as in the question above where each
>>> target has a name lable but I'm not quite getting there. Either the
>>> Endpoint URL is missing the target IP or the label is missing the name of
>>> the endpoint.
>>>
>>> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:38:59 UTC+2:
>>>
>>>> nevermind - sorry for the question, should have just googled learn
>>>> regular expression.
>>>> I'm fine for now.
>>>> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:35:21 UTC+2:
>>>>
>>>>> Thanks so much!
>>>>>
>>>>> Can you tell me where I could kinda "learn" regex? I find it very
>>>>> difficult to get known to regex because it's nothing im using on daily
>>>>> basis ^^
>>>>>
>>>>>
>>>>>
>>>>> Brian Candler schrieb am Donnerstag, 27. April 2023 um 17:24:26 UTC+2:
>>>>>
>>>>>> So to be clear, if you want your targets file to look like this:
>>>>>>
>>>>>> - targets:
>>>>>> - SVR-DS01 172.25.X0.XXX:9182
>>>>>> - SRV-DS02 172.21.X1.XXX:9182
>>>>>>
>>>>>> you can do something like this:
>>>>>>
>>>>>> relabel_configs:
>>>>>> # When __address__ is a single item, set the instance
>>>>>> # label to the part without the port
>>>>>> - source_labels: [__address__]
>>>>>> regex: '([^ ]+):[0-9]+'
>>>>>> replacement: '${1}'
>>>>>> target_label: instance
>>>>>>
>>>>>> # When __address__ is space-separated "name address:port",
>>>>>> # put the first part in the "instance" label and leave the
>>>>>> second part
>>>>>> # in "__address__"
>>>>>> - source_labels: [__address__]
>>>>>> regex: '(.+) (.+)'
>>>>>> target_label: instance
>>>>>> replacement: '${1}'
>>>>>> - source_labels: [__address__]
>>>>>> regex: '(.+) (.+)'
>>>>>> target_label: __address__
>>>>>> replacement: '${2}'
>>>>>>
>>>>>> If your target entries *always* consist of two items separated by a
>>>>>> space, then you don't need the first rule. It's only there in case you
>>>>>> have
>>>>>> entries in the old format, i.e.
>>>>>>
>>>>>> - targets:
>>>>>> - 172.25.X0.XXX:9182
>>>>>>
>>>>>> (in which case, the instance is set to "172.25.X0.XXX")
>>>>>>
>>>>>> On Thursday, 27 April 2023 at 16:16:38 UTC+1 Brian Candler wrote:
>>>>>>
>>>>>>> Well, there is a good reason to keep the port out of the "instance"
>>>>>>> label: it's very awkward to make a query to join two different
>>>>>>> different
>>>>>>> metrics collected from two different exporters on the same host, if one
>>>>>>> has
>>>>>>> (say) instance="foo:9100" and another has instance="foo:9104". A
>>>>>>> second
>>>>>>> reason is that it's nicer in dashboards to see "foo" rather than
>>>>>>> "foo:9100"
>>>>>>>
>>>>>>> See https://www.robustperception.io/controlling-the-instance-label
>>>>>>>
>>>>>>> IMO the cleanest way to achieve this is to leave the port out of the
>>>>>>> targets file, copy __address__ to instance, and then append the
>>>>>>> exporter
>>>>>>> port statically to __address__.
>>>>>>>
>>>>>>> However, you could leave the port number in the targets file, and
>>>>>>> strip it out to create the instance label:
>>>>>>>
>>>>>>> - source_labels: [__address__]
>>>>>>> regex: '(.*):[0-9]+'
>>>>>>> target_label: instance
>>>>>>> replacement: '${1}'
>>>>>>>
>>>>>>> This would be useful if the same exporter were running on different
>>>>>>> ports on different hosts (fairly uncommon I think).
>>>>>>>
>>>>>>> Also, you risk a collision of metrics if targets list includes the
>>>>>>> same host twice on two different ports in the same scrape job (which
>>>>>>> should
>>>>>>> be very unlikely).
>>>>>>>
>>>>>>> On Thursday, 27 April 2023 at 15:14:03 UTC+1 Kolja Krückmann wrote:
>>>>>>>
>>>>>>>> Hi Brian,
>>>>>>>>
>>>>>>>> thank you very much for your response!
>>>>>>>> I just wanted to clarify, if it's better to "regex" the port to the
>>>>>>>> end of the address, or if it is equally fine to just add the port in
>>>>>>>> the
>>>>>>>> target.yml to each target? (That's what im currently doing) Is it just
>>>>>>>> best
>>>>>>>> practice with the regex expression or is there actually a technical
>>>>>>>> reason
>>>>>>>> behind that?
>>>>>>>>
>>>>>>>> Kind regards
>>>>>>>> Kolja
>>>>>>>>
>>>>>>>> Brian Candler schrieb am Dienstag, 18. April 2023 um 13:34:53 UTC+2:
>>>>>>>>
>>>>>>>>> Alternatively, you use rewriting rules.
>>>>>>>>>
>>>>>>>>> - targets:
>>>>>>>>> - SVR-DS01 172.25.X0.XXX
>>>>>>>>> - SRV-DS02 172.21.X1.XXX
>>>>>>>>> ... etc
>>>>>>>>>
>>>>>>>>> and the corresponding relabel_configs in your scrape job could be
>>>>>>>>> something like this:
>>>>>>>>>
>>>>>>>>> relabel_configs:
>>>>>>>>> # When __address__ consists of just a name or IP address,
>>>>>>>>> # copy it to the "instance" label. This keeps the port
>>>>>>>>> # number out of the instance label.
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> regex: '([^ ]+)'
>>>>>>>>> target_label: instance
>>>>>>>>>
>>>>>>>>> # When __address__ is of the form "name address", extract
>>>>>>>>> # name to "instance" label and address to "__address__"
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> regex: '(.+) (.+)'
>>>>>>>>> target_label: instance
>>>>>>>>> replacement: '${1}'
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> regex: '(.+) (.+)'
>>>>>>>>> target_label: __address__
>>>>>>>>> replacement: '${2}'
>>>>>>>>>
>>>>>>>>> # Append port number to __address__ so that scrape gets
>>>>>>>>> # sent to the right port
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> target_label: __address__
>>>>>>>>> replacement: '${1}:9182'
>>>>>>>>>
>>>>>>>>> On Tuesday, 18 April 2023 at 11:14:23 UTC+1 Julius Volz wrote:
>>>>>>>>>
>>>>>>>>>> The labels in the SD file can only be provided for each target
>>>>>>>>>> group (the top-level list item type of the YAML file), so you'd have
>>>>>>>>>> to do
>>>>>>>>>> something like this:
>>>>>>>>>>
>>>>>>>>>> - targets:
>>>>>>>>>> - [...first list of targets...]
>>>>>>>>>> labels:
>>>>>>>>>> instance: "A"
>>>>>>>>>> - targets:
>>>>>>>>>> - [...second list of targets...]
>>>>>>>>>> labels:
>>>>>>>>>> instance: "B"
>>>>>>>>>> - targets:
>>>>>>>>>> - [...third list of targets...]
>>>>>>>>>> labels:
>>>>>>>>>> instance: "C"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 18, 2023 at 11:20 AM Kolja Krückmann <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi y'all I'm trying to get a label for each target in my
>>>>>>>>>>> file_sd_config
>>>>>>>>>>>
>>>>>>>>>>> my prom.yml is:
>>>>>>>>>>> (redacted)
>>>>>>>>>>>
>>>>>>>>>>> # my global config
>>>>>>>>>>> global:
>>>>>>>>>>> scrape_interval: 1m # Set the scrape interval to every 15
>>>>>>>>>>> seconds. Default is every 1 minute.
>>>>>>>>>>> evaluation_interval: 30s # Evaluate rules every 15 seconds.
>>>>>>>>>>> The default is every 1 minute.
>>>>>>>>>>> scrape_timeout: 30s
>>>>>>>>>>>
>>>>>>>>>>> scrape_configs:
>>>>>>>>>>> # The job name is added as a label `job=<job_name>` to any
>>>>>>>>>>> timeseries scraped from this config.
>>>>>>>>>>> - job_name: 'node'
>>>>>>>>>>> file_sd_configs:
>>>>>>>>>>> - files:
>>>>>>>>>>> -
>>>>>>>>>>> C:/Prometheus/prometheus-2.41.0.windows-amd64/target_cluster_b.yml
>>>>>>>>>>>
>>>>>>>>>>> my target_cluster_b.yml:
>>>>>>>>>>> - targets:
>>>>>>>>>>> - 172.25.X0.XXX:9182
>>>>>>>>>>> labels:
>>>>>>>>>>> instance: "SVR-DS01"
>>>>>>>>>>> - 172.21.X1.XXX:9182
>>>>>>>>>>> - 172.25.X2.XXX:9182
>>>>>>>>>>> - 172.25.X3.XXX:9182
>>>>>>>>>>>
>>>>>>>>>>> as seen in the target.yml I want to try to get a label to each
>>>>>>>>>>> target - to see in the prom dashboard which ip is which targetname
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>> Kind regards Kolja
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "Prometheus Users" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to [email protected].
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com
>>>>>>>>>>>
>>>>>>>>>>> <https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Julius Volz
>>>>>>>>>> PromLabs - promlabs.com
>>>>>>>>>>
>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7d53a32c-7ee8-4390-9251-f9e15141bcb6n%40googlegroups.com.
