> How would i go about adding labels to the metrics?
You have this:
static_configs:
- targets:
- api.telemetry.confluent.cloud
This means you are only scraping one endpoint, one time. If you wanted to
add the same labels to every metric received from that endpoint, you would
do this:
static_configs:
- labels:
foo: bar
baz: qux
targets:
- api.telemetry.confluent.cloud
Of course, that's not what you're asking.
The question now is, do the metrics that you get back all carry a label
which identifies the cluster, such as {cluster="lkc-1"}?
If so, then it's a simple case of metric relabelling to add the department
labels corresponding to each cluster ID. Add to the scrape job:
metric_relabel_configs:
- source_labels: [cluster]
regex: lkc-1
target_label: departmentID
replacement: Accounts
- source_labels: [cluster]
regex: lkc-2
target_label: departmentID
replacement: Engineering
# etc
If you don't have such a label, then you will need to scrape the API
endpoint separately, once for each value of resource.kafka.id
The dumb option is multiple scrape jobs:
scrape_configs:
- job_name: Confluent Cloud lkc-1
scrape_interval: 1m
scrape_timeout: 1m
static_configs:
- labels:
department: Accounts
targets:
- api.telemetry.confluent.cloud
scheme: https
basic_auth:
username: <Cloud API Key>
password: <Cloud API Secret>
metrics_path: /v2/metrics/cloud/export
params:
"resource.kafka.id": [lkc-1]
- job_name: Confluent Cloud lkc-2
scrape_interval: 1m
scrape_timeout: 1m
static_configs:
- labels:
department: Engineering
targets:
- api.telemetry.confluent.cloud
scheme: https
basic_auth:
username: <Cloud API Key>
password: <Cloud API Secret>
metrics_path: /v2/metrics/cloud/export
params:
"resource.kafka.id": [lkc-2]
# ... etc
That should work just fine, but is annoyingly verbose and repetitive.
The second option, which I would normally use in this situation, is to set
the query parameter using a __param_XXXX label:
scrape_configs:
- job_name: Confluent Cloud
scrape_interval: 1m
scrape_timeout: 1m
static_configs:
- labels:
department: Accounts
"__param_resource.kafka.id": lkc-1
targets:
- api.telemetry.confluent.cloud
- labels:
department: Engineering
"__param_resource.kafka.id": lkc-2
targets:
- api.telemetry.confluent.cloud
- labels:
department: Special Projects
"__param_resource.kafka.id": lkc-3
targets:
- api.telemetry.confluent.cloud
# etc
scheme: https
basic_auth:
username: <Cloud API Key>
password: <Cloud API Secret>
metrics_path: /v2/metrics/cloud/export
Here, the parameter value is set to a single value each time using the
magic label "__param_<paramname>" instead of using "params: { name: [
list_of_values ] }"
Unfortunately, the problem is that I'm not sure that __param supports
parameter names with dots in them, because dots are technically not valid
in a label name
<https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels>.
You would need to try it to find out if it works, and I wouldn't be
surprised if it were rejected.
Aside:
- You should almost never use "honor_timestamps" so I have removed it in
the examples above. If you do use it, you have to be very sure why, and
understand how it may break things.
- When there are multiple targets like this I would use file_sd_configs
rather than static_configs for this (it's easier to maintain).
The downside to these approaches is that you are now hitting the same API
endpoint N times (each returning 1/Nth of the data). This only matters if
you get charged per API call.
If you still want to fetch the responses in a single API call as you are
now, then you will have to use metric_relabelling, and somehow decide for
each metric that comes back which kafka cluster it came from by examining
the labels - which is the first approach I proposed.
HTH,
Brian.
On Friday, 18 November 2022 at 09:51:41 UTC [email protected] wrote:
> Hello,
>
> I am trying to add labels to metrics fetched from Confluent Cloud.
> We are monitoring some 35 Kafka clusters.
>
> scrape_configs:
> - job_name: Confluent Cloud
> scrape_interval: 1m
> scrape_timeout: 1m
> honor_timestamps: true
> static_configs:
> - targets: - api.telemetry.confluent.cloud
> scheme: https
> basic_auth:
> username: <Cloud API Key>
> password: <Cloud API Secret>
> metrics_path: /v2/metrics/cloud/export
> params:
> "resource.kafka.id":
> - lkc-1
> - lkc-2
> - lkc-3
> - lkc-4
> - lkc-5
> - lkc-6
> - lkc-etc etc
>
>
> Each lkc-xxxx represent a cluster which belongs to a department.
> I would like to add a departmentID to the metrics belonging to to each
> cluster.
> For example lkc-1 and lkc-5 would beong to department "analytics"
>
> How would i go about adding labels to the metrics?
>
> Best regards
> Christian Oelsner
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/a5e81663-f5f2-49c3-bef2-a34cad3a2ab4n%40googlegroups.com.
