On Saturday, 19 November 2022 at 11:43:26 UTC [email protected] wrote:
> A quick query in prometheus for example gives me this:
>
> confluent_kafka_server_retained_bytes{instance="api.telemetry.confluent.cloud:443",
> job="Confluent-Cloud", kafka_id="lkc-0x3v22",
> topic="confluent-kafka-connect-qa.confluent-kafka_configs"}
>
> Does that mean that i have a label simply called kafka_id?
>
Yes indeed. So if you can relate the values of that to the department,
then you can use the simple metric relabelling I showed originally to add
the departmentID label. But you need a separate rewrite rule for each
kafka_id to department mapping - so you'll have to update the config every
time you add a new cluster (which you're already doing to add the new query
params).
There is another approach to consider: you can make a separate set of
static timeseries with the metadata bindings, like this:
kafka_cluster_info{kafka_id="lkc-0x3v22", departmentID="Engineering",
env="production"} 1
kafka_cluster_info{kafka_id="lkc-0x3v25", departmentID="Accounts",
env="test"} 1
...
(A static timeseries can be made using node_exporter textfile_collector, or
a static web page that you scrape)
The "kafka_id" label here has to match the "kafka_id" label values in the
scraped data. Then whenever you do a query on one of the main metrics, you
can do a join to add the extra metadata labels, something like this:
confluent_kafka_server_retained_bytes * on (kafka_id)
group_left(departmentID,env) kafka_cluster_info
Or you can do filtering on the metadata to select only the clusters
belonging to a particular department or for a particular environment, e.g.
confluent_kafka_server_retained_bytes * on (kafka_id)
group_left(departmentID) kafka_cluster_info{env="production"}
For the full details of this approach see:
https://www.robustperception.io/how-to-have-labels-for-machine-roles
https://www.robustperception.io/exposing-the-software-version-to-prometheus
https://www.robustperception.io/left-joins-in-promql
https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches
The tradeoff here is that your queries get more complex whenever you need
the departmentID or environment labels, especially in alerting rules.
Adding the extra labels at scrape time keeps your queries simpler.
You can also combine both approaches: use recording rules with join queries
like those above, to create new metrics with the extra labels.
> I did infact try to wrap my head around using file_sd_configs but could
> not work out how the params part of it, so i gave up on that. It would be
> nice though, since our list of clusters keps growing every week.
>
If you're only scraping the API once (because you have an API limit to
avoid) then a single target with static_configs is fine.
Regards,
Brian.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/1bf921a0-4c0f-4f9b-85f0-8ff2be591478n%40googlegroups.com.
