Hi, We have a server that has a high cardinality of metrics, mainly due to a label that is tagged on the majority of the metrics. However, most of our dashboards/queries don't use this label, and just use aggregate queries. There are specific scenarios where we would need to debug and sort based on the label, but this doesn't happen that often.
Is it a common design pattern to separate out two metrics endpoints, one for aggregates, one for labelled metrics, with different scrape intervals? This way we could limit the impact of the high cardinality time series, by scraping the labelled metrics less frequently. Couple of follow-up questions: - When a query that uses the aggregate metric comes in, does it matter that the data is potentially duplicated between the two endpoints? How do we ensure that it doesn't try loading all the different time series with the label and then aggregating, and instead directly use the aggregate metric itself? - How could we make sure this new setup is more efficient than the old one? What criteria/metrics would be best (query evaluation time? amount of data ingested?) Thanks in advance -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1f3f41fe-90de-4a05-a04f-b1bb682ebde6n%40googlegroups.com.
