> With the recording rule, we created a new static label called highcardinality=“true” but this creates new time series. When doing remote write to our long term storage we are dropping those time series which has highcardinality=“true” but the original metric doesn’t have this label so its still getting into our remote write system.
Why don't you configure <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write> your remote_write so that it only sends metrics with highcardinality="true"? Use write_relabel_configs with "keep" or "drop" rules. > We are thinking of add a new label as part of metric_relabeling section with highcardinality=“false” and update the label to true using recording rules Again, not sure exactly why you'd want to do that. Changing a label from one value to another also creates a new timeseries, because the bundle of labels is what defines a timeseries, so it's not really any different. But your recording rules *are* generating a new timeseries anyway. I'm also not sure why you are saying that the recorded metrics have a "high" cardinality when compared to the original. Otherwise, you seem to have more or less the right ideas: 1. If you want to add a label like highcardinality="X" to your original source metrics, you can do this at scrape time, either using target relabelling (if it applies to all metrics from a given target) or metric relabelling (if it only applies to specific metrics) 2. You can set or override a label like highcardinality="Y" in your recording rules. You don't need label_replace() to do that; the recording rule itself <https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule> has a "labels" block. BTW, it's standard practice that recording rules should be generating metrics with a different name. If you did this, you can match on the name pattern for remote storage. This is a case where label_replace <https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace> may do the job; I'm not sure if it's allowed to change __name__ with that, but it's worth a try. There are some hints on how to name metrics in recording rules at https://prometheus.io/docs/practices/rules/#recording-rules OTOH, I can see why you don't want to change the metric names, when you're not really rolling up metrics into a summary, you're just dropping a subset of metrics that are not of interest. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6f6db63e-e987-4168-b243-387f21a2bf54n%40googlegroups.com.
