Hi, Thanks for your reply!
The promQL is working just fine and returning rows as per our expectation.
We have multiple clusters and we created a recording rule using the below
promql.
{job=“cluster-1”} unless on (uid,pod)
kube_pod_info{job=“cluster-1“,created_by_kind=“<none>”}
With the recording rule, we created a new static label called
highcardinality=“true” but this creates new time series. When doing remote
write to our long term storage we are dropping those time series which has
highcardinality=“true” but the original metric doesn’t have this label so
its still getting into our remote write system.
So the recording rule was kind of duplicating the metrics collected. Is
there a way to drop all the metrics that are returned by above promQL you
shared as we don’t want the time series to end up in our long term TSDB.
Local Prometheus. —> Pull Metrics from — > Kube state metrics Statefulset
|
|
| — - - - > Remote write to (Drop metrics that has highcardinality =“true”)
. —> Long Term TSDB (Cortex)
We are thinking of add a new label as part of metric_relabeling section
with highcardinality=“false” and update the label to true using recording
rules and label_replace function instead of the static label implementation
. Is this the right way to do it or is there any other better options that
you can suggest us with.
Thank You
On Friday, 1 December 2023 at 17:33:22 UTC+8 [email protected] wrote:
> Hi All,
>
> We want to drop set of metrics where the pod is created cronjobs and jobs
> (shortlived pods)
>
> kube_pod_info has a label created_by_kind with value as "<none>". I have
> another set of metrics such a kube_pod_status_phase ,
> kube_pod_status_reason etc.
>
> Now I want to drop all the three metrics but I don't have the label
> created_by_kind in the kube_pod_status_phase , kube_pod_status_reason
> metrics.
>
> The common label between all three timeseries is UID and pod .
>
> Is it possible to drop a set of metrics based on the output of another
> metric in the metric_relabeling phase or using any recording rule ?
>
> Looking for some assistance on how we can achieve this
>
> Thank You
> Regards
> Sakthi
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/5cd2b3a1-2c0c-4bce-a18f-4f67c9187ce5n%40googlegroups.com.
