I've a peculiar use case where there are large number of time series
published with a label set, that is causing overhead at query time when the
filter includes shared label value. e.g.
my_metric1{l1="fixed",l2="..",l3=".."}
my_metric2{l1="fixed",l2="..",l3=".."}
my_metric3{l1="fixed",l2="..",l3=".."}
...
query: my_metric1{l1="fixed",...}
The overhead is not dependent on number of samples query, but due to the
way prometheus loads metric metadata, specifically label postings and finds
an intersection of these lists to find the time series to load from DB. If
I remove the l1 filter above, query runs much faster and overhead is
minimal.
Due to number of queries hitting the system and high friction of changes in
user applications, I am looking for ways to solve this problem in query
processing. One option I was considering was filtering this label out
before sending query to remote read destination but the remote read
configuration does not support metric relabeling. Can you suggest
alternatives? Modifying the ingestion path or metric modeling is not
feasible at this point and will take a long time to implement.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/1dd94781-381e-44fc-a3ba-49d98418deb4n%40googlegroups.com.
