Also: how many timeseries are you working with, in terms of the "my_series"
that you are querying, and globally on the whole system?
On Wednesday, 5 April 2023 at 18:12:11 UTC+1 Brian Candler wrote:
> Adding a constant label to every timeseries should have almost zero impact
> on memory usage.
>
> Can you clarify what you're saying, and how you've come to your diagnosis?
> What version of prometheus are you running? When you say "backends" in the
> plural, how have you set this up?
>
> At one point you seem to be saying it's something to do with ingestion,
> but then you seem to be saying it's something to do with queries (*"Without
> this filter, the queries run reasonably well"*). Can you give specific
> examples of filters which show the difference in behaviour?
>
> Again: the queries
> my_series{global_label="constant-value", l1="..", l2=".."}
> my_series{l1="..", l2=".."}
> should perform almost identically, as they will select the same subset of
> timeseries.
>
> On Wednesday, 5 April 2023 at 17:42:33 UTC+1 Johny wrote:
>
>> There is a performance related issue we're facing in Prometheus coming
>> from a label with a constant value across all (thousands of) time series.
>> The label filter in query causes a large quantity of metadata to load in
>> memory overwhelming Prometheus backends. Without this filter, the queries
>> run reasonably well. We are planning to exclude this label in ingestion in
>> future, but for now we need a workaround.
>>
>> my_series{*global_label="constant-value"*, l1="..", l2=".."}
>>
>> Is there a mechanism to automatically exclude global_label in query
>> configuration: remote_read subsection, or elsewhere?
>>
>> thanks,
>> Johny
>>
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/0716b36e-835e-44fd-b36c-0b63834b8447n%40googlegroups.com.
