I am tracking the number of jobs in a queue at specific time intervals
using a gauge metric. Prometheus scrapes this every minute.
However, when I attempt to determine the highest number of jobs in the
queue on a given day using the max_over_time query, I receive two distinct
values for the same day based on different time ranges.
I am using the query max_over_time(job_count_by_service{service="ServiceA",
tenant="TenantA"}[1d]). When I run this query for a 1-day time range (from
2023-08-19 00:00:00 to 2023-08-19 23:59:59), the value I get is 38.
However, when I run the same query for a 5-day time range (from 2023-08-18
00:00:00 to 2023-08-22 23:59:59), the result for Aug 19th is 35.
2023-08-19 00:00:00 to 2023-08-19 23:59:59 [Range Selected in Grafana]
[image: RSxCO.png]
2023-08-18 00:00:00 to 2023-08-22 23:59:59 [Range Selected in Grafana]
[image: gmW3m.png]
In Grafana I have configured the Min Step as 1d and Type as Range. I'm not
sure whether that could affect the values in any way.
I assumed that max_over_time would pick the max value among all the values
that fall in the range vector specified time period. For example, if on Day
1 the values are [1,2,7,6,5] and on Day 2 the values are [8,1,2,3,1] then
the query would return 7 & 8 respectively for each day.
To elaborate when I select 2023-08-19 00:00:00 to 2023-08-19 23:59:59 as
the time range the values from 2023-08-18T00:00:00.000 ...
2023-08-19T00:00:00.000 is considered to pick a maximum. When I select the
time range 2023-08-18 00:00:00 to 2023-08-22 23:59:59 won't the same values
again be considered for picking the maximum since the range vector is 1d?
So how can the value change from 38 to 35?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/fc4f15ff-914c-4cae-8aa4-964f5a99f5b6n%40googlegroups.com.
