> #metrics(5min) /#metrics(30 mins) > 50
Thinks: if you're only interested in the *number* of timeseries for each
metric name, then you can do
count by (__name__) ({__name__=~".+"})
(warning: potentially expensive query if you have many timeseries). Then
you could move the metric name into a label:
label_replace(count by (__name__) ({__name__=~".+"}), "metric", "$1",
"__name__", "(.+)") * 1
At that point, you have something you could alert on. Example: find metrics
which have at least 1% more timeseries than they did 30 minutes ago:
(label_replace(count by (__name__) ({__name__=~".+"}), "metric", "$1",
"__name__", "(.+)") * 1) / (label_replace(count by (__name__)
({__name__=~".+"} offset 30m), "metric", "$1", "__name__", "(.+)") * 1) >
1.01
This won't detect *completely new* metrics which appear, but you could have
a separate rule for these, e.g. (untested):
(label_replace(count by (__name__) ({__name__=~".+"}), "metric", "$1",
"__name__", "(.+)") * 1) unless (label_replace(count by (__name__)
({__name__=~".+"} offset 30m), "metric", "$1", "__name__", "(.+)") * 1)
Or to detect *every* new timeseries, including new timeseries for existing
metrics:
{__name__=~".+"} unless {__name__=~".+"} offset 30m
On Friday, 24 March 2023 at 08:41:12 UTC Agarwal ,Naveen wrote:
> Thanks Brian. Insightful.
>
> Sent from Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
> *From:* [email protected] <[email protected]> on
> behalf of Brian Candler <[email protected]>
> *Sent:* Friday, March 24, 2023 1:05:30 PM
> *To:* Prometheus Users <[email protected]>
> *Subject:* [prometheus-users] Re: How to define metric type as a variable
>
> No, because binary operators like division are designed to work between
> different metrics (with the same set of labels, but different metric name),
> e.g.
>
> node_filesystem_avail_bytes / node_filesystem_size_bytes
>
> You can however generate your alerting rules programatically: make a
> script that writes out a rules file, then hits the reload
> <https://prometheus.io/docs/prometheus/latest/management_api/#reload>
> endpoint.
>
> On Friday, 24 March 2023 at 00:57:16 UTC Agarwal ,Naveen wrote:
>
> Hi:
>
> Our prometheus database contains around 5k+ unique type of metrics. With
> time, we have defined alerting rules to detect deviations.
>
> However, given the number of growing metrics, it is becoming difficult to
> expand the alerting rules.
>
> Generally we are interested in increase/decrease of values in the metrics
> when compared to a previous time-interval. Keeping this in mind, is it
> possible to write a query where metric name is not specified, instead it
> picks up all metric names available in database in sequence.
>
>
> e.g. #metrics(5min) /#metrics(30 mins) > 50
> all unique metric names are picked from database.
>
> Thanks,
> Naveen
>
> Sent from Outlook for Android <https://aka.ms/AAb9ysg>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/11bed9ec-d186-4757-ac29-4d6147d61e76n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/prometheus-users/11bed9ec-d186-4757-ac29-4d6147d61e76n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/57eb9103-fc93-4dca-b7c7-663ddcd6a453n%40googlegroups.com.
