Prometheus does not store or return "null". Prometheus will store a "staleness marker" when it performs a scrape and doesn't find a timeseries (i.e. metric+label set) which was present in the previous scrape. Internally, it's stored as a special kind of floating-point NaN. However it's not returned in queries. Instead, the timeseries just "disappears" as if it didn't exist.
A timeseries consists of a series of values V1, V2, V3 .. at timestamps T1, T2, T3 ... So when you perform an instant query for some arbitrary time T, then prometheus has to look back in time to find the nearest data point at or before time T. It will look back up to --query.lookback-delta (by default 5m), but if it finds a staleness marker, it just doesn't return any value, i.e. that timeseries is excluded from the vector result. This is all described here <https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness>. If you want to look gaps in a particular timeseries, you can build a query using absent() <https://prometheus.io/docs/prometheus/latest/querying/functions/#absent>, giving a specific timeseries (metric+labels) that you're looking for, and prometheus will tell you when the time series doesn't exist (i.e. there is no data point within the previous 5 minutes *or* the series has been marked stale) absent{foo{instance="bar",job="baz"}) If you want to see when arbitrary timeseries disappear, without hardcoding specific label sets, you can try something like this: foo offset 5m unless foo # it existed 5 minutes ago, but doesn't exist now present_over_time(foo[24h] offset 5m) unless foo # existed any time in the last 24 hours until 5 minutes ago, but doesn't exist now On Tuesday, 25 January 2022 at 18:59:23 UTC [email protected] wrote: > I know in Grafana I can choose to display NULL as zero or as missing but I > actually want to graph when a value is null - to visualise downtime where a > series has gaps. Can I craft a query which will spit out two values for > NULL / not NULL? > > Thanks. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3cc45b2f-115b-4ec6-a529-a474498b1ed7n%40googlegroups.com.
