This is Prometheus staleness handling. In order to allow queries to function, Prometheus graph query evaluation (range query) is actually a series of query evaluations over time. Each one is independent of the next.
In order to support millisecond accurate timestamps, Prometheus will look back up to 5 minutes for samples from each evaluation timestamp. There are a couple ways around this. * Scrape faster. It's perfectly common and normal to scrape every 15s-60s in Prometheus. It basically doesn't take any more storage space due to the way Prometheus compresses samples. In fact, scraping slower wastes data by creating incomplete blocks. * Use a function like `avg_over_time(metric[$__rate_interval])` to effectively increase the lookback. This needs to be combined with setting the "min step" in the Grafana query options to 30min to match your scrape interval. On Thu, Jul 18, 2024 at 8:48 AM Tareerat Pansuntia < tareerat.pan...@gmail.com> wrote: > hi, everyone > > I have configured Prometheus to scrape data every 30 minutes. This means > Prometheus will scrape data every thirty minutes and then display it in > Grafana. However, the graph in Grafana shows the data for only a brief > moment and then disappears. After thirty minutes, the graph reappears > again. What could be causing this issue, and how can I fix it? > > [image: Screenshot 2024-07-18 112523.png] > > In the graph , the green line appears every thirty minutes and then > disappears until the next thirty-minute interval. I have set the > scrape_interval for that website to every 30 minutes. What could be causing > the data to disappear in between, and how can I fix this issue? > > Thanks in advance, > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/13d491cd-f8c0-42a3-bb3e-9f5a0fb94454n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/13d491cd-f8c0-42a3-bb3e-9f5a0fb94454n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpJGquNkiq972pxeHVm0bLWLpDEPvPvrOSCiCEOwXnjaA%40mail.gmail.com.
