i guess this 5 min looking back is what did the trick for me; thanks! the datasource configuration comes from kube-prometheus; i did not yet ever notice a slightly different output after a reload; i guess thats not a problem/use-case for me then. i'll have a look at promxy though anyway.
cheers, fil On Saturday, February 25, 2023 at 7:47:34 PM UTC+1 Brian Candler wrote: > > grafana has the datasource, which points to the k8s service, which is a > "loadbalancer", which has two endpoints; one for each prometheus pod. > > Then each query from grafana goes to *either* prometheus A *or* prometheus > B. If both prometheus servers are scraping the same endpoints then they'll > have roughly the same data in them, so you'll see roughly the same graphs. > > However, using a loadbalancer in this way is not recommended, as it's not > deterministic which prometheus server your query will hit. > > I suggest you look at promxy <https://github.com/jacksontj/promxy>, which > is designed for this use case and can do useful things like filling any > gaps from data in prometheus A with data from prometheus B, or vice versa. > > > take it that my two prometheus do not sync timeseries; > > That is correct. They'll both independently scrape and store. I say > they'll have "roughly" the same data because they won't scrape at the same > points in time. > > > but with grafana i see no gaps in my graphs. > > Firstly, restarting prometheus is pretty quick. > > Secondly, PromQL queries will look back in time up to 5 minutes (by > default) to find a previous data point. Up to that interval, your grafana > graphs will look flat but without gaps. > > On Thursday, 23 February 2023 at 22:13:46 UTC+8 Mario Cornaccini wrote: > >> hi, >> >> i just updated our kube-prometheus custom ressource, which manages a >> deployment of two prometheus pods, which flawlessly upgraded the pods one >> after each other..(deployment strategy) >> >> but with grafana i see no gaps in my graphs. >> grafana has the datasource, which points to the k8s service, which is a >> "loadbalancer", which has two endpoints; one for each prometheus pod. >> (maybe thats a question for the grafana mailinglist then) >> >> i take it that my two prometheus do not sync timeseries; >> (as opposed to alertmanager which does gossip) >> >> das anybody know why there seem to be no gaps in prometheus scraping >> timeseries ? >> although i do see a gap when i look at each pods' cpu-usage. of course, >> it's because they got restarted. >> >> just curious.. >> cheers, >> fil >> >> >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/69c252c2-293c-48ff-99f7-d600942e1324n%40googlegroups.com.
