> grafana has the datasource, which points to the k8s service, which is a "loadbalancer", which has two endpoints; one for each prometheus pod.
Then each query from grafana goes to *either* prometheus A *or* prometheus B. If both prometheus servers are scraping the same endpoints then they'll have roughly the same data in them, so you'll see roughly the same graphs. However, using a loadbalancer in this way is not recommended, as it's not deterministic which prometheus server your query will hit. I suggest you look at promxy <https://github.com/jacksontj/promxy>, which is designed for this use case and can do useful things like filling any gaps from data in prometheus A with data from prometheus B, or vice versa. > take it that my two prometheus do not sync timeseries; That is correct. They'll both independently scrape and store. I say they'll have "roughly" the same data because they won't scrape at the same points in time. > but with grafana i see no gaps in my graphs. Firstly, restarting prometheus is pretty quick. Secondly, PromQL queries will look back in time up to 5 minutes (by default) to find a previous data point. Up to that interval, your grafana graphs will look flat but without gaps. On Thursday, 23 February 2023 at 22:13:46 UTC+8 Mario Cornaccini wrote: > hi, > > i just updated our kube-prometheus custom ressource, which manages a > deployment of two prometheus pods, which flawlessly upgraded the pods one > after each other..(deployment strategy) > > but with grafana i see no gaps in my graphs. > grafana has the datasource, which points to the k8s service, which is a > "loadbalancer", which has two endpoints; one for each prometheus pod. > (maybe thats a question for the grafana mailinglist then) > > i take it that my two prometheus do not sync timeseries; > (as opposed to alertmanager which does gossip) > > das anybody know why there seem to be no gaps in prometheus scraping > timeseries ? > although i do see a gap when i look at each pods' cpu-usage. of course, > it's because they got restarted. > > just curious.. > cheers, > fil > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ed619ddd-f48e-4fc3-83b9-c9d85f231706n%40googlegroups.com.
