Ok, thanks for confirming my suspicion 👍 On Wednesday, November 22, 2023 at 1:01:51 PM UTC+1 Bryan Boreham wrote:
> Federation is a bit of a neglected feature. The Thanos project is rather > more popular as a way to aggregate data from multiple Prometheus. > > (Other projects also exist which can let you centralise metrics storage) > > Bryan > > On Thursday, 16 November 2023 at 15:51:12 UTC [email protected] wrote: > >> I take the silence as a "no, it is not possible to connect recording >> rules with federation requests (or vice versa)" ...? >> >> On Tuesday, October 31, 2023 at 5:54:05 PM UTC+1 [email protected] >> wrote: >> >>> *Is it possible to include an aggregation formula in a federation query?* >>> This would avoid creating and storing aggregated metrics on the >>> lower-level Prometheus, and also remove the delay between aggregation and >>> federation. >>> >>> *Or, can a recording rule fetch metrics directly from another Prometheus >>> server?* >>> This would avoid the delay between federation and execution of the >>> recording rule. >>> >>> We've got hundreds of pods, delivering millions of metrics. We plan to >>> partition our pods and deploy one Prometheus per partition. A top-level >>> Prometheus will then offer globally aggregated metrics. >>> >>> How should we set this up? My current assumption is based on aggregation >>> recording rules within each partition, then again at the top-level to get >>> the global aggregation. This seems both complicated and a waste of >>> resources, plus also introduces delays since recording rules and federation >>> cannot be synced to each other. >>> >>> To minimize delay, each partition-level Prometheus needs to aggregate as >>> often as possible to offer "fresh" metrics to federation requests. Then the >>> top level Prometheus also needs to federate these aggregated metrics as >>> often as possible to offer "fresh" values at the global level. Doing things >>> as often as possible "just in case" seems wasteful, which is why I am >>> asking if this is the right approach for us. >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ab32c10a-eb9d-4867-bb6f-de47c342983bn%40googlegroups.com.
