The sidecar of Thanos and Prometheus federations are not the same things!
A federation allows aggregate metrics to another Prometheus instance, and this is useful because, for example, you might have 'non-standard' blocks. As a reminder, if that's the case, you have no choice but to go through a Prometheus with fixed 2-hour non-compacted blocks for compatibility with the sidecar. At least, that was in the documentation. However, in practice, it's true that the sidecar push metrics to object storage such as S3, Azure, etc.., even if the blocks are compacted. There is an option for that: '--shipper.upload-compacted'. However, it doesn't tag, at least not to my knowledge! So, no, it's not obsolete; it's quite practical! In other words, you can tag the data coming from multiple Prometheuses by adding regional tags or information that makes it easy to query multisite. However, I haven't seen that the sidecar of Thanos can add tags? In essence, what needs to be understood in my opinion is: prometheus1 (with exporter)---- prometheus2 (with exporter) | prometheus3 (with exporter) |----------------prometheus (federate)------->thanos-sidecars----->S3 storage prometheus4 (with exporter)--- (with different tags for prometheus1,2,3,4 Prometheus 1-4 store no more than 15 days of metrics (default behavior) Prometheus-federate is the same All metrics exceeding 15 days are stored on object storage such as S3, Azure, or any other of your choice! Indeed, if we could tag from the sidecar, it would allow us to bypass Prometheus federations. But for now, I admit I don't know if it's feasible Le jeudi 23 novembre 2023 à 10:24:19 UTC+1, Bryan Boreham a écrit : > Not going to argue with you but if that is the official position we should > put it in the docs. > > Bryan > > On Wednesday, 22 November 2023 at 22:45:57 UTC [email protected] wrote: > >> I'd say less neglected, but more obsoleted by. Remote Write and Thanos >> Sidecar are the more functional modern replacements for the original >> Federation method. >> >> On Wed, Nov 22, 2023 at 1:01 PM Bryan Boreham <[email protected]> wrote: >> >>> Federation is a bit of a neglected feature. The Thanos project is >>> rather more popular as a way to aggregate data from multiple Prometheus. >>> >>> (Other projects also exist which can let you centralise metrics storage) >>> >>> Bryan >>> >>> On Thursday, 16 November 2023 at 15:51:12 UTC [email protected] >>> wrote: >>> >>>> I take the silence as a "no, it is not possible to connect recording >>>> rules with federation requests (or vice versa)" ...? >>>> >>>> On Tuesday, October 31, 2023 at 5:54:05 PM UTC+1 [email protected] >>>> wrote: >>>> >>>>> *Is it possible to include an aggregation formula in a federation >>>>> query?* >>>>> This would avoid creating and storing aggregated metrics on the >>>>> lower-level Prometheus, and also remove the delay between aggregation and >>>>> federation. >>>>> >>>>> *Or, can a recording rule fetch metrics directly from another >>>>> Prometheus server?* >>>>> This would avoid the delay between federation and execution of the >>>>> recording rule. >>>>> >>>>> We've got hundreds of pods, delivering millions of metrics. We plan to >>>>> partition our pods and deploy one Prometheus per partition. A top-level >>>>> Prometheus will then offer globally aggregated metrics. >>>>> >>>>> How should we set this up? My current assumption is based on >>>>> aggregation recording rules within each partition, then again at the >>>>> top-level to get the global aggregation. This seems both complicated and >>>>> a >>>>> waste of resources, plus also introduces delays since recording rules and >>>>> federation cannot be synced to each other. >>>>> >>>>> To minimize delay, each partition-level Prometheus needs to aggregate >>>>> as often as possible to offer "fresh" metrics to federation requests. >>>>> Then >>>>> the top level Prometheus also needs to federate these aggregated metrics >>>>> as >>>>> often as possible to offer "fresh" values at the global level. Doing >>>>> things >>>>> as often as possible "just in case" seems wasteful, which is why I am >>>>> asking if this is the right approach for us. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/6ec7e309-1e6f-4700-97d8-496283d74fb9n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/6ec7e309-1e6f-4700-97d8-496283d74fb9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c6e66843-6046-4253-b2d1-3a2cd403d1ffn%40googlegroups.com.
