Very good question, have the same concern as Navid. Any ideas on how to get this work in a DB that doesn't provide native support for de-dupe?
On Friday, November 4, 2022 at 4:40:45 PM UTC+1 Navid Shakibapour wrote: > > You would need additional tools like corosync or keepalived to achieve > this. > Thanks for pointing me to those tools. I did a quick search and didn't see > much info on setting up Prometheus HA with corosync/keepalived. Except that > it looks like AWS managed Prometheus uses that approach [1]: > *When you set up deduplication, Amazon Managed Service for Prometheus > makes one Prometheus instance a leader replica and ingests data samples > only from that replica. If the leader replica stops sending data samples to > Amazon Managed Service for Prometheus for 30 seconds, Amazon Managed > Service for Prometheus automatically makes another Prometheus instance a > leader replica and ingests data from the new leader.* > > I'm a bit surprised that this approach isn't used more. > > Re using multiple replicas, my concern is the cost in bigger clusters. The > cost will add up to having multiple replicas per shard knowing the data > would be deduplicated downstream. > On Friday, November 4, 2022 at 10:58:37 AM UTC-4 [email protected] > wrote: > >> I faced this issue in the design phase too, found this >> https://github.com/beorn-/highlander but it looks a bit dead now... >> else you can push duplicated metrics to thanos (while setting a unique >> label for every prometheus instance) and then configure thanos querier to >> deduplicate when serving metrics >> On Friday, November 4, 2022 at 3:40:29 PM UTC+1 Julien Pivotto wrote: >> >>> You would need additional tools like corosync or keepalived to achieve >>> this. In prometheus terms, all the prometheis servers are active and >>> deduplication is done at the remote write end if needed. >>> >>> On 04 Nov 07:24, Navid Shakibapour wrote: >>> > Is there a way to set up multiple Prometheus servers in an >>> active/passive >>> > way? Meaning only 1 server scrapes targets at a time and other ones >>> only >>> > scrape if the active one doesn't "work" after x seconds/minutes. >>> > >>> > On Friday, November 4, 2022 at 3:00:15 AM UTC-4 [email protected] >>> wrote: >>> > >>> > > There is no option without using a single instance. If your 3rd >>> party >>> > > solution does not handle HA, you can not use HA. >>> > > >>> > > On Fri, Nov 4, 2022 at 7:57 AM Navid Shakibapour < >>> [email protected]> >>> > > wrote: >>> > > >>> > >> We are evaluating using Prometheus agent mode in a highly available >>> > >> manner to forward metrics to a 3rd party remote storage using >>> remote write >>> > >> API. >>> > >> We need to have multiple shards and multiple replicas per shard. >>> Our 3rd >>> > >> party remote storage does not handle deduplication on their side. >>> > >> >>> > >> Could someone point me to different options for metrics >>> deduplication on >>> > >> ingestion? >>> > >> >>> > >> >>> > >> -- >>> > >> You received this message because you are subscribed to the Google >>> Groups >>> > >> "Prometheus Users" group. >>> > >> To unsubscribe from this group and stop receiving emails from it, >>> send an >>> > >> email to [email protected]. >>> > >> To view this discussion on the web visit >>> > >> >>> https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com >>> >>> > >> < >>> https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> >>> >>> > >> . >>> > >> >>> > > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/91c8b0d4-67e2-40d3-8314-bd58409cf937n%40googlegroups.com. >>> >>> >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f3f21dd9-9aed-43e1-ac24-031331efc6fen%40googlegroups.com.
