Thanos supports downsampling, but the main driver is to enable faster queries over large time ranges, not to reduce storage space. https://thanos.io/v0.8/components/compact/ https://thanos.io/v0.8/components/compact/#storage-space-consumption
Do remember that prometheus' storage is already extremely efficient at compression, especially for timeseries which are static or rarely change. On Thursday, 15 June 2023 at 08:10:09 UTC+1 sayf eddine Hammemi wrote: > Hello > What you need is downsampling, which is not supported directly by > prometheus (as a matter of fact, storing historical data is not either > recommended on bare prometheus), > It is time to check external storage solutions for prometheus and choose > the one that suits your needs (mimir, cortex, thanos, m3db, etc ...) > > On Thu, Jun 15, 2023 at 9:01 AM Boris Behrens <[email protected]> wrote: > >> Hi Ben, >> I've read the link you gave me and I will try to answer the best I can. >> >> > What problem are you solving for? >> I want to keep historical data for longer periods with less disk space >> used. >> I currently keep around 4 month of data for around 1.3T of disk space. >> and I would love to have 1-2 years of data without having to buy 20TB >> disks :) >> >> > Why do you think you need this? >> For debugging and monitoring I only need the last couple of days with >> a raw resolution. And seeing trends can be downsampled (is this the >> correct term?) because it is just historical trend data. >> I think it's rather a use case than an actual problem I am working on. >> I would like to see in the past how things changed and overlay it with >> timestamps from changes. Things begin to looks way different when you >> check the past. >> >> > You say "a lot of endpoints", but what is a lot? 1,000? 10,000? >> 100,000? 1,000,000? How many series? >> A couple hundred endpoints with around 1.5m of series. But I don't >> think that this information is helpful in the current context. >> >> Thank you for your time and effort to help me with my stupid little >> problem. >> Cheers >> Boris >> >> Am Do., 15. Juni 2023 um 06:24 Uhr schrieb Ben Kochie <[email protected]>: >> > >> > My first question is "why?" (https://xyproblem.info/) >> > >> > What problem are you solving for? Why do you think you need this? >> > >> > You say "a lot of endpoints", but what is a lot? 1,000? 10,000? >> 100,000? 1,000,000? How many series? >> > >> > On Thu, Jun 15, 2023 at 6:21 AM Boris <[email protected]> wrote: >> >> >> >> Dear prometheus community, >> >> >> >> my prometheus scrapes a lot of endpoints and I only need to 10s for a >> month. After that I would like to rollup the data to 5m averages and after >> 6 months I would like to have only hourly averages. >> >> >> >> So I can have quite a long time of data and can see changes over a >> long time with the same dashboard. >> >> >> >> I am sorry to ask this question but it seems like I don't have the >> correct search term. Downsampling seems to keep the raw data and all >> searches lead me to timescaledb. >> >> >> >> Cheers >> >> Boris >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups "Prometheus Users" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/8D01579C-5696-40BB-9841-A1E3E0094DD1%40kervyn.de >> . >> >> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend >> im groüen Saal. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAHjiyFTAWUHwrpfT9NJZxwudSrUveSvsKpG7bcpb3nRm_-CrRA%40mail.gmail.com >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a74b97e1-8315-45d9-a921-cbb6b797be96n%40googlegroups.com.
