Those numbers seem to be well within the expected normal range. One thing we noticed is that Kubernetes kubelet/cAdvisor has a lot of less useful/duplicate metrics exposed.
This is the metric filter we ended up with for "/metrics/cadvisor" https://gist.github.com/SuperQ/d61e405e3464da4a766808f6329c5c1e On Wed, May 25, 2022 at 2:55 PM [email protected] <[email protected]> wrote: > Sounds about right. > I do have a number of instances with 6-6.4M time series and they use > around 20-35GB of go_memstats_alloc_bytes and around 66-96GB of RSS memory. > That's all with GOGC=40. > go_memstats_alloc_bytes / prometheus_tsdb_head_series is between 4-8KB on > most of them. > > On Wednesday, 25 May 2022 at 13:45:09 UTC+1 [email protected] wrote: > >> Thanks for this info, so if I am understanding correctly. >> >> prometheus_tsdb_head_series returns 6181063 , when graphing this it >> almost looks vertical. >> >> go_memstats_alloc_bytes / prometheus_tsdb_head_series returns 6741 >> >> Doing a calculation on that, would come up to roughly 41GB , prior to any >> other memory utilization requirements of Prometheus. With that said then >> you double it, so looking more like 82GB of memory + allocation for >> prometheus overhead. Does this seem about accurate? >> >> Thanks! >> >> >>>> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/773e7e8f-55a6-40d1-8067-6a2c064b23c1n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/773e7e8f-55a6-40d1-8067-6a2c064b23c1n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoy3Z_p2Dvd2_gWQBbnTZZGyPtTQTRN5MQ9wUZTwr1a_A%40mail.gmail.com.
