thank you everyone.
We are planning for 10K nodes and each node has 128 cores. So the
timeseries data is: 128 * 8 * 10000 = 10,240,000. Meanwhile, there are
other timeseries data from kubeletăkube-state-metrics and more(business
data). Totally, all data comes to around 30M timeseries, then prometheus
eats 170G memory (as tested), and we think there should be some buffer
(maybe 100G). So it makes sense to reduce the node-exporter timeseries data
to 128 * 1 * 10000 = 1,280,000.
We will have a try keeping the idle mode only for cpu usage. Some expr
considered:
1 - avg without(cpu, mode)
(rate(node_cpu_seconds_total{job="node-exporter",mode="idle"}[5m]))
avg(1 -
avg(rate(node_cpu_seconds_total{origin_prometheus=~"$origin_prometheus",job=~"$job",mode="idle"}[$interval]))
by (instance)) * 100
On Monday, February 6, 2023 at 4:46:10 AM UTC+8 [email protected] wrote:
> On Sunday, 5 February 2023 at 11:13:23 UTC Brian Candler wrote:
> Timeseries in Prometheus are extremely cheap. If you're talking 10K nodes
> and 96 cores per node, that's less than 1m timeseries; compared to the cost
> of the estate you are managing, it's a drop in the ocean :-) How many
> *other* timeseries are you storing from node_exporter?
>
> A single timeseries eats >=4KB on all nodes I touch. Having a lot of
> labels (or long labels) will make it more expensive.
> So 1M timeseries will eat 4GB of memory.
> Not everyone would call that extremely cheap, especially if that's just to
> tell what's the cpu usage of each server.
>
> There was a PR that tried to implement scrape time recording rules, which
> would help here, but it didn't seem to go far -
> https://github.com/prometheus/prometheus/pull/10529
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/410dd3e5-6abf-4aa3-ae4e-7fb55184c227n%40googlegroups.com.
