Re: How to scale a Kafka Cluster, what all should we consider

2025-02-02 Thread Sachin Mittal
Hi Paul, For the current cluster I see CPU utilization of 10% and memory utilization of 50%. We were thinking of 7 nodes, each 16 vcpu cluster which would translate to 116 vCPU at cluster level, which would be significantly more than 87, you have recommended. wrt to the number of partitions 35 see

Re: How to scale a Kafka Cluster, what all should we consider

2025-02-02 Thread Brebner, Paul
Hi Sachin, I’m not an “operational” Kafka person but do have some limited experience with Kafka benchmarking etc, so here are a few ideas. I’m playing around with a Kafka tiered storage sizing model at present, designed to predict min IO and/or network with local and tiered storage enabled. Th