Hi, I just wanted to have some general discussions around the topic of how to scale up a Kafka Cluster ?
Currently we are running a 5 node Kafka Cluster. Each node has* 4 vcpu *and *8 GiB* memory. I have a topic which is partitioned *35* ways. I have *5* producers publishing messages to that topic. I have *1* consumer consuming messages from that topic. Each message is a JSON string say *2 - 4KB* uncompressed and *600Bytes* compressed. Right now this cluster can handle *10,000 *messages per second. I see no lag in the producer or consumer side. I would like to scale the cluster to handle *1 million* messages per second. What are the areas I should look into and in what order ? My producers and consumers can scale independently. I can run 35 producers and consumers if need be. Questions I have are 1. Should I increase the number of partitions for that topic from 35 ? 2. Should I increase the number of brokers from 5 ? 3. Should I increase the instance size in terms of memory or CPU per node ? 4. Or should it be combinations of any of the above options ? 5. Are there any other settings in the producer or broker side or topic side I should consider ? Thanks Sachin
