Hi, The number of partitions drives the parailism of consumers. In general, the more partitions, the more parallel consumer can be added , the more throughput can be provided. In other words, if you have 10 partitions, the most number of consumer is 10. So you need to assume the throughput a consumer can provide is C, and the target throughput is T. Then the minimum number of partitions, that is, the number of consumers, is T/C.
1095193...@qq.com From: shalom sagges Date: 2019-03-21 06:43 To: users Subject: Partition Count Dilemma Hi All, I'm really new to Kafka and wanted to know if anyone can help me better understand partition count in relation to the Kafka cluster (apologies in advance for noob questions). I was requested to increase a topic's partition count from 30 to 100 in order to increase workers' parallelism (there are already other topics in this cluster with 100-200 partition counts per topic). The cluster is built of 4 physical servers. Each server has 132 GB RAM, 40 CPU cores, 6 SAS disks 1.1 TB each. Is PartitionCount:100 considered a high number of partitions per topic in relation to the cluster? Is there a good way for me to predetermine what an optimal partition count might be? Thanks a lot!
