Hi 100 partitions is not a high number for this cluster. The downsides of having more partitions are : - having more file descriptors open, check that the limit for the user running kafka are high enough - more work to perform for the brokers and more memory used for keeping the metadata about the partitions (but 30 to 100 should be fine) - if the clean strategy has not changed, you will use more disk space
So you have to consider these cons versus the benefit you get from the parallelism On Thu, Mar 21, 2019 at 3:59 AM 1095193...@qq.com <1095193...@qq.com> wrote: > Hi, > The number of partitions drives the parailism of consumers. In general, > the more partitions, the more parallel consumer can be added , the more > throughput can be provided. In other words, if you have 10 partitions, the > most number of consumer is 10. So you need to assume the throughput a > consumer can provide is C, and the target throughput is T. Then the minimum > number of partitions, that is, the number of consumers, is T/C. > > > > 1095193...@qq.com > > From: shalom sagges > Date: 2019-03-21 06:43 > To: users > Subject: Partition Count Dilemma > Hi All, > > I'm really new to Kafka and wanted to know if anyone can help me better > understand partition count in relation to the Kafka cluster (apologies in > advance for noob questions). > > I was requested to increase a topic's partition count from 30 to 100 in > order to increase workers' parallelism (there are already other topics in > this cluster with 100-200 partition counts per topic). > The cluster is built of 4 physical servers. Each server has 132 GB RAM, 40 > CPU cores, 6 SAS disks 1.1 TB each. > > Is PartitionCount:100 considered a high number of partitions per topic in > relation to the cluster? > Is there a good way for me to predetermine what an optimal partition count > might be? > > Thanks a lot! >
