Thanks a lot guys! That's really helpful! So now I understand that the partition count affects the number of consumers I can efficiently use. Does partition count affect producers as well?
Thanks! On Thu, Mar 21, 2019 at 10:13 AM Vincent Maurin <vincent.maurin...@gmail.com> wrote: > Hi > > 100 partitions is not a high number for this cluster. > The downsides of having more partitions are : > - having more file descriptors open, check that the limit for the user > running kafka are high enough > - more work to perform for the brokers and more memory used for keeping the > metadata about the partitions (but 30 to 100 should be fine) > - if the clean strategy has not changed, you will use more disk space > > So you have to consider these cons versus the benefit you get from the > parallelism > > On Thu, Mar 21, 2019 at 3:59 AM 1095193...@qq.com <1095193...@qq.com> > wrote: > > > Hi, > > The number of partitions drives the parailism of consumers. In general, > > the more partitions, the more parallel consumer can be added , the more > > throughput can be provided. In other words, if you have 10 partitions, > the > > most number of consumer is 10. So you need to assume the throughput a > > consumer can provide is C, and the target throughput is T. Then the > minimum > > number of partitions, that is, the number of consumers, is T/C. > > > > > > > > 1095193...@qq.com > > > > From: shalom sagges > > Date: 2019-03-21 06:43 > > To: users > > Subject: Partition Count Dilemma > > Hi All, > > > > I'm really new to Kafka and wanted to know if anyone can help me better > > understand partition count in relation to the Kafka cluster (apologies in > > advance for noob questions). > > > > I was requested to increase a topic's partition count from 30 to 100 in > > order to increase workers' parallelism (there are already other topics in > > this cluster with 100-200 partition counts per topic). > > The cluster is built of 4 physical servers. Each server has 132 GB RAM, > 40 > > CPU cores, 6 SAS disks 1.1 TB each. > > > > Is PartitionCount:100 considered a high number of partitions per topic in > > relation to the cluster? > > Is there a good way for me to predetermine what an optimal partition > count > > might be? > > > > Thanks a lot! > > >
