Hi, I have a few blogs on Kafka partitions performance for background if you are interested:
1. https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/ The Power of Apache Kafka® Partitions: How to Get the Most out of Your Kafka Cluster<https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/> Partitions are the main concurrency mechanism in Kafka. Learn how to derive maximum benefit by using partitions for your kafka cluster. www.instaclustr.com 2. New 3 part KRaft series: https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/ 3. ApacheCon NA 2022 talk https://apachecon.com/acna2022/slides/04_Brebner_The_Impact_Of.pdf Basically throughput with increasing partitions tends to drop due to replication overhead, but newer versions are better. The "problem" with KRaft mode may be that it's too easy to create topics with lots of partitions, with potential performance implications still. Regards, Paul ________________________________ From: 牟忠玉 <muzhon...@yeah.net> Sent: 21 February 2023 03:10 AM To: users@kafka.apache.org <users@kafka.apache.org> Subject: Kafka partition problem NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Why is Kafka's QPS performance very poor with a large number of partitions (such as 10,000 partitions)? What is the reason for this? Do you have any plans to optimize this in the future? Or are there already optimizations in newer versions?
