Hi all, I'm giving a talk at ApacheCon in early October (in the Performance Engineering track) on Kafka Zookeer vs. KRaft modes. I've already got some "interesting" results but wondering if anyone can help with the following please:
1) What benchmarking has been done on KRaft clusters, particularly with large numbers of Partitions? Any instructions on how this was done and results would be useful. 2) I'm trying to test out the assertion that KRaft enables clusters with Millions of partitions. This is proving difficult. The main problem is how to create a topic with a million partitions, or potentially lots of topics with a total of 1M partitions. In particular, if anyone has done this before (there are blogs claiming 2M partitions successfully created, but no details) I'd appreciate any hints in terms of process to do it, and what resources required for a cluster. Currently I get errors around 250K partitions which prevent me getting any higher (possibly a memory error at least). Note that I'm not trying to get to 1M partitions with any load on the cluster as that will need a massive cluster based on past benchmarking with RF=3, so I'm using RF=1 to prevent any replication CPU overhead with increased partitions so it's pretty much an artificial scenario for a start. If I can reach 1M partitions with RF=1 I will explore a more production ready configuration. Regards, Paul Brebner
