Hi all,

I'm giving a talk at ApacheCon in early October (in the Performance
Engineering track) on Kafka Zookeer vs. KRaft modes. I've already got some
"interesting" results but wondering if anyone can help with the following
please:

1) What benchmarking has been done on KRaft clusters, particularly with
large numbers of Partitions? Any instructions on how this was done and
results would be useful.

2) I'm trying to test out the assertion that KRaft enables clusters with
Millions of partitions. This is proving difficult. The main problem is how
to create a topic with a million partitions, or potentially lots of topics
with a total of 1M partitions. In particular, if anyone has done this
before (there are blogs claiming 2M partitions successfully created, but no
details) I'd appreciate any hints in terms of process to do it, and what
resources required for a cluster. Currently I get errors around 250K
partitions which prevent me getting any higher (possibly a memory error at
least). Note that I'm not trying to get to 1M partitions with any load on
the cluster as that will need a massive cluster based on past benchmarking
with RF=3, so I'm using RF=1 to prevent any replication CPU overhead with
increased partitions so it's pretty much an artificial scenario for a
start. If I can reach 1M partitions with RF=1 I will explore a more
production ready configuration.

Regards, Paul Brebner

Reply via email to