Yes, you can have 1000 partitions. But, there are implications of having a large number. Each partition has a leader. Clients downloading metadata to find those 1000 leaders will be a bit slower than finding 100 leaders. Producers that are buffering messages in order to use batching create a buffer per partition, so you may see increased memory usage. Likewise, if one of your brokers failed, the more partition leaders on it, the more leader elections that have to occur.
I'd suggest benchmarking your use case with different partition counts and see where your sweet spot is. This old blog post has some good ideas: https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ Cheers, Liam Clarke On Tue, Nov 24, 2020 at 10:51 PM Gowtham S <gowtham.co....@gmail.com> wrote: > Hi, > Can we have 1000 partitions in a Single topic? If not how many partitions > will a single topic have at the max? > Anyone, please tell. > > Thanks and regards, > Gowtham S >
