Hi Sven, Normally I'd say that it mainly depends on your latency and throughput requirements, so starting from https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ is probably a good idea. As a rule of the thumb I'd say that for a "typical" cloud deployment you should not exceed high, single-digit thousands of partitions per broker (my personal ballpark figure would be 6-8k tops, but in the cluster I'm currently responsible for I try to not exceed 1k).
However, to be hones, the numbers you provided sound very, *very* odd to me (Thousands of partitions per topic in 1k topics? Millions of partitions in total? ONE consumer per topic and only 2 msg per minute?), so even without any additional context I have a strong feeling that you either: a) have a very unusual use case and Kafka may not be the best tool for the job b) there's a way to "optimise" your design to make it fit better into what Kafka does best. If you could shed some light on what you're trying to do, maybe we can find a better solution. Kind regards, Michał On 13 October 2017 at 20:55, Sven Ludwig <s_lud...@gmx.de> wrote: > Hello, > > with recent Kafka, would it be okay to have about 1000 topics, with > between 1000 to 3000 partitions each, on a 6-node Kafka cluster with > replication factor 3? > > Each partition would be written to by one producer and consumed by one > consumer, with about 2 messages per minute coming in. > > Would Kafka and its management overhead be okay given the large number of > partitions in the millions? > > Kind Regards, > Sven > >
