Ian, 1500 partitions for a topic should be fine, assuming that you don't have too many topics. In general, the more partitions, the more open file handles are required in broker and the more more space is required in ZK.
Thanks, Jun On Tue, Mar 19, 2013 at 2:53 PM, Ian Friedman <i...@flurry.com> wrote: > Hello, I am a complete newbie to Kafka and am trying to evaluate its > usefulness for our particular application. I plan to have a lot of > consumers in a single group, and it seems like the best way to load balance > messages across consumers without knowing ahead of time exactly how many > consumers you will have is to simply have a large number of partitions so > that each consumer takes a significant portion of them. > > This line of thought has lead me to the following question: What is the > potential overhead from creating a large number of partitions on a topic? > > We would probably have at least 300 consumers to start and want to have > room to grow. And I'd probably want a number of partitions about 5x the > number of consumers to absorb consumers dying/being brought online. Should > I be concerned about this in terms of memory overhead or performance? > > Thanks, > Ian Friedman > >
