With more topics, you may hit one of those limits: (1) # dirs allowed in a FS; (2) open file handlers (we keep all log segments open in the broker); (3) ZK nodes.
Thanks, Jun On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <jason.hu...@icare.com> wrote: > Just curious - I don't see any limit in the # of topics that you can > have in a Kafka cluster. > > So in principle, you could have as many topics as you want, so long as > your hardware can keep up, right? > > Jason > > On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <jun...@gmail.com> wrote: > > At LinkedIn, our largest cluster has more than 2K topics. 5K topics > should > > be fine. > > > > Thanks, > > > > Jun > > > > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <suyog....@gmail.com> wrote: > > > >> Hello, > >> > >> Wanted to check if there is any known limit on the # of topics in a > Kafka > >> cluster? I wanted to design a system which has say 5k topics and > >> multi-threaded consumers reading messages from these topics. Does anyone > >> have experience with such large topic size? I see in Kafka's page a test > >> for throughput w.r.t. to topic size, but the max topic size is 50. > >> > >> Thanks > >> >
