We've seen big performace degradation when we tested 1024 topics, so we've opted to go for a much smaller topic count (< 100).
On the read side, I think performance is largely driven by the ability of the operating system to effectively cache access to #partitions*topic files. Clearly if you divide your available memory by 1024 topics you'll have less in the file cache per topic than you would with only 3 topics (for example). On Wed, Mar 27, 2013 at 11:14 AM, Jun Rao <jun...@gmail.com> wrote: > With more topics, you may hit one of those limits: (1) # dirs allowed in a > FS; (2) open file handlers (we keep all log segments open in the broker); > (3) ZK nodes. > > Thanks, > > Jun > > On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <jason.hu...@icare.com> > wrote: > > > Just curious - I don't see any limit in the # of topics that you can > > have in a Kafka cluster. > > > > So in principle, you could have as many topics as you want, so long as > > your hardware can keep up, right? > > > > Jason > > > > On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <jun...@gmail.com> wrote: > > > At LinkedIn, our largest cluster has more than 2K topics. 5K topics > > should > > > be fine. > > > > > > Thanks, > > > > > > Jun > > > > > > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <suyog....@gmail.com> > wrote: > > > > > >> Hello, > > >> > > >> Wanted to check if there is any known limit on the # of topics in a > > Kafka > > >> cluster? I wanted to design a system which has say 5k topics and > > >> multi-threaded consumers reading messages from these topics. Does > anyone > > >> have experience with such large topic size? I see in Kafka's page a > test > > >> for throughput w.r.t. to topic size, but the max topic size is 50. > > >> > > >> Thanks > > >> > > > -- Matthew Rathbone Foursquare | Software Engineer | Server Engineering Team matt...@foursquare.com | @rathboma <http://twitter.com/rathboma> | 4sq<http://foursquare.com/rathboma>
