Hello Suchi, Each topic will have its own partitions and each partition is represented as a list of file segments. Hence having multiple smaller topics means you will have # topics * avg. # partitions_per_topic file handlers.
Volume is usually not a problem for Kafka topics, however, if you have consumers consuming not at relatively the tail (i.e., pretty caught up with the produce rate) the it will hit the disk but not the file cache. This will incur large latency not only for these consumers but also for others since they are sharing the socket server processors. So if all your producer/consumers of the topic have roughly the same consuming behavior, you can go option 1; otherwise it is better to keep groups of similar produce/consume behavioral cases separate topics. Guozhang On Mon, Oct 28, 2013 at 12:24 PM, Suchi Amalapurapu <su...@bloomreach.com>wrote: > We are using kafka 0.7.2 > Suchi > > > On Tue, Oct 29, 2013 at 12:53 AM, Suchi Amalapurapu <su...@bloomreach.com > >wrote: > > > All > > Are there any limitations on max topic size in Kafka. How large can a > > topic get? > > What are the implications on resources, latencies, performance and > > feasability? > > What are the pros and cons of using a single large topic vs multiple > > smaller topics? > > Suchi > > > -- -- Guozhang
