That depends on the data and your consumers. It’s not necessarily an issue at all. That said, we prefer to have consumers read off a single topic but our data is high frequency and easily hashed across partitions so we have hundreds of consumers reading off some topics.
I would try to partition your topics on some key that gives you a balance across consumers before deciding that you need hundreds of topics. > On Jul 11, 2016, at 1:58 AM, Vladi Feigin <vla...@liveperson.com> wrote: > > thank you for the answer > > Is there any kind of limit on a number of topics in Kafka? > If one consumer (can not do more than one consumer in a group) reads lets say > 100 topics it can be an issue, isn't it? > > > > On Sun, Jul 10, 2016 at 6:53 PM, <r...@chartbeat.com > <mailto:r...@chartbeat.com>> wrote: > Without knowing much else about your requirements - Yes, your consumers > should not be listening to topics they are not interested in. There's nothing > wrong with having more topics. > > > On Jul 10, 2016, at 10:37 AM, Vladi Feigin <vla...@liveperson.com > > <mailto:vla...@liveperson.com>> wrote: > > > > Hi All, > > > > Will appreciate if you share your thoughts based on your experience > > regarding the following question: > > What's a recommended strategy for Kafka multiple topics? > > We have just a few topics and hundreds of event types and a lot of producers > > Having just a few topics causes our consumers (many of them just need a > > couple of event types to process) to work very hard > > So we're planing moving to multi-topics > > Do you think having hundreds of topics (topic per event type) is a good > > approach? > > Or maybe is better to create topic per a group of event types based on a > > business relationship? > > > > > > Thank you in advance, > > Vladi > > > > -- > > This message may contain confidential and/or privileged information. > > If you are not the addressee or authorized to receive this on behalf of the > > addressee you must not use, copy, disclose or take action based on this > > message or any information herein. > > If you have received this message in error, please advise the sender > > immediately by reply email and delete this message. Thank you. > > > > -- > > Vladi Feigin > R&D System Architect > T: +972-74-700-4376 > <http://www.linkedin.com/company/164748> > <http://twitter.com/liveperson> > <http://www.facebook.com/LivePersonInc> We Create Meaningful > Connections > <http://roia.biz/im/n/ndiXvq1BAAGhL0MAABW7QgABwExmMQA-A/> > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of the > addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you.
signature.asc
Description: Message signed with OpenPGP using GPGMail
