> On Feb 22, 2017, at 1:02 PM, Sunil Patil <sunilpa...@mapr.onmicrosoft.com> > wrote: > > Hi Antony, > > > 1) Kafka only allows one consumer per partition, to guarantee order within > partition
This is not true. Any number of Consumers can read data from a given Partition. It’s true that within a single Consumer Group, only one Consumer will read from the Partition, but the blanket statement of ‘one consumer per partition’ is definitely not correct. Ian. > > 2) Only one Kafka broker can be leader for one partition. > > > Based on these 2 building blocks you could split your topic into number of > partitions and that way kafka server load gets distributed across different > nodes. You could have multiple consumers running on different machines with > single consumer in every group (I am assuming you need lot of consumers but > the amount of data consumption per consumer is not huge so you dont need > group) > > > Sunil > > ________________________________ > From: Antony Vo <antony.van...@gmail.com> > Sent: Wednesday, February 22, 2017 9:17:13 AM > To: users@kafka.apache.org > Subject: Scaling To Many Kafka Consumers For Particular Topic > > Hello, > > I am currently using a single publisher to publish to a single topic with a > single partition. I would like to support many consumers that listen to > the same data and cache the topic's data in-memory. What would be the best > approach for scaling to potentially hundreds of consumers for this data? > Right now, it seems that if I use a separate consumer group for each cache, > they would all go to the same leader for the partition and potentially > overload that node. > > Is the solution to separate out into multiple partitions for the topic so > that each partition's node is serving out a smaller amount of data? Or > maybe duplicating the data across multiple topics could work? Is it > potentially the case that Kafka isn't built to support an arbitrary number > of consumers (each consuming the same data) for a given topic? Suggestions > appreciated! > > Thanks, > Antony