> On Feb 22, 2017, at 1:02 PM, Sunil Patil <sunilpa...@mapr.onmicrosoft.com> 
> wrote:
> 
> Hi Antony,
> 
> 
> 1) Kafka only allows one consumer per partition, to guarantee order within 
> partition

This is not true. Any number of Consumers can read data from a given Partition. 
It’s true that within a single Consumer Group, only one Consumer will read from 
the Partition, but the blanket statement of ‘one consumer per partition’ is 
definitely not correct.

Ian.

> 
> 2) Only one Kafka broker can be leader for one partition.
> 
> 
> Based on these 2 building blocks you could split your topic into number of 
> partitions and that way kafka server load gets distributed across different 
> nodes. You could have multiple consumers running on different machines with 
> single consumer in every group (I am assuming you need lot of consumers but 
> the amount of data consumption per consumer is not huge so you dont need 
> group)
> 
> 
> Sunil
> 
> ________________________________
> From: Antony Vo <antony.van...@gmail.com>
> Sent: Wednesday, February 22, 2017 9:17:13 AM
> To: users@kafka.apache.org
> Subject: Scaling To Many Kafka Consumers For Particular Topic
> 
> Hello,
> 
> I am currently using a single publisher to publish to a single topic with a
> single partition.  I would like to support many consumers that listen to
> the same data and cache the topic's data in-memory.  What would be the best
> approach for scaling to potentially hundreds of consumers for this data?
> Right now, it seems that if I use a separate consumer group for each cache,
> they would all go to the same leader for the partition and potentially
> overload that node.
> 
> Is the solution to separate out into multiple partitions for the topic so
> that each partition's node is serving out a smaller amount of data?  Or
> maybe duplicating the data across multiple topics could work?  Is it
> potentially the case that Kafka isn't built to support an arbitrary number
> of consumers (each consuming the same data) for a given topic?  Suggestions
> appreciated!
> 
> Thanks,
> Antony

Reply via email to