Hi Aruna, On 1/10/19 8:19 AM, aruna ramachandran wrote:
I am using keyed partitions with 1000 partitions, so I need to create 1000 consumers because consumers groups and re balancing concepts is not worked in the case of manually assigned consumers.Is there any replacement for the above problem.
What API are you using in the KafkaConsumer? Are you using subscribe(Collection<String> topics) or are you using assign(Collection<TopicPartition> partitions) ?
The 1st one (subscribe) is the one you should be using for your usecase. With that call, when you subscribe to a multi-partition topic and you have multiple KafkaConsumer(s) configured with the same consumer group id, then partitions of the topic are dynamically assigned (and possibly reassigned when consumers come or go) to a set of live consumers. Will this work for you (and why not)?
Regards, Peter
