Aruna, Instead of using 1000 partitions you can chose to create dynamic topics and have a consumer for each topic.
On Fri, Jan 11, 2019 at 6:43 AM Peter Levart <peter.lev...@gmail.com> wrote: > > > On 1/10/19 2:26 PM, Sven Ludwig wrote: > > Okay, but > > > > what if one also needs to preserve the order of messages coming from a > particular device? > > > > With Kafka, this is perhaps possible if all messages from a particular > device go into the same partition. > > > > Would it be a good and efficient solution for this approach to set the > key of each Kafka ProducerRecord to the unique ID of the Device > > Exactly! > > > AND deactivate the key-based log-cleaner on the broker so that it does > not delete older records that have the same key? > > How old records are cleaned is independent of what you do with processed > records. You usually retain them for enough time so you don't loose them > before processing them + some safety time... > > Regards, Peter > > > > > Sven > > > > > > Gesendet: Donnerstag, 10. Januar 2019 um 08:35 Uhr > > Von: "Peter Levart" <peter.lev...@gmail.com> > > An: users@kafka.apache.org, "aruna ramachandran" <arunaeie...@gmail.com> > > Betreff: Re: Doubts in Kafka > > Hi Aruna, > > > > On 1/10/19 8:19 AM, aruna ramachandran wrote: > >> I am using keyed partitions with 1000 partitions, so I need to create > 1000 > >> consumers because consumers groups and re balancing concepts is not > worked > >> in the case of manually assigned consumers.Is there any replacement for > the > >> above problem. > >> > > What API are you using in the KafkaConsumer? Are you using > > subscribe(Collection<String> topics) or are you using > > assign(Collection<TopicPartition> partitions) ? > > > > The 1st one (subscribe) is the one you should be using for your usecase. > > With that call, when you subscribe to a multi-partition topic and you > > have multiple KafkaConsumer(s) configured with the same consumer group > > id, then partitions of the topic are dynamically assigned (and possibly > > reassigned when consumers come or go) to a set of live consumers. Will > > this work for you (and why not)? > > > > Regards, Peter > >
