Hi, One option is to partition the data using key and consume from relevant partition. Or your current approach (filtering messages in the application) should be OK.
Using separate getMetaData/getkey and getMessage may hit the consumer performance/throughput. Regards, Kumar On Fri, Jan 2, 2015 at 9:53 PM, Mukesh Jha <me.mukesh....@gmail.com> wrote: > Any pointers guys? > On 1 Jan 2015 15:26, "Mukesh Jha" <me.mukesh....@gmail.com> wrote: > > > Hello Experts, > > > > I'm using a kafka topic to store bunch of messages where the key contains > > metadata and value is the data (avro file in our case). > > There are multiple consumers for each topic and the consumer can decide > if > > the message is relevant for it or not based on the metadata i.e. the key > of > > the message. > > Using this a group-consumer can check the key if the message is required > > by it and then it can retrieve the entire message otherwise it'll just > > commit the offset and move on to the next message. > > > > So I was wondering if kafka has an api in kafka that lets the consumer to > > get just the metadata i.e. key, something like getMetadata instead > > of getMessageAndMetadata? > > > > If kafka has something like this can you help me out with some > > documentation for the same? > > I think this will be useful in a lot of scenarios so if its not there I > > can file a JIRA and take a dig at it. Let me know what you all think. > > > > Thanks for your help & suggestions. > > > > -- > > Thanks & Regards, > > > > *Mukesh Jha <me.mukesh....@gmail.com>* > > >
