The high level API gives you access to the raw Kafka event which is MessageAndMetadata, which has two pieces: key and actual message.
Best Regards, -Jonathan On Aug 18, 2014, at 9:50 AM, Josh J <joshjd...@gmail.com> wrote: >> One tactic that might be worth exploring is to rely on the message key to > facilitate this. > >> It would require engineering careful functions for the key which hashes > to the partitions for your topic(s). It would also mean that your consumers > for the topic would be evaluating the key and discarding messages that > aren’t relevant. > >> The only other option I can think of if you are using the high-level API > would be finer-grained topics. > > Yes, this works great ! My producer is bucketing the messages based on the > key (the key is a timestamp and I simply mod the timestamp as a function of > the number of buckets). I can then do the same function on the consumer > when it reads the key. I'm essentially implementing consumer sliding > window. Any suggestions or tips on where I would implement reading the > message key? > > Thanks, > Josh > > > On Mon, Aug 18, 2014 at 6:43 PM, Jonathan Weeks <jonathanbwe...@gmail.com> > wrote: > >> One tactic that might be worth exploring is to rely on the message key to >> facilitate this. >> >> It would require engineering careful functions for the key which hashes to >> the partitions for your topic(s). It would also mean that your consumers >> for the topic would be evaluating the key and discarding messages that >> aren’t relevant. >> >> The only other option I can think of if you are using the high-level API >> would be finer-grained topics. >> >> Best Regards, >> >> -Jonathan >> >> On Aug 18, 2014, at 9:14 AM, Josh J <joshjd...@gmail.com> wrote: >> >>> Is it possible to modify and use the high level consumer so that I can >>> ignore processing certain partitions? >>> >>> >>> On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sharnin...@gmail.com> >> wrote: >>> >>>> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <joshjd...@gmail.com> wrote: >>>> >>>>>> You can see an example of using the SimpleConsumer here >>>>> < >>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example >>>>>> >>>>> >>>>> Any suggestions on where in the code to modify the high level producer >> to >>>>> support reading from specific partitions ? >>>>> >>>>> >>>> High level producer? I'm assuming you meant to write the high level >>>> consumer, in which case it isn't possible. The link above, which has an >>>> example for reading messages off a specific partition, is for the Simple >>>> consumer, which ironically, is more complex than the high level >> consumer. >>>> >>>> In short, if you have a usecase where you want to read from a specific >>>> partition, you will need to implement a simple consumer. >>>> >>>> -- >>>> Sharninder >>>> >>>> Josh >>>>> . >>>>> >>>>> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede < >> neha.narkh...@gmail.com> >>>>> wrote: >>>>> >>>>>> You can see an example of using the SimpleConsumer here >>>>>> < >>>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example >>>>>>> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sharnin...@gmail.com> >>>>> wrote: >>>>>> >>>>>>> Implement the low level "Simple Consumer". >>>>>>> >>>>>>> -- >>>>>>> Sharninder >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <joshjd...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Suppose I have N partitions. I would like to have X different >>>>> consumer >>>>>>>> threads ( X < N) read from a specified set of partitions. How can I >>>>>>> achieve >>>>>>>> this? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Josh >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> >>
