> One tactic that might be worth exploring is to rely on the message key to facilitate this.
> It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant. > The only other option I can think of if you are using the high-level API would be finer-grained topics. Yes, this works great ! My producer is bucketing the messages based on the key (the key is a timestamp and I simply mod the timestamp as a function of the number of buckets). I can then do the same function on the consumer when it reads the key. I'm essentially implementing consumer sliding window. Any suggestions or tips on where I would implement reading the message key? Thanks, Josh On Mon, Aug 18, 2014 at 6:43 PM, Jonathan Weeks <jonathanbwe...@gmail.com> wrote: > One tactic that might be worth exploring is to rely on the message key to > facilitate this. > > It would require engineering careful functions for the key which hashes to > the partitions for your topic(s). It would also mean that your consumers > for the topic would be evaluating the key and discarding messages that > aren’t relevant. > > The only other option I can think of if you are using the high-level API > would be finer-grained topics. > > Best Regards, > > -Jonathan > > On Aug 18, 2014, at 9:14 AM, Josh J <joshjd...@gmail.com> wrote: > > > Is it possible to modify and use the high level consumer so that I can > > ignore processing certain partitions? > > > > > > On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sharnin...@gmail.com> > wrote: > > > >> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <joshjd...@gmail.com> wrote: > >> > >>>> You can see an example of using the SimpleConsumer here > >>> < > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > >>>> > >>> > >>> Any suggestions on where in the code to modify the high level producer > to > >>> support reading from specific partitions ? > >>> > >>> > >> High level producer? I'm assuming you meant to write the high level > >> consumer, in which case it isn't possible. The link above, which has an > >> example for reading messages off a specific partition, is for the Simple > >> consumer, which ironically, is more complex than the high level > consumer. > >> > >> In short, if you have a usecase where you want to read from a specific > >> partition, you will need to implement a simple consumer. > >> > >> -- > >> Sharninder > >> > >> Josh > >>> . > >>> > >>> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede < > neha.narkh...@gmail.com> > >>> wrote: > >>> > >>>> You can see an example of using the SimpleConsumer here > >>>> < > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > >>>>> > >>>> . > >>>> > >>>> > >>>> > >>>> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sharnin...@gmail.com> > >>> wrote: > >>>> > >>>>> Implement the low level "Simple Consumer". > >>>>> > >>>>> -- > >>>>> Sharninder > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <joshjd...@gmail.com> wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> Suppose I have N partitions. I would like to have X different > >>> consumer > >>>>>> threads ( X < N) read from a specified set of partitions. How can I > >>>>> achieve > >>>>>> this? > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Josh > >>>>>> > >>>>> > >>>> > >>> > >> > >
