Thanks will take a look. On Friday, October 24, 2014, Neha Narkhede <neha.narkh...@gmail.com> wrote:
> Is there a better way to do this where I can build one big requests with > broker -> partition mappings and call a consumer.fetch() with one giant > request? > > Unfortunately, not until the 0.9 consumer is released. Until then, you can > take a look at ZookeeperConsumerConnector on how it manages multiple > SimpleConsumers. > > Thanks, > Neha > > On Fri, Oct 24, 2014 at 1:46 PM, Rajiv Kurian <ra...@signalfuse.com > <javascript:;>> wrote: > > > I saw the example at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > > and it answered most of my questions. I am still trying to figure out the > > pattern to be used when I want to use a single simple consumer to fetch > > messages from different partitions (possibly from different offsets) and > > possibly managed by different leaders. > > > > My use case right now is that I have a consumer which dynamically decides > > which partitions it is responsible for and decides to fetch messages from > > them at potentially different offsets. > > > > Right now it seems like I would need a new SimpleConsumer for each broker > > since the SimpleConsumer takes the leadBroker in it's constructor. Then > I'd > > have to build a FetchRequest for each broker and ensure that the > > addFetch(...) calls are made with partitions that correspond to the > leader > > broker that a SimpleConsumer is managing. Finally I'll need to make up to > > numBrokers number of consumer.fetch(req) calls since each request is for > a > > separate broker. > > > > Is there a better way to do this where I can build one big requests with > > broker -> partition mappings and call a consumer.fetch() with one giant > > request? Otherwise if I am doing this in a single thread I have head of > > line blocking with one request blocking another. > > > > Thanks! > > >
