Re: HighLevelConsumer question

2014-04-14 Thread Clark Breyman
Many thanks Joel. On Mon, Apr 14, 2014 at 11:27 AM, Joel Koshy wrote: > > > > A follow-on question: what is the fairness policy when a single stream > > serves multiple topic-partitions? At the chunk level? The reason I ask is > > that I'm trying to manage processing latency across partitions.

Re: HighLevelConsumer question

2014-04-14 Thread Joel Koshy
> > A follow-on question: what is the fairness policy when a single stream > serves multiple topic-partitions? At the chunk level? The reason I ask is > that I'm trying to manage processing latency across partitions. The consumer will issue multi-fetch requests across all the brokers that it cons

Re: HighLevelConsumer question

2014-04-11 Thread Clark Breyman
Thanks Joel - I now see this working locally. I had conflated the LinkedBlockingQueue "channel" with an io channel. A follow-on question: what is the fairness policy when a single stream serves multiple topic-partitions? At the chunk level? The reason I ask is that I'm trying to manage processing

Re: HighLevelConsumer question

2014-04-11 Thread Joel Koshy
A single stream (or consumer iterator) receive data from multiple partitions. Thanks, Joel On Fri, Apr 11, 2014 at 01:37:23PM -0700, Clark Breyman wrote: > I was under the impression that a KafkaStream would only own a single > topic/partition at a time. Is this correct or will it multiple multi