Thanks Joel - I now see this working locally. I had conflated the LinkedBlockingQueue "channel" with an io channel.
A follow-on question: what is the fairness policy when a single stream serves multiple topic-partitions? At the chunk level? The reason I ask is that I'm trying to manage processing latency across partitions. Is the following solution a bad idea? - for each host: create T threads (proportional to processor count) - for each thread: -- create a consumer with partitionCount/(hostCount*threadsPerHost) streams -- each of the streams with "consumer.timeout.ms" set to 0 (adding a pause when no messages on all streams polled) I need true at-least-once semantics, so auto commit is false. Thanks again. Clark On Fri, Apr 11, 2014 at 5:35 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > A single stream (or consumer iterator) receive data from multiple > partitions. > > Thanks, > > Joel > > On Fri, Apr 11, 2014 at 01:37:23PM -0700, Clark Breyman wrote: > > I was under the impression that a KafkaStream would only own a single > > topic/partition at a time. Is this correct or will it multiple multiple > > topic-partitions into a single KafkaStream? > > > > Thanks, > > Clark > >
