> > A follow-on question: what is the fairness policy when a single stream > serves multiple topic-partitions? At the chunk level? The reason I ask is > that I'm trying to manage processing latency across partitions.
The consumer will issue multi-fetch requests across all the brokers that it consumes from. i.e., it will fetch chunks (if available) from all the brokers and put them into "chunk-queues" that the consumer stream grabs as you iterate over messages. There is one issue to be aware of: if any of your consumer iterators exit due to some error, a chunk queue can become full and effectively block progress on all partitions. Your solution looks reasonable if you have enough partitions to spread across your instances. Also, you should set the consumer timeout to a reasonable value (depending on your anticipated message rate). Joel > > Is the following solution a bad idea? > - for each host: create T threads (proportional to processor count) > - for each thread: > -- create a consumer with partitionCount/(hostCount*threadsPerHost) streams > -- each of the streams with "consumer.timeout.ms" set to 0 (adding a pause > when no messages on all streams polled) > > I need true at-least-once semantics, so auto commit is false. > > Thanks again. > Clark > > > > On Fri, Apr 11, 2014 at 5:35 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > A single stream (or consumer iterator) receive data from multiple > > partitions. > > > > Thanks, > > > > Joel > > > > On Fri, Apr 11, 2014 at 01:37:23PM -0700, Clark Breyman wrote: > > > I was under the impression that a KafkaStream would only own a single > > > topic/partition at a time. Is this correct or will it multiple multiple > > > topic-partitions into a single KafkaStream? > > > > > > Thanks, > > > Clark > > > >
