Many thanks Joel.
On Mon, Apr 14, 2014 at 11:27 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > A follow-on question: what is the fairness policy when a single stream > > serves multiple topic-partitions? At the chunk level? The reason I ask is > > that I'm trying to manage processing latency across partitions. > > The consumer will issue multi-fetch requests across all the brokers > that it consumes from. i.e., it will fetch chunks (if available) from > all the brokers and put them into "chunk-queues" that the consumer > stream grabs as you iterate over messages. > > There is one issue to be aware of: if any of your consumer iterators > exit due to some error, a chunk queue can become full and effectively > block progress on all partitions. > > Your solution looks reasonable if you have enough partitions to spread > across your instances. Also, you should set the consumer timeout to a > reasonable value (depending on your anticipated message rate). > > Joel > > > > > Is the following solution a bad idea? > > - for each host: create T threads (proportional to processor count) > > - for each thread: > > -- create a consumer with partitionCount/(hostCount*threadsPerHost) > streams > > -- each of the streams with "consumer.timeout.ms" set to 0 (adding a > pause > > when no messages on all streams polled) > > > > I need true at-least-once semantics, so auto commit is false. > > > > Thanks again. > > Clark > > > > > > > > On Fri, Apr 11, 2014 at 5:35 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > > A single stream (or consumer iterator) receive data from multiple > > > partitions. > > > > > > Thanks, > > > > > > Joel > > > > > > On Fri, Apr 11, 2014 at 01:37:23PM -0700, Clark Breyman wrote: > > > > I was under the impression that a KafkaStream would only own a single > > > > topic/partition at a time. Is this correct or will it multiple > multiple > > > > topic-partitions into a single KafkaStream? > > > > > > > > Thanks, > > > > Clark > > > > > > > >
