Great, that's reassuring! What's the time frame for having a more or less stable version to try out?
Jason On Mon, Jul 7, 2014 at 12:59 PM, Guozhang Wang <wangg...@gmail.com> wrote: > I see your point now. The old consumer does have a hard-coded > "round-robin-per-topic" logic which have this issue. In the new consumer, > we will make the assignment logic customizable so that people can specify > different rebalance algorithms they like. > > Also I will soon send out a new consumer design summary email for more > comments. Feel free to give us more thoughts you have about the new > consumer design. > > Guozhang > > > On Mon, Jul 7, 2014 at 8:44 AM, Jason Rosenberg <j...@squareup.com> wrote: > > > Guozhang, > > > > I'm not suggesting we parallelize within a partition.... > > > > The problem with the current high-level consumer is, if you use a regex > to > > select multiple topics, and then have multiple consumers in the same > group, > > usually the first consumer will 'own' all the topics, and no amount of > > sub-sequent rebalancing will allow other consumers in the group to own > some > > of the topics. Re-balancing does allow other consumers to own multiple > > partitions, but if a topic has only 1 partition, only the first consumer > to > > initialize will get all the work. > > > > So, I'm wondering if the new api will be better about re-balancing the > work > > at the partition level, and not the topic level, as such. > > > > Jason > > > > > > On Mon, Jul 7, 2014 at 11:17 AM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hi Jason, > > > > > > In the new design the consumption is still at the per-partition > > > granularity. The main rationale of doing this is ordering: Within a > > > partition we want to preserve the ordering such that message B produced > > > after message A will also be consumed and processed after message A. > And > > > producers can use keys to make sure messages with the same ordering > group > > > will be in the same partition. To do this we have to make one partition > > > only being consumed by a single client at a time. On the other hand, > when > > > one wants to add the number of consumers beyond the number of > partitions, > > > he can always use the topic tool to dynamically add more partitions to > > the > > > topic. > > > > > > Do you have a specific scenario in mind that would require > > single-partition > > > topics? > > > > > > Guozhang > > > > > > > > > > > > On Mon, Jul 7, 2014 at 7:43 AM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > > > I've been looking at the new consumer api outlined here: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > > > > > > > One issue in the current high-level consumer, is that it does not do > a > > > good > > > > job of distributing a set of topics between multiple consumers, > unless > > > each > > > > topic has multiple partitions. This has always seemed strange to me, > > > since > > > > at the end of the day, even for single partition topics, the basic > unit > > > of > > > > consumption is still at the partition level (so you'd expect > > rebalancing > > > to > > > > try to evenly distribute partitions (regardless of the topic)). > > > > > > > > It's not clearly spelled out in the new consumer api wiki, so I'll > just > > > > ask, will this issue be addressed in the new api? I think I've asked > > > this > > > > before, but I wanted to go check again, and am not seeing this > > explicitly > > > > addressed in the design. > > > > > > > > Thanks > > > > > > > > Jason > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
