Re: quick question about new consumer api

2014-07-07 Thread Guozhang Wang
We plan to have a working prototype ready end of September. Guozhang On Mon, Jul 7, 2014 at 11:05 AM, Jason Rosenberg wrote: > Great, that's reassuring! > > What's the time frame for having a more or less stable version to try out? > > Jason > > > On Mon, Jul 7, 2014 at 12:59 PM, Guozhang Wang

Re: quick question about new consumer api

2014-07-07 Thread Jason Rosenberg
Great, that's reassuring! What's the time frame for having a more or less stable version to try out? Jason On Mon, Jul 7, 2014 at 12:59 PM, Guozhang Wang wrote: > I see your point now. The old consumer does have a hard-coded > "round-robin-per-topic" logic which have this issue. In the new co

Re: quick question about new consumer api

2014-07-07 Thread Guozhang Wang
I see your point now. The old consumer does have a hard-coded "round-robin-per-topic" logic which have this issue. In the new consumer, we will make the assignment logic customizable so that people can specify different rebalance algorithms they like. Also I will soon send out a new consumer desig

Re: quick question about new consumer api

2014-07-07 Thread Jason Rosenberg
Guozhang, I'm not suggesting we parallelize within a partition The problem with the current high-level consumer is, if you use a regex to select multiple topics, and then have multiple consumers in the same group, usually the first consumer will 'own' all the topics, and no amount of sub-sequ

Re: quick question about new consumer api

2014-07-07 Thread Guozhang Wang
Hi Jason, In the new design the consumption is still at the per-partition granularity. The main rationale of doing this is ordering: Within a partition we want to preserve the ordering such that message B produced after message A will also be consumed and processed after message A. And producers c