Hi, Everyone, Sorry for being late on this thread. I just came across this thread. I have a couple of concerns on this. (1) It seems the amount of delay will be application specific. So, it seems that it's better for the delay to be a client side config instead of a server side one? (2) When running console consumer in quickstart, a minimum of 3 sec delay seems to be a bad experience for our users.
Since we are getting late into the release cycle, it may be a bit too late to make big changes in the 0.11 release. Perhaps we should at least consider overriding the delay in config/server.properties to 0 to improve the quickstart experience? Thanks, Jun On Tue, Apr 11, 2017 at 12:19 AM, Damian Guy <damian....@gmail.com> wrote: > Hi Onur, > > It was in my previous email. But here it is again. > > ============================================================ > > 1. Better rebalance timing. We will try to rebalance only when all the > consumers in a group have joined. The challenge would be someone has to > define what does ALL consumers mean, it could either be a time or number of > consumers, etc. > > 2. Avoid frequent rebalance. For example, if there are 100 consumers in a > group, today, in the worst case, we may end up with 100 rebalances even if > all the consumers joined the group in a reasonably small amount of time. > Frequent rebalance is also a bad thing for brokers. > > Having a client side configuration may solve problem 1 better because each > consumer group can potentially configure their own timing. However, it does > not really prevent frequent rebalance in general because some of the > consumers can be misconfigured. (This may have something to do with KIP-124 > as well. But if quota is applied on the JoinGroup/SyncGroup request it may > cause some unwanted cascading effects.) > > Having a broker side configuration may result in less flexibility for each > consumer group, but it can prevent frequent rebalance better. I think with > some reasonable design, the rebalance timing issue can be resolved on the > broker side as well. Matthias had a good point on extending the delay when > a new consumer joins a group (we actually did something similar to batch > ISR change propagation). For example, let's say on the broker side, we will > always delay 2 seconds each time we see a new consumer joining a consumer > group. This would probably work for most of the consumer groups and will > also limit the rebalance frequency to protect the brokers. > > I am not sure about the streams use case here, but if something like 2 > seconds of delay is acceptable for streams, I would prefer adding the > configuration to the broker so that we can address both problems. > > On Thu, 6 Apr 2017 at 17:11 Onur Karaman <onurkaraman.apa...@gmail.com> > wrote: > > > Hi Damian. > > > > Can you copy the point Becket made earlier that you say isn't addressed? > > > > On Thu, Apr 6, 2017 at 2:51 AM, Damian Guy <damian....@gmail.com> wrote: > > > > > Thanks all, the Vote is now closed and the KIP has been accepted with 9 > > +1s > > > > > > 3 binding:: > > > Guozhang, > > > Jason, > > > Ismael > > > > > > 6 non-binding: > > > Bill, > > > Eno, > > > Mathieu, > > > Matthias, > > > Dong, > > > Mickael > > > > > > Thanks, > > > Damian > > > > > > On Thu, 6 Apr 2017 at 09:26 Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > Thanks for the KIP, +1 (binding). > > > > > > > > Ismael > > > > > > > > On Thu, Mar 30, 2017 at 8:55 PM, Jason Gustafson <ja...@confluent.io > > > > > > wrote: > > > > > > > > > +1 Thanks for the KIP! > > > > > > > > > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang < > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > Sorry about the previous email, Gmail seems be collapsing them > > into a > > > > > > single thread on my inbox. > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang < > > wangg...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Damian, could you create a new thread for the voting process? > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck < > bbej...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > >> +1(non-binding) > > > > > > >> > > > > > > >> On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska < > > > > eno.there...@gmail.com > > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >> > +1 (non binding) > > > > > > >> > > > > > > > >> > Thanks > > > > > > >> > Eno > > > > > > >> > > On 30 Mar 2017, at 18:01, Matthias J. Sax < > > > > matth...@confluent.io> > > > > > > >> wrote: > > > > > > >> > > > > > > > > >> > > +1 > > > > > > >> > > > > > > > > >> > > On 3/30/17 3:46 AM, Damian Guy wrote: > > > > > > >> > >> Hi All, > > > > > > >> > >> > > > > > > >> > >> I'd like to start the voting thread on KIP-134: > > > > > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > >> > 134%3A+Delay+initial+consumer+group+rebalance > > > > > > >> > >> > > > > > > >> > >> Thanks, > > > > > > >> > >> Damian > > > > > > >> > >> > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > >
