Just recapping on client-side v.s. broker-side config: we did discuss about adding this as a client-side config and bump up join-group request (I think both Ismael and Ewen questioned about it) to include this configured value to the broker. I cannot remember if there is any strong motivations against going to the client-side config, except that we felt a default non-zero value will benefit most users assuming they start with more than one member in their group but only advanced users would really realize this config existing and tune it themselves.
I agree that we could re-consider it for the next release if we observe that it is actually affecting more users than benefiting them. Guozhang On Wed, Jun 7, 2017 at 2:26 AM, Damian Guy <damian....@gmail.com> wrote: > Hi Jun/Ismael, > > Sounds good to me. > > Thanks, > Damian > > On Tue, 6 Jun 2017 at 23:08 Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Jun, > > > > The console consumer issue also came up in a conversation I was having > > recently. Seems like the config/server.properties change is a reasonable > > compromise given that we have other defaults that are for development. > > > > Ismael > > > > On Tue, Jun 6, 2017 at 10:59 PM, Jun Rao <j...@confluent.io> wrote: > > > > > Hi, Everyone, > > > > > > Sorry for being late on this thread. I just came across this thread. I > > have > > > a couple of concerns on this. (1) It seems the amount of delay will be > > > application specific. So, it seems that it's better for the delay to > be a > > > client side config instead of a server side one? (2) When running > console > > > consumer in quickstart, a minimum of 3 sec delay seems to be a bad > > > experience for our users. > > > > > > Since we are getting late into the release cycle, it may be a bit too > > late > > > to make big changes in the 0.11 release. Perhaps we should at least > > > consider overriding the delay in config/server.properties to 0 to > improve > > > the quickstart experience? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Tue, Apr 11, 2017 at 12:19 AM, Damian Guy <damian....@gmail.com> > > wrote: > > > > > > > Hi Onur, > > > > > > > > It was in my previous email. But here it is again. > > > > > > > > ============================================================ > > > > > > > > 1. Better rebalance timing. We will try to rebalance only when all > the > > > > consumers in a group have joined. The challenge would be someone has > to > > > > define what does ALL consumers mean, it could either be a time or > > number > > > of > > > > consumers, etc. > > > > > > > > 2. Avoid frequent rebalance. For example, if there are 100 consumers > > in a > > > > group, today, in the worst case, we may end up with 100 rebalances > even > > > if > > > > all the consumers joined the group in a reasonably small amount of > > time. > > > > Frequent rebalance is also a bad thing for brokers. > > > > > > > > Having a client side configuration may solve problem 1 better because > > > each > > > > consumer group can potentially configure their own timing. However, > it > > > does > > > > not really prevent frequent rebalance in general because some of the > > > > consumers can be misconfigured. (This may have something to do with > > > KIP-124 > > > > as well. But if quota is applied on the JoinGroup/SyncGroup request > it > > > may > > > > cause some unwanted cascading effects.) > > > > > > > > Having a broker side configuration may result in less flexibility for > > > each > > > > consumer group, but it can prevent frequent rebalance better. I think > > > with > > > > some reasonable design, the rebalance timing issue can be resolved on > > the > > > > broker side as well. Matthias had a good point on extending the delay > > > when > > > > a new consumer joins a group (we actually did something similar to > > batch > > > > ISR change propagation). For example, let's say on the broker side, > we > > > will > > > > always delay 2 seconds each time we see a new consumer joining a > > consumer > > > > group. This would probably work for most of the consumer groups and > > will > > > > also limit the rebalance frequency to protect the brokers. > > > > > > > > I am not sure about the streams use case here, but if something like > 2 > > > > seconds of delay is acceptable for streams, I would prefer adding the > > > > configuration to the broker so that we can address both problems. > > > > > > > > On Thu, 6 Apr 2017 at 17:11 Onur Karaman < > onurkaraman.apa...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi Damian. > > > > > > > > > > Can you copy the point Becket made earlier that you say isn't > > > addressed? > > > > > > > > > > On Thu, Apr 6, 2017 at 2:51 AM, Damian Guy <damian....@gmail.com> > > > wrote: > > > > > > > > > > > Thanks all, the Vote is now closed and the KIP has been accepted > > > with 9 > > > > > +1s > > > > > > > > > > > > 3 binding:: > > > > > > Guozhang, > > > > > > Jason, > > > > > > Ismael > > > > > > > > > > > > 6 non-binding: > > > > > > Bill, > > > > > > Eno, > > > > > > Mathieu, > > > > > > Matthias, > > > > > > Dong, > > > > > > Mickael > > > > > > > > > > > > Thanks, > > > > > > Damian > > > > > > > > > > > > On Thu, 6 Apr 2017 at 09:26 Ismael Juma <ism...@juma.me.uk> > wrote: > > > > > > > > > > > > > Thanks for the KIP, +1 (binding). > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 8:55 PM, Jason Gustafson < > > > ja...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 Thanks for the KIP! > > > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang < > > > > wangg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > Sorry about the previous email, Gmail seems be collapsing > > them > > > > > into a > > > > > > > > > single thread on my inbox. > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang < > > > > > wangg...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Damian, could you create a new thread for the voting > > process? > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck < > > > > bbej...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> +1(non-binding) > > > > > > > > > >> > > > > > > > > > >> On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska < > > > > > > > eno.there...@gmail.com > > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > >> > +1 (non binding) > > > > > > > > > >> > > > > > > > > > > >> > Thanks > > > > > > > > > >> > Eno > > > > > > > > > >> > > On 30 Mar 2017, at 18:01, Matthias J. Sax < > > > > > > > matth...@confluent.io> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > +1 > > > > > > > > > >> > > > > > > > > > > > >> > > On 3/30/17 3:46 AM, Damian Guy wrote: > > > > > > > > > >> > >> Hi All, > > > > > > > > > >> > >> > > > > > > > > > >> > >> I'd like to start the voting thread on KIP-134: > > > > > > > > > >> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > > > >> > 134%3A+Delay+initial+consumer+group+rebalance > > > > > > > > > >> > >> > > > > > > > > > >> > >> Thanks, > > > > > > > > > >> > >> Damian > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- -- Guozhang
