Hey friends, we will be closing this vote thread since we already got 3 binding votes (Guozhang, Harsha and Jason), and 2 non-binding votes (Stanislav, Mayuresh). Thanks again for everyone who have been actively contributing to this thread (Dong, Matthias, Colin, Mike, John, Konstantine), this has been a really exciting journey and I have learnt a lot through the whole process!
For next steps, we will further separate the KStream changes of exposing consumer info in a separate KIP, which will be led by Guozhang. I shall discuss with with Matthias for the rollout plan since he will be managing 2.2 release and hope to get some work done before the deadline. Feel free to keep raising comments or questions on the discussion thread, I will reply them at earliest convenience. Best, Boyang ________________________________ From: Boyang Chen <bche...@outlook.com> Sent: Saturday, January 5, 2019 8:58 AM To: dev@kafka.apache.org; dev@kafka.apache.org Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances Thanks so much Harsha and Jason! Will address the comment and make it a tuple. Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Jason Gustafson <ja...@confluent.io> Sent: Friday, January 4, 2019 3:50 PM To: dev Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances Hey Boyang, I had just a follow-up to my comment above. I wanted to suggest an alternative schema for LeaveGroup: LeaveGroupRequest => GroupId [GroupInstanceId MemberId] So we have a single array instead of two arrays. Each element identifies a single member. For dynamic members, we would use GroupInstanceId="" and provide the dynamic MemberId, which is consistent with JoinGroup. For static members, GroupInstanceId must be provided and Member could be considered optional. I think this makes the schema more coherent, but I'll leave it to you if there is a good reason to keep them separate. In any case, my vote is +1. Thanks for the hard work on this KIP! Best, Jason On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen <bche...@outlook.com> wrote: > Thanks Guozhang for the proposal! The update is done. > > ________________________________ > From: Guozhang Wang <wangg...@gmail.com> > Sent: Saturday, January 5, 2019 3:33 AM > To: dev > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > reduce consumer rebalances > > Hello Boyang, > > I've made another pass on the wiki page again. One minor comment is on the > "Server Behavior Changes" section, we should have a paragraph on the > logical changes on handling new versions of LeaveGroupRequest (e.g. how to > handle dynamic member v.s. static member etc). > > Other than that, I do not have further comments. I think we can continue > the voting process after that. > > Guozhang > > On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen <bche...@outlook.com> wrote: > > > Thanks Jason for the comment! I answered it on the discuss thread. > > > > Folks, could we continue the vote for this KIP? This is a very critical > > improvement for our streaming system > > stability and we need to get things rolling right at the start of 2019. > > > > Thank you for your time! > > Boyang > > > > ________________________________ > > From: Jason Gustafson <ja...@confluent.io> > > Sent: Tuesday, December 18, 2018 7:40 AM > > To: dev > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > reduce consumer rebalances > > > > Hi Boyang, > > > > Thanks, the KIP looks good. Just one comment. > > > > The new schema for the LeaveGroup request is slightly odd since it is > > handling both the single consumer use case and the administrative use > case. > > I wonder we could make it consistent from a batching perspective. > > > > In other words, instead of this: > > LeaveGroupRequest => GroupId MemberId [GroupInstanceId] > > > > Maybe we could do this: > > LeaveGroupRequest => GroupId [GroupInstanceId MemberId] > > > > For dynamic members, GroupInstanceId could be empty, which is consistent > > with JoinGroup. What do you think? > > > > Also, just for clarification, what is the expected behavior if the > current > > memberId of a static member is passed to LeaveGroup? Will the static > member > > be removed? I know the consumer will not do this, but we'll still have to > > handle the case on the broker. > > > > Best, > > Jason > > > > > > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen <bche...@outlook.com> > wrote: > > > > > Thanks Stanislav！ > > > > > > Get Outlook for iOS<https://aka.ms/o0ukef> > > > > > > ________________________________ > > > From: Stanislav Kozlovski <stanis...@confluent.io> > > > Sent: Monday, December 10, 2018 11:28 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > > reduce consumer rebalances > > > > > > This is great work, Boyang. Thank you very much. > > > > > > +1 (non-binding) > > > > > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen <bche...@outlook.com> > wrote: > > > > > > > Hey there, could I get more votes on this thread? > > > > > > > > Thanks for the vote from Mayuresh and Mike :) > > > > > > > > Best, > > > > Boyang > > > > ________________________________ > > > > From: Mayuresh Gharat <gharatmayures...@gmail.com> > > > > Sent: Thursday, December 6, 2018 10:53 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > > > reduce consumer rebalances > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger < > > > mike.freyber...@xandr.com> > > > > wrote: > > > > > > > > > +1 (non binding) > > > > > > > > > > On 12/4/18, 9:43 AM, "Patrick Williams" < > > > patrick.willi...@storageos.com > > > > > > > > > > wrote: > > > > > > > > > > Pls take me off this VOTE list > > > > > > > > > > Best, > > > > > > > > > > Patrick Williams > > > > > > > > > > Sales Manager, UK & Ireland, Nordics & Israel > > > > > StorageOS > > > > > +44 (0)7549 676279 > > > > > patrick.willi...@storageos.com > > > > > > > > > > 20 Midtown > > > > > 20 Proctor Street > > > > > Holborn > > > > > London WC1V 6NX > > > > > > > > > > Twitter: @patch37 > > > > > LinkedIn: linkedin.com/in/patrickwilliams4 < > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2F&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > On 03/12/2018, 17:34, "Guozhang Wang" <wangg...@gmail.com> wrote: > > > > > > > > > > Hello Boyang, > > > > > > > > > > I've browsed through the new wiki and there are still a couple of > > > > > minor > > > > > things to notice: > > > > > > > > > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > > > > > > > > > > 2. LeaveGroupRequest added a list of group instance id, but still > > > > > keep the > > > > > member id as a singleton; is that intentional? I think to make > > > > the > > > > > protocol > > > > > consistent both member id and instance ids could be plural. > > > > > > > > > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm > > > > > wondering if we > > > > > can defer adding this while just add the corresponding calls of > > > > the > > > > > LeaveGroupRequest inside Streams until we have used it in > > > > > production and > > > > > hence have a better understanding on how flexible or extensible > > > > if > > > > > we want > > > > > to add any cmd tools. The rationale is that if we do not > > > > > necessarily need > > > > > it now, we can always add it later with a more think-through API > > > > > design, > > > > > but if we add the tool in a rush, we may need to extend or modify > > > > > it soon > > > > > after we realize its limits in operations. > > > > > > > > > > Otherwise, I'm +1 on the proposal. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bche...@outlook.com> > > > > > wrote: > > > > > > > > > > > Hey community friends, > > > > > > > > > > > > after another month of polishing, KIP-345< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > design is ready for vote. Feel free to add your comment on the > > > > > discussion > > > > > > thread or here. > > > > > > > > > > > > Thanks for your time! > > > > > > > > > > > > Boyang > > > > > > ________________________________ > > > > > > From: Boyang Chen <bche...@outlook.com> > > > > > > Sent: Friday, November 9, 2018 6:35 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: [VOTE] KIP-345: Introduce static membership protocol > > > > to > > > > > reduce > > > > > > consumer rebalances > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > > > > thanks so much for all the inputs on KIP-345 so far. The > > > > > original proposal > > > > > > has enhanced a lot with your help. To make sure the > > > > > implementation go > > > > > > smoothly without back and forth, I would like to start a vote > > > > on > > > > > the final > > > > > > design agreement now: > > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=g4%2BMXKpkiQLZXg5HJWfJhw1kc1PbDNwyiX9zkREVqGE%3D&reserved=0 > > > > < > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > KIP-345: Introduce static membership protocol to reduce ...< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > > cwiki.apache.org > > > > > > For stateful applications, one of the biggest performance > > > > > bottleneck is > > > > > > the state shuffling. In Kafka consumer, there is a concept > > > > called > > > > > > "rebalance" which means that for given M partitions and N > > > > > consumers in one > > > > > > consumer group, Kafka will try to balance the load between > > > > > consumers and > > > > > > ideally have ... > > > > > > > > > > > > > > > > > > Let me know if you have any questions. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Boyang > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > > -- > > > Best, > > > Stanislav > > > > > > > > -- > -- Guozhang >
