Oops, sure thing Mayuresh :) I have only one open question for Guozhang. Will definitely move the discussion back.
Boyang ________________________________ From: Mayuresh Gharat <gharatmayures...@gmail.com> Sent: Tuesday, December 4, 2018 9:52 AM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances Hi Folks, Would it be good to move this to the DISCUSS thread and keep this thread only for voting purposes, else it will be hard to coordinate responses between 2 threads. Thanks, Mayuresh On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen <bche...@outlook.com> wrote: > Thanks Guozhang for the reply! > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > Added the definition. > 2. LeaveGroupRequest added a list of group instance id, but still keep the > member id as a singleton; is that intentional? I think to make the protocol > consistent both member id and instance ids could be plural. > Since a dynamic member would send LeaveGroupRequest with its member.id, > I feel it's ok to keep the existing API instead of expanding singleton to > a list. Haven't > been able to define a scenario where we need to pass a list of `member.id > `. > What do you think? > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we > can defer adding this while just add the corresponding calls of the > LeaveGroupRequest inside Streams until we have used it in production and > hence have a better understanding on how flexible or extensible if we want > to add any cmd tools. The rationale is that if we do not necessarily need > it now, we can always add it later with a more think-through API design, > but if we add the tool in a rush, we may need to extend or modify it soon > after we realize its limits in operations. > Totally agree. I moved this part to the future work, because tooling > options could be addressed > in a separate KIP and a universally favorable solution could be discussed > independently (for different > company setup) > > Best, > Boyang > > ________________________________ > From: Guozhang Wang <wangg...@gmail.com> > Sent: Tuesday, December 4, 2018 1:27 AM > To: dev > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > reduce consumer rebalances > > Hello Boyang, > > I've browsed through the new wiki and there are still a couple of minor > things to notice: > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > > 2. LeaveGroupRequest added a list of group instance id, but still keep the > member id as a singleton; is that intentional? I think to make the protocol > consistent both member id and instance ids could be plural. > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we > can defer adding this while just add the corresponding calls of the > LeaveGroupRequest inside Streams until we have used it in production and > hence have a better understanding on how flexible or extensible if we want > to add any cmd tools. The rationale is that if we do not necessarily need > it now, we can always add it later with a more think-through API design, > but if we add the tool in a rush, we may need to extend or modify it soon > after we realize its limits in operations. > > Otherwise, I'm +1 on the proposal. > > Guozhang > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bche...@outlook.com> wrote: > > > Hey community friends, > > > > after another month of polishing, KIP-345< > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794861519727292&sdata=s6MOIBWvgDrGPXXBu4vXcsI1Z2dcJIteJQT4zdo%2FSMY%3D&reserved=0 > > > > design is ready for vote. Feel free to add your comment on the discussion > > thread or here. > > > > Thanks for your time! > > > > Boyang > > ________________________________ > > From: Boyang Chen <bche...@outlook.com> > > Sent: Friday, November 9, 2018 6:35 AM > > To: dev@kafka.apache.org > > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce > > consumer rebalances > > > > Hey all, > > > > > > thanks so much for all the inputs on KIP-345 so far. The original > proposal > > has enhanced a lot with your help. To make sure the implementation go > > smoothly without back and forth, I would like to start a vote on the > final > > design agreement now: > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794861519727292&sdata=9QF8gdy%2FGD2NSOjNDsfZ3J3FPlk4ShZ8nL%2BW%2FVU5mpc%3D&reserved=0 > < > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794861519727292&sdata=s6MOIBWvgDrGPXXBu4vXcsI1Z2dcJIteJQT4zdo%2FSMY%3D&reserved=0 > > > > > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances< > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794861519727292&sdata=s6MOIBWvgDrGPXXBu4vXcsI1Z2dcJIteJQT4zdo%2FSMY%3D&reserved=0 > > > > > > > KIP-345: Introduce static membership protocol to reduce ...< > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794861519727292&sdata=s6MOIBWvgDrGPXXBu4vXcsI1Z2dcJIteJQT4zdo%2FSMY%3D&reserved=0 > > > > > cwiki.apache.org > > For stateful applications, one of the biggest performance bottleneck is > > the state shuffling. In Kafka consumer, there is a concept called > > "rebalance" which means that for given M partitions and N consumers in > one > > consumer group, Kafka will try to balance the load between consumers and > > ideally have ... > > > > > > Let me know if you have any questions. > > > > > > Best, > > > > Boyang > > > > > > -- > -- Guozhang > -- -Regards, Mayuresh R. Gharat (862) 250-7125
