Hi Folks, Would it be good to move this to the DISCUSS thread and keep this thread only for voting purposes, else it will be hard to coordinate responses between 2 threads.
Thanks, Mayuresh On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen <bche...@outlook.com> wrote: > Thanks Guozhang for the reply! > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > Added the definition. > 2. LeaveGroupRequest added a list of group instance id, but still keep the > member id as a singleton; is that intentional? I think to make the protocol > consistent both member id and instance ids could be plural. > Since a dynamic member would send LeaveGroupRequest with its member.id, > I feel it's ok to keep the existing API instead of expanding singleton to > a list. Haven't > been able to define a scenario where we need to pass a list of `member.id > `. > What do you think? > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we > can defer adding this while just add the corresponding calls of the > LeaveGroupRequest inside Streams until we have used it in production and > hence have a better understanding on how flexible or extensible if we want > to add any cmd tools. The rationale is that if we do not necessarily need > it now, we can always add it later with a more think-through API design, > but if we add the tool in a rush, we may need to extend or modify it soon > after we realize its limits in operations. > Totally agree. I moved this part to the future work, because tooling > options could be addressed > in a separate KIP and a universally favorable solution could be discussed > independently (for different > company setup) > > Best, > Boyang > > ________________________________ > From: Guozhang Wang <wangg...@gmail.com> > Sent: Tuesday, December 4, 2018 1:27 AM > To: dev > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > reduce consumer rebalances > > Hello Boyang, > > I've browsed through the new wiki and there are still a couple of minor > things to notice: > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > > 2. LeaveGroupRequest added a list of group instance id, but still keep the > member id as a singleton; is that intentional? I think to make the protocol > consistent both member id and instance ids could be plural. > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we > can defer adding this while just add the corresponding calls of the > LeaveGroupRequest inside Streams until we have used it in production and > hence have a better understanding on how flexible or extensible if we want > to add any cmd tools. The rationale is that if we do not necessarily need > it now, we can always add it later with a more think-through API design, > but if we add the tool in a rush, we may need to extend or modify it soon > after we realize its limits in operations. > > Otherwise, I'm +1 on the proposal. > > Guozhang > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bche...@outlook.com> wrote: > > > Hey community friends, > > > > after another month of polishing, KIP-345< > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0 > > > > design is ready for vote. Feel free to add your comment on the discussion > > thread or here. > > > > Thanks for your time! > > > > Boyang > > ________________________________ > > From: Boyang Chen <bche...@outlook.com> > > Sent: Friday, November 9, 2018 6:35 AM > > To: dev@kafka.apache.org > > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce > > consumer rebalances > > > > Hey all, > > > > > > thanks so much for all the inputs on KIP-345 so far. The original > proposal > > has enhanced a lot with your help. To make sure the implementation go > > smoothly without back and forth, I would like to start a vote on the > final > > design agreement now: > > > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3D&reserved=0 > < > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0 > > > > > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances< > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0 > > > > > > > KIP-345: Introduce static membership protocol to reduce ...< > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0 > > > > > cwiki.apache.org > > For stateful applications, one of the biggest performance bottleneck is > > the state shuffling. In Kafka consumer, there is a concept called > > "rebalance" which means that for given M partitions and N consumers in > one > > consumer group, Kafka will try to balance the load between consumers and > > ideally have ... > > > > > > Let me know if you have any questions. > > > > > > Best, > > > > Boyang > > > > > > -- > -- Guozhang > -- -Regards, Mayuresh R. Gharat (862) 250-7125
