Static-Group membership ships with AK 2.3 (the open tickets of the KIP are minor):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances There is also KIP-415 for Kafka Connect in AK 2.3: https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect Currently WIP is KIP-429 and KIP-441: https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams On 7/19/19 12:31 PM, Jeff Widman wrote: > I am also interested in learning how others are handling this. > > I also support several services where average message processing time takes > 20 seconds per message but p99 time is about 20 minutes and the > stop-the-world rebalancing is very painful > > On Fri, Jul 19, 2019, 11:38 AM Raman Gupta <rocketra...@gmail.com> wrote: > >> I've found >> https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing:+Support+and+Policies >> and >> https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing+for+Streams >> . >> This is *exactly* what I need, right down to the Kubernetes pod >> restart case. The number of issues with the current approach to >> rebalancing elucidated in these documents is downright scary, and now >> I am not surprised I am having tonnes of issues. >> >> Are there any plans to start implementing delayed imbalance and >> standby bootstrap? >> >> Are there any short-term best practices that can help alleviate these >> issues? My main problem right now is the "Instance Bounce" and >> "Instance Failover" scenarios, and according to this wiki page, >> num.standby.replicas should help with at least the former. Can someone >> explain what this does? >> >> Regards, >> Raman >> >> On Fri, Jul 19, 2019 at 12:53 PM Raman Gupta <rocketra...@gmail.com> >> wrote: >>> >>> I have a situation in which the current rebalancing algorithm seems to >>> be extremely sub-optimal. >>> >>> I have a topic with 100 partitions, and up to 100 separate consumers. >>> Processing each message on this topic takes between 1 and 20 minutes, >>> depending on the message. >>> >>> If any of the 100 consumers dies or drops out of the group, there is a >>> huge amount of idle time as many consumers (up to 99 of them) finish >>> their work and sit around idle, just waiting for the rebalance to >>> complete. >>> >>> In addition, with 100 consumers, its not unusual for one to die for >>> one reason or another, so these stop-the-world rebalances are >>> happening all the time, making the entire system slow to a snail's >>> pace. >>> >>> It surprises me that rebalance is so inefficient. I would have thought >>> that partitions would just be assigned/unassigned to consumers in >>> real-time without waiting for the entire consumer group to quiesce. >>> >>> Is there anything I can do to improve matters? >>> >>> Regards, >>> Raman >> >
signature.asc
Description: OpenPGP digital signature