Greetings everybody, I have enriched the KIP a bit with a bigger Motivation section and also renamed it. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Introduce+a+configurable+consumer+group+size+limit
I'm looking forward to discussions around it. Best, Stanislav On Tue, Nov 20, 2018 at 1:47 PM Stanislav Kozlovski <stanis...@confluent.io> wrote: > Hey there everybody, > > Thanks for the introduction Boyang. I appreciate the effort you are > putting into improving consumer behavior in Kafka. > > @Matt > I also believe the default value is high. In my opinion, we should aim to > a default cap around 250. This is because in the current model any consumer > rebalance is disrupting to every consumer. The bigger the group, the longer > this period of disruption. > > If you have such a large consumer group, chances are that your client-side > logic could be structured better and that you are not using the high number > of consumers to achieve high throughput. > 250 can still be considered of a high upper bound, I believe in practice > users should aim to not go over 100 consumers per consumer group. > > In regards to the cap being global/per-broker, I think that we should > consider whether we want it to be global or *per-topic*. For the time > being, I believe that having it per-topic with a global default might be > the best situation. Having it global only seems a bit restricting to me and > it never hurts to support more fine-grained configurability (given it's the > same config, not a new one being introduced). > > On Tue, Nov 20, 2018 at 11:32 AM Boyang Chen <bche...@outlook.com> wrote: > >> Thanks Matt for the suggestion! I'm still open to any suggestion to >> change the default value. Meanwhile I just want to point out that this >> value is a just last line of defense, not a real scenario we would expect. >> >> >> In the meanwhile, I discussed with Stanislav and he would be driving the >> 389 effort from now on. Stanislav proposed the idea in the first place and >> had already come up a draft design, while I will keep focusing on KIP-345 >> effort to ensure solving the edge case described in the JIRA< >> https://issues.apache.org/jira/browse/KAFKA-7610>. >> >> >> Thank you Stanislav for making this happen! >> >> >> Boyang >> >> ________________________________ >> From: Matt Farmer <m...@frmr.me> >> Sent: Tuesday, November 20, 2018 10:24 AM >> To: dev@kafka.apache.org >> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member >> metadata growth >> >> Thanks for the KIP. >> >> Will this cap be a global cap across the entire cluster or per broker? >> >> Either way the default value seems a bit high to me, but that could just >> be >> from my own usage patterns. I’d have probably started with 500 or 1k but >> could be easily convinced that’s wrong. >> >> Thanks, >> Matt >> >> On Mon, Nov 19, 2018 at 8:51 PM Boyang Chen <bche...@outlook.com> wrote: >> >> > Hey folks, >> > >> > >> > I would like to start a discussion on KIP-389: >> > >> > >> > >> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BEnforce%2Bgroup.max.size%2Bto%2Bcap%2Bmember%2Bmetadata%2Bgrowth&data=02%7C01%7C%7Cb0ee4fe97ad44cc046eb08d64e8f5d90%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636782774981237462&sdata=Q2T7hIoVq8GiPVhr0HIxVkGNChkiz1Pvk2zyLD5gCu8%3D&reserved=0 >> > >> > >> > This is a pretty simple change to cap the consumer group size for broker >> > stability. Give me your valuable feedback when you got time. >> > >> > >> > Thank you! >> > >> > > > -- > Best, > Stanislav > -- Best, Stanislav
