Thanks Stanislav for the update! One suggestion I have is that it would be helpful to put your
reasoning on deciding the current default value. For example, in certain use cases at Pinterest we are very likely to have more consumers than 250 when we configure 8 stream instances with 32 threads. For the effectiveness of this KIP, we should encourage people to discuss their opinions on the default setting and ideally reach a consensus. Best, Boyang ________________________________ From: Stanislav Kozlovski <stanis...@confluent.io> Sent: Monday, November 26, 2018 6:14 PM To: dev@kafka.apache.org Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth Hey everybody, It's been a week since this KIP and not much discussion has been made. I assume that this is a straight forward change and I will open a voting thread in the next couple of days if nobody has anything to suggest. Best, Stanislav On Thu, Nov 22, 2018 at 12:56 PM Stanislav Kozlovski <stanis...@confluent.io> wrote: > Greetings everybody, > > I have enriched the KIP a bit with a bigger Motivation section and also > renamed it. > KIP: > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BIntroduce%2Ba%2Bconfigurable%2Bconsumer%2Bgroup%2Bsize%2Blimit&data=02%7C01%7C%7C085ed04564f2472e50f308d65387f4fd%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636788240721218938&sdata=C6aXV4T6JWcNPtJhVSNxPrHSm2oTP%2BtGN4XvD4jSUOU%3D&reserved=0 > > I'm looking forward to discussions around it. > > Best, > Stanislav > > On Tue, Nov 20, 2018 at 1:47 PM Stanislav Kozlovski < > stanis...@confluent.io> wrote: > >> Hey there everybody, >> >> Thanks for the introduction Boyang. I appreciate the effort you are >> putting into improving consumer behavior in Kafka. >> >> @Matt >> I also believe the default value is high. In my opinion, we should aim to >> a default cap around 250. This is because in the current model any consumer >> rebalance is disrupting to every consumer. The bigger the group, the longer >> this period of disruption. >> >> If you have such a large consumer group, chances are that your >> client-side logic could be structured better and that you are not using the >> high number of consumers to achieve high throughput. >> 250 can still be considered of a high upper bound, I believe in practice >> users should aim to not go over 100 consumers per consumer group. >> >> In regards to the cap being global/per-broker, I think that we should >> consider whether we want it to be global or *per-topic*. For the time >> being, I believe that having it per-topic with a global default might be >> the best situation. Having it global only seems a bit restricting to me and >> it never hurts to support more fine-grained configurability (given it's the >> same config, not a new one being introduced). >> >> On Tue, Nov 20, 2018 at 11:32 AM Boyang Chen <bche...@outlook.com> wrote: >> >>> Thanks Matt for the suggestion! I'm still open to any suggestion to >>> change the default value. Meanwhile I just want to point out that this >>> value is a just last line of defense, not a real scenario we would expect. >>> >>> >>> In the meanwhile, I discussed with Stanislav and he would be driving the >>> 389 effort from now on. Stanislav proposed the idea in the first place and >>> had already come up a draft design, while I will keep focusing on KIP-345 >>> effort to ensure solving the edge case described in the JIRA< >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-7610&data=02%7C01%7C%7C085ed04564f2472e50f308d65387f4fd%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636788240721218938&sdata=PyOSGb6FhjcIS0XL2vcv2YEUSaYk9lL593ioHS4rRHk%3D&reserved=0>. >>> >>> >>> Thank you Stanislav for making this happen! >>> >>> >>> Boyang >>> >>> ________________________________ >>> From: Matt Farmer <m...@frmr.me> >>> Sent: Tuesday, November 20, 2018 10:24 AM >>> To: dev@kafka.apache.org >>> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member >>> metadata growth >>> >>> Thanks for the KIP. >>> >>> Will this cap be a global cap across the entire cluster or per broker? >>> >>> Either way the default value seems a bit high to me, but that could just >>> be >>> from my own usage patterns. I’d have probably started with 500 or 1k but >>> could be easily convinced that’s wrong. >>> >>> Thanks, >>> Matt >>> >>> On Mon, Nov 19, 2018 at 8:51 PM Boyang Chen <bche...@outlook.com> wrote: >>> >>> > Hey folks, >>> > >>> > >>> > I would like to start a discussion on KIP-389: >>> > >>> > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BEnforce%2Bgroup.max.size%2Bto%2Bcap%2Bmember%2Bmetadata%2Bgrowth&data=02%7C01%7C%7C085ed04564f2472e50f308d65387f4fd%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636788240721218938&sdata=DXlRY6ydvXSjMU0CaTvoEj65DOC4d0p02hzu6IdGyk8%3D&reserved=0 >>> > >>> > >>> > This is a pretty simple change to cap the consumer group size for >>> broker >>> > stability. Give me your valuable feedback when you got time. >>> > >>> > >>> > Thank you! >>> > >>> >> >> >> -- >> Best, >> Stanislav >> > > > -- > Best, > Stanislav > -- Best, Stanislav
