When you say that it will "break", what does this breakage look like? Will the consumer-group be non-functional? Will just those instances be non-functional? Or will the group be functional, but the rebalancing be non-optimal and require more round-trips/data-transfer? (similar to the current algorithm)
I'm trying to assess the potential for user-error and the impact of user-error. -James > On Jul 27, 2018, at 11:25 AM, Boyang Chen <bche...@outlook.com> wrote: > > Hey James, > > > the algorithm is relying on client side to provide unique consumer member id. > It will break unless we enforce some sort of validation (host + port) on the > server side. To simplify the first version, we do not plan to enforce > validation. A good comparison would be the EOS producer which is in charge of > generating unique transaction id sequence. IMO for broker logic, the > tolerance of client side error is not unlimited. > > > Thank you! > > > ________________________________ > From: James Cheng <wushuja...@gmail.com> > Sent: Saturday, July 28, 2018 1:26 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by > specifying member id > > >> On Jul 26, 2018, at 11:09 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> >> Hi Boyang, >> >> Thanks for the proposed KIP. I made a pass over the wiki and here are some >> comments / questions: >> >> 1. In order to preserve broker compatibility, we need to make sure the >> broker version discovery logic can be integrated with this new logic. I.e. >> if a newer versioned consumer is talking to an older versioned broker who >> does not recognize V4, the client needs to downgrade its JoinGroupRequest >> version to V3 and not setting the member-id specifically. You can take a >> look at the ApiVersionsRequest and see how to work with it. >> >> 2. There may exist some manners to validate that two different clients do >> not send with the same member id, for example if we pass along the >> host:port information from KafkaApis to the GroupCoordinator interface. But >> I think this is overly complicated the logic and may not worthwhile than >> relying on users to specify unique member ids. > > Boyang, > > Thanks for the KIP! How will the algorithm behave if multiple consumers > provide the same member id? > > -James > >> 3. Minor: you would need to bumping up the version of JoinGroupResponse to >> V4 as well. >> >> 4. Minor: in the wiki page, you need to specify the actual string value for >> `MEMBER_ID`, for example "member.id". >> >> 5. When this additional config it specified by users, we should consider >> setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false, >> since otherwise its effectiveness would be less. >> >> >> Guozhang >> >> >> >>> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen <bche...@outlook.com> wrote: >>> >>> Hey friends, >>> >>> >>> I would like to open a discussion thread on KIP-345: >>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A >>> +Reduce+multiple+consumer+rebalances+by+specifying+member+id >>> >>> >>> This KIP is trying to resolve multiple rebalances by maintaining the >>> consumer member id across rebalance generations. I have verified the theory >>> on our internal Stream application, and it could reduce rebalance time to a >>> few seconds when service is rolling restart. >>> >>> >>> Let me know your thoughts, thank you! >>> >>> >>> Best, >>> >>> Boyang >>> >> >> >> >> -- >> -- Guozhang
