Hi David, Thanks for the reminder.
DJ4: Add related content to the KIP. Bump validVersions in ConsumerGroupDescribeResponse to “0-1”. Add a new field “Protocol” to ConsumerGroupDescribeResponse. Thanks, PoAn > On Nov 19, 2024, at 3:57 PM, David Jacot <dja...@confluent.io.INVALID> wrote: > > Hi PoAn, > > DJ4: Could you please also include the required changes to the > ConsumerGroupDescribed API in the public interface section? We basically > need to bump the version, add the new field, etc. > > Thanks, > David > > On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <yangp...@gmail.com> wrote: > >> Hi David, >> >> DJ4: Add a new field protocol to MemberDescription, >> so the command line tool can show protocol information when users describe >> members. >> >> If there is no further suggestion, I will start a vote thread today. >> >> Thanks, >> PoAn >> >>> On Nov 15, 2024, at 11:50 PM, David Jacot <dja...@confluent.io.INVALID> >> wrote: >>> >>> Hi, >>> >>> DJ2: Using "-" sounds good to me. >>> >>> DJ3: That seems reasonable to me. >>> >>> DJ4: Why not add it right now? I don't want to change the output of the >>> tool too many times. >>> >>> Best, >>> David >>> >>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <yangp...@gmail.com> wrote: >>> >>>> Hi David / Andrew, >>>> >>>> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh >>>> implementation. >>>> I will handle kafka-consumer-groups.sh. >>>> >>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new >> format >>>> is more clear. >>>> >>>> The new format will be like >>>> >> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> >>>> >>>> Using colon(:) to concat topic name and partition IDs. >>>> Using comma(,) to concat partition IDs within same topic name. >>>> Using semicolon(;) to concat topic strings. >>>> >>>> AS3: Fix it with kafka-share-groups.sh. Thanks. >>>> >>>> If there is no further suggestion, I will start a vote thread next >> Monday. >>>> >>>> Thanks, >>>> PoAn >>>> >>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < >>>> andrew_schofield_j...@outlook.com> wrote: >>>>> >>>>> Hi PoAn, >>>>> DJ2: I was just going to comment that "-" would be a more appropriate >>>> missing value, but >>>>> you got there first. >>>>> >>>>> AS3: The examples for kafka-share-groups.sh include >>>> kafka-consumer-groups.sh in the >>>>> command line. >>>>> >>>>> If this is accepted in time, I'm happy to pick up the implementation of >>>> the share groups >>>>> part of this if it helps. >>>>> >>>>> Thanks, >>>>> Andrew >>>>> >>>>> ________________________________________ >>>>> From: Frank Yang <yangp...@gmail.com> >>>>> Sent: 14 November 2024 10:48 >>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command >>>> line tool to support new consumer group >>>>> >>>>> Hi David, >>>>> >>>>> Thanks for the review and suggestion! I would like to get this in AK >> 4.0 >>>> as well. I will do my best. >>>>> >>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before >>>> #MEMBERS. >>>>> >>>>> DJ2: I prefer to follow current missing column value “-“. (reference < >>>> >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 >>>>> ) >>>>> >>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH >>>> TARGET-ASSIGNMENT. >>>>> Remove GROUP-EPOCH. >>>>> >>>>> For assignment value, it follows current output (reference < >>>> >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 >>> ). >>>> I think the form of `topicid-partitionid` is more clear. >>>>> If we would like to use this form, I will update both output in >>>> kafka-consumer-groups.sh and kafka-share-groups.sh. >>>>> >>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at >>>> group level. >>>>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData >>>> don’t have protocol at member level. >>>>> Could we use a followup to add it? >>>>> >>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET. >>>>> >>>>> Thanks, >>>>> PoAn >>>>> >>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot <dja...@confluent.io.INVALID >>> >>>> wrote: >>>>>> >>>>>> Hi PoAn, >>>>>> >>>>>> Thanks for the KIP! I have a few minor comments/suggestions: >>>>>> >>>>>> DJ1: In the output of `--describe --verbose`, I would suggest putting >>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`. >>>>>> DJ2: Continuing on the above, I assume that we will print out N/A for >>>> the >>>>>> fields not supported by classic groups. Is this correct? >>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we >>>>>> should use the following order and terms: CURRENT-EPOCH >>>> CURRENT-ASSIGNMENT >>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH >> because >>>> it >>>>>> is already in the group description. The value `(0)` used for the >>>>>> assignment is incorrect. Here I suppose that we will print out the >> list >>>> of >>>>>> partitions in the form of `topicid-partitionid`. >>>>>> DJ4: Continuing on the above, I wonder if we should also add the >>>> protocol >>>>>> used `classic` or `consumer`. For context, it is possible to have >>>> `classic` >>>>>> members and `consumer` members in a `consumer` group during an online >>>>>> upgrade from the classic protocol to the consumer protocol. Having >> this >>>>>> information may be handy for administrators. What do you think? >>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would >> suggest >>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. >>>>>> >>>>>> It would be great if we could get this one in AK 4.0 as it changes the >>>>>> output of the command. >>>>>> >>>>>> Thanks, >>>>>> DJ >>>>>> >>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <yangp...@gmail.com> wrote: >>>>>> >>>>>>> Hi Sean / Andrew / Lianet, >>>>>>> >>>>>>> Thanks for all your review and suggestions. >>>>>>> >>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give >>>> —verbose >>>>>>> option. >>>>>>> —describe —verbose: shows group epoch and target assignment epoch. >>>>>>> —describe —members —verbose: shows above information, member epoch, >> and >>>>>>> target assignment. >>>>>>> >>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition. >>>>>>> >>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional >>>> fields in >>>>>>> ConsumerGroupDescription. >>>>>>> For share group, it relies on KIP-848. It must have epoch, so I use >> int >>>>>>> fields in ShareGroupDescription. >>>>>>> >>>>>>> LM3: Remove —state change. We can get group level information by >>>> —describe >>>>>>> —verbose. >>>>>>> >>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose. >>>>>>> >>>>>>> Thanks. >>>>>>> PoAn >>>>>>> >>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <liane...@gmail.com> wrote: >>>>>>>> >>>>>>>> Hello Frank, thanks for the KIP! A few comments: >>>>>>>> >>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a >>>>>>>> --verbose option. I expect someone would only attempt to make sense >> of >>>>>>> the >>>>>>>> epochs while debugging an issue, not while trying to view the group >> or >>>>>>>> member's info. (Also member-epoch makes more sense to me, even more >>>> if we >>>>>>>> end up agreeing in a --verbose). Related to both issues, my take is >>>> that >>>>>>>> whoever is close to the protocol will expect member-epoch, whoever >> is >>>> not >>>>>>>> will probably won't even need to see the epochs at all. >>>>>>>> >>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to >> always >>>>>>> have >>>>>>>> them? with 0 initially, defined on the broker side for the group >> ones, >>>>>>> and >>>>>>>> client side for the member) >>>>>>>> >>>>>>>> LM3. Why is the KIP including the –state option in the proposed >>>> changes? >>>>>>> ( >>>>>>>> >>>>>>> >>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>>>>>>> ). >>>>>>>> I get that the output would change in that example, but it’s not >>>> because >>>>>>> of >>>>>>>> any change to the –state option itself. It's because of the change >>>>>>> proposed >>>>>>>> to the –described (required with the --state), and the changes to >>>>>>>> --describe are already explained above (seems confusing, got me >>>> looking >>>>>>> for >>>>>>>> a change to the state filter which seemed unrelated). >>>>>>>> >>>>>>>> LM4. Group epoch and target assignment epoch are conceptually at the >>>>>>> group >>>>>>>> level. vs member epoch that lands at a member level. So wonder if we >>>>>>> should >>>>>>>> show them accordingly (ex. using the --verbose option) >>>>>>>> –describe –verbose => shows group epoch and target assignment epoch >>>>>>>> –describe –members –verbose => shows member epoch (along >>>>>>>> with group epoch and target assignment epoch) >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Lianet >>>>>>>> >>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>> >>>>>>>>> Hi PoAn, >>>>>>>>> Thanks for the KIP. I have a few comments. >>>>>>>>> >>>>>>>>> AS1: It seems to me that these new additions are most useful to >>>> people >>>>>>>>> trying to understand >>>>>>>>> the progress of rebalancing in quite some detail. The information >> is >>>>>>>>> really not understandable >>>>>>>>> for most users who do not have deep knowledge of KIP-848/932. >>>>>>>>> >>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a >>>>>>> --members >>>>>>>>> --verbose option >>>>>>>>> and only include the new information in the output for that option, >>>>>>> rather >>>>>>>>> than changing the >>>>>>>>> non-verbose --members output. >>>>>>>>> >>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh >>>> --members >>>>>>>>> and only add the >>>>>>>>> new information for the --verbose output. >>>>>>>>> >>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of >>>>>>>>> CONSUMER-EPOCH. >>>>>>>>> I think it's more confusing not following the terminology of the >>>> KIPs. >>>>>>> For >>>>>>>>> one thing, >>>>>>>>> we already have "member" in the admin client such as >>>> MemberDescription. >>>>>>>>> It's not a >>>>>>>>> new term. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Andrew >>>>>>>>> ________________________________________ >>>>>>>>> From: PoAn Yang <pay...@apache.org> >>>>>>>>> Sent: 25 October 2024 13:55 >>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups >> command >>>>>>> line >>>>>>>>> tool to support new consumer group >>>>>>>>> >>>>>>>>> Hi Lucas, >>>>>>>>> >>>>>>>>> Thanks for the review! >>>>>>>>> >>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. >>>> Could >>>>>>>>> you take a look? Thanks for the suggestion. >>>>>>>>> >>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, >>>> users >>>>>>> may >>>>>>>>> confuse what is different between CONSUMER and MEMBER. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> PoAn >>>>>>>>> >>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>>>>>>> Hi Frank, >>>>>>>>>> >>>>>>>>>> thanks for the KIP! >>>>>>>>>> >>>>>>>>>> 1) For consistency, should we do the same for >>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do >>>> not >>>>>>>>>> implement it right now if the share group implementation may still >>>> be >>>>>>>>>> incomplete, it may make sense to include it in the KIP. >>>>>>>>>> >>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem >>>> more >>>>>>>>>> consistent. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Lucas >>>>>>>>>> >>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <yangp...@gmail.com> >>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP >>>> enhances >>>>>>>>> the >>>>>>>>>>> kafka-consumer-groups tools to include state which is introduced >> by >>>>>>>>> KIP-848. >>>>>>>>>>> >>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to >> support >>>>>>> new >>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation >>>>>>>>>>> < >>>>>>>>> >>>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>> >>>>>>>>>>> cwiki.apache.org >>>>>>>>>>> < >>>>>>>>> >>>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>> >>>>>>>>>>> [image: favicon.ico] >>>>>>>>>>> < >>>>>>>>> >>>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>> >>>>>>>>>>> < >>>>>>>>> >>>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> PoAn >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>> >>>> >> >>
