Hi David, DJ4: Add a new field protocol to MemberDescription, so the command line tool can show protocol information when users describe members.
If there is no further suggestion, I will start a vote thread today. Thanks, PoAn > On Nov 15, 2024, at 11:50 PM, David Jacot <dja...@confluent.io.INVALID> wrote: > > Hi, > > DJ2: Using "-" sounds good to me. > > DJ3: That seems reasonable to me. > > DJ4: Why not add it right now? I don't want to change the output of the > tool too many times. > > Best, > David > > On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <yangp...@gmail.com> wrote: > >> Hi David / Andrew, >> >> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh >> implementation. >> I will handle kafka-consumer-groups.sh. >> >> DJ3: After discussing with @Chia-Ping Tsai, we think that using new format >> is more clear. >> >> The new format will be like >> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> >> >> Using colon(:) to concat topic name and partition IDs. >> Using comma(,) to concat partition IDs within same topic name. >> Using semicolon(;) to concat topic strings. >> >> AS3: Fix it with kafka-share-groups.sh. Thanks. >> >> If there is no further suggestion, I will start a vote thread next Monday. >> >> Thanks, >> PoAn >> >>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < >> andrew_schofield_j...@outlook.com> wrote: >>> >>> Hi PoAn, >>> DJ2: I was just going to comment that "-" would be a more appropriate >> missing value, but >>> you got there first. >>> >>> AS3: The examples for kafka-share-groups.sh include >> kafka-consumer-groups.sh in the >>> command line. >>> >>> If this is accepted in time, I'm happy to pick up the implementation of >> the share groups >>> part of this if it helps. >>> >>> Thanks, >>> Andrew >>> >>> ________________________________________ >>> From: Frank Yang <yangp...@gmail.com> >>> Sent: 14 November 2024 10:48 >>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command >> line tool to support new consumer group >>> >>> Hi David, >>> >>> Thanks for the review and suggestion! I would like to get this in AK 4.0 >> as well. I will do my best. >>> >>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before >> #MEMBERS. >>> >>> DJ2: I prefer to follow current missing column value “-“. (reference < >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 >>> ) >>> >>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH >> TARGET-ASSIGNMENT. >>> Remove GROUP-EPOCH. >>> >>> For assignment value, it follows current output (reference < >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>). >> I think the form of `topicid-partitionid` is more clear. >>> If we would like to use this form, I will update both output in >> kafka-consumer-groups.sh and kafka-share-groups.sh. >>> >>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at >> group level. >>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData >> don’t have protocol at member level. >>> Could we use a followup to add it? >>> >>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET. >>> >>> Thanks, >>> PoAn >>> >>>> On Nov 14, 2024, at 3:43 PM, David Jacot <dja...@confluent.io.INVALID> >> wrote: >>>> >>>> Hi PoAn, >>>> >>>> Thanks for the KIP! I have a few minor comments/suggestions: >>>> >>>> DJ1: In the output of `--describe --verbose`, I would suggest putting >>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`. >>>> DJ2: Continuing on the above, I assume that we will print out N/A for >> the >>>> fields not supported by classic groups. Is this correct? >>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we >>>> should use the following order and terms: CURRENT-EPOCH >> CURRENT-ASSIGNMENT >>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because >> it >>>> is already in the group description. The value `(0)` used for the >>>> assignment is incorrect. Here I suppose that we will print out the list >> of >>>> partitions in the form of `topicid-partitionid`. >>>> DJ4: Continuing on the above, I wonder if we should also add the >> protocol >>>> used `classic` or `consumer`. For context, it is possible to have >> `classic` >>>> members and `consumer` members in a `consumer` group during an online >>>> upgrade from the classic protocol to the consumer protocol. Having this >>>> information may be handy for administrators. What do you think? >>>> DJ5: In the output of `--describe --offsets --verbose`, I would suggest >>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. >>>> >>>> It would be great if we could get this one in AK 4.0 as it changes the >>>> output of the command. >>>> >>>> Thanks, >>>> DJ >>>> >>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <yangp...@gmail.com> wrote: >>>> >>>>> Hi Sean / Andrew / Lianet, >>>>> >>>>> Thanks for all your review and suggestions. >>>>> >>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give >> —verbose >>>>> option. >>>>> —describe —verbose: shows group epoch and target assignment epoch. >>>>> —describe —members —verbose: shows above information, member epoch, and >>>>> target assignment. >>>>> >>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition. >>>>> >>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional >> fields in >>>>> ConsumerGroupDescription. >>>>> For share group, it relies on KIP-848. It must have epoch, so I use int >>>>> fields in ShareGroupDescription. >>>>> >>>>> LM3: Remove —state change. We can get group level information by >> —describe >>>>> —verbose. >>>>> >>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose. >>>>> >>>>> Thanks. >>>>> PoAn >>>>> >>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <liane...@gmail.com> wrote: >>>>>> >>>>>> Hello Frank, thanks for the KIP! A few comments: >>>>>> >>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a >>>>>> --verbose option. I expect someone would only attempt to make sense of >>>>> the >>>>>> epochs while debugging an issue, not while trying to view the group or >>>>>> member's info. (Also member-epoch makes more sense to me, even more >> if we >>>>>> end up agreeing in a --verbose). Related to both issues, my take is >> that >>>>>> whoever is close to the protocol will expect member-epoch, whoever is >> not >>>>>> will probably won't even need to see the epochs at all. >>>>>> >>>>>> LM2. Why are the epochs defined as Optional (don't we expect to always >>>>> have >>>>>> them? with 0 initially, defined on the broker side for the group ones, >>>>> and >>>>>> client side for the member) >>>>>> >>>>>> LM3. Why is the KIP including the –state option in the proposed >> changes? >>>>> ( >>>>>> >>>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>>>>> ). >>>>>> I get that the output would change in that example, but it’s not >> because >>>>> of >>>>>> any change to the –state option itself. It's because of the change >>>>> proposed >>>>>> to the –described (required with the --state), and the changes to >>>>>> --describe are already explained above (seems confusing, got me >> looking >>>>> for >>>>>> a change to the state filter which seemed unrelated). >>>>>> >>>>>> LM4. Group epoch and target assignment epoch are conceptually at the >>>>> group >>>>>> level. vs member epoch that lands at a member level. So wonder if we >>>>> should >>>>>> show them accordingly (ex. using the --verbose option) >>>>>> –describe –verbose => shows group epoch and target assignment epoch >>>>>> –describe –members –verbose => shows member epoch (along >>>>>> with group epoch and target assignment epoch) >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Lianet >>>>>> >>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>> >>>>>>> Hi PoAn, >>>>>>> Thanks for the KIP. I have a few comments. >>>>>>> >>>>>>> AS1: It seems to me that these new additions are most useful to >> people >>>>>>> trying to understand >>>>>>> the progress of rebalancing in quite some detail. The information is >>>>>>> really not understandable >>>>>>> for most users who do not have deep knowledge of KIP-848/932. >>>>>>> >>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a >>>>> --members >>>>>>> --verbose option >>>>>>> and only include the new information in the output for that option, >>>>> rather >>>>>>> than changing the >>>>>>> non-verbose --members output. >>>>>>> >>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh >> --members >>>>>>> and only add the >>>>>>> new information for the --verbose output. >>>>>>> >>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of >>>>>>> CONSUMER-EPOCH. >>>>>>> I think it's more confusing not following the terminology of the >> KIPs. >>>>> For >>>>>>> one thing, >>>>>>> we already have "member" in the admin client such as >> MemberDescription. >>>>>>> It's not a >>>>>>> new term. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Andrew >>>>>>> ________________________________________ >>>>>>> From: PoAn Yang <pay...@apache.org> >>>>>>> Sent: 25 October 2024 13:55 >>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command >>>>> line >>>>>>> tool to support new consumer group >>>>>>> >>>>>>> Hi Lucas, >>>>>>> >>>>>>> Thanks for the review! >>>>>>> >>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. >> Could >>>>>>> you take a look? Thanks for the suggestion. >>>>>>> >>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, >> users >>>>> may >>>>>>> confuse what is different between CONSUMER and MEMBER. >>>>>>> >>>>>>> Thanks, >>>>>>> PoAn >>>>>>> >>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>>>>> Hi Frank, >>>>>>>> >>>>>>>> thanks for the KIP! >>>>>>>> >>>>>>>> 1) For consistency, should we do the same for >>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do >> not >>>>>>>> implement it right now if the share group implementation may still >> be >>>>>>>> incomplete, it may make sense to include it in the KIP. >>>>>>>> >>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem >> more >>>>>>>> consistent. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Lucas >>>>>>>> >>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <yangp...@gmail.com> >> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP >> enhances >>>>>>> the >>>>>>>>> kafka-consumer-groups tools to include state which is introduced by >>>>>>> KIP-848. >>>>>>>>> >>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support >>>>> new >>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation >>>>>>>>> < >>>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>> >>>>>>>>> cwiki.apache.org >>>>>>>>> < >>>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>> >>>>>>>>> [image: favicon.ico] >>>>>>>>> < >>>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>> >>>>>>>>> < >>>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>> >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> PoAn >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >> >>
