Hi PoAn, DJ2: I was just going to comment that "-" would be a more appropriate missing value, but you got there first.
AS3: The examples for kafka-share-groups.sh include kafka-consumer-groups.sh in the command line. If this is accepted in time, I'm happy to pick up the implementation of the share groups part of this if it helps. Thanks, Andrew ________________________________________ From: Frank Yang <yangp...@gmail.com> Sent: 14 November 2024 10:48 To: dev@kafka.apache.org <dev@kafka.apache.org> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group Hi David, Thanks for the review and suggestion! I would like to get this in AK 4.0 as well. I will do my best. DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before #MEMBERS. DJ2: I prefer to follow current missing column value “-“. (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92>) DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH TARGET-ASSIGNMENT. Remove GROUP-EPOCH. For assignment value, it follows current output (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>). I think the form of `topicid-partitionid` is more clear. If we would like to use this form, I will update both output in kafka-consumer-groups.sh and kafka-share-groups.sh. DJ4: It looks like DescribeGroupsResponseData only has protocol type at group level. Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData don’t have protocol at member level. Could we use a followup to add it? DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET. Thanks, PoAn > On Nov 14, 2024, at 3:43 PM, David Jacot <dja...@confluent.io.INVALID> wrote: > > Hi PoAn, > > Thanks for the KIP! I have a few minor comments/suggestions: > > DJ1: In the output of `--describe --verbose`, I would suggest putting > `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`. > DJ2: Continuing on the above, I assume that we will print out N/A for the > fields not supported by classic groups. Is this correct? > DJ3: In the output of `--describe --members --verbose`, I wonder if we > should use the following order and terms: CURRENT-EPOCH CURRENT-ASSIGNMENT > TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because it > is already in the group description. The value `(0)` used for the > assignment is incorrect. Here I suppose that we will print out the list of > partitions in the form of `topicid-partitionid`. > DJ4: Continuing on the above, I wonder if we should also add the protocol > used `classic` or `consumer`. For context, it is possible to have `classic` > members and `consumer` members in a `consumer` group during an online > upgrade from the classic protocol to the consumer protocol. Having this > information may be handy for administrators. What do you think? > DJ5: In the output of `--describe --offsets --verbose`, I would suggest > putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. > > It would be great if we could get this one in AK 4.0 as it changes the > output of the command. > > Thanks, > DJ > > On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <yangp...@gmail.com> wrote: > >> Hi Sean / Andrew / Lianet, >> >> Thanks for all your review and suggestions. >> >> AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose >> option. >> —describe —verbose: shows group epoch and target assignment epoch. >> —describe —members —verbose: shows above information, member epoch, and >> target assignment. >> >> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition. >> >> LM2: For classic group, it doesn’t have epoch, so I use Optional fields in >> ConsumerGroupDescription. >> For share group, it relies on KIP-848. It must have epoch, so I use int >> fields in ShareGroupDescription. >> >> LM3: Remove —state change. We can get group level information by —describe >> —verbose. >> >> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose. >> >> Thanks. >> PoAn >> >>> On Nov 1, 2024, at 5:08 AM, Lianet M. <liane...@gmail.com> wrote: >>> >>> Hello Frank, thanks for the KIP! A few comments: >>> >>> LM1. I strongly agree with Andrew's suggestion of moving this into a >>> --verbose option. I expect someone would only attempt to make sense of >> the >>> epochs while debugging an issue, not while trying to view the group or >>> member's info. (Also member-epoch makes more sense to me, even more if we >>> end up agreeing in a --verbose). Related to both issues, my take is that >>> whoever is close to the protocol will expect member-epoch, whoever is not >>> will probably won't even need to see the epochs at all. >>> >>> LM2. Why are the epochs defined as Optional (don't we expect to always >> have >>> them? with 0 initially, defined on the broker side for the group ones, >> and >>> client side for the member) >>> >>> LM3. Why is the KIP including the –state option in the proposed changes? >> ( >>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>> ). >>> I get that the output would change in that example, but it’s not because >> of >>> any change to the –state option itself. It's because of the change >> proposed >>> to the –described (required with the --state), and the changes to >>> --describe are already explained above (seems confusing, got me looking >> for >>> a change to the state filter which seemed unrelated). >>> >>> LM4. Group epoch and target assignment epoch are conceptually at the >> group >>> level. vs member epoch that lands at a member level. So wonder if we >> should >>> show them accordingly (ex. using the --verbose option) >>> –describe –verbose => shows group epoch and target assignment epoch >>> –describe –members –verbose => shows member epoch (along >>> with group epoch and target assignment epoch) >>> >>> Thanks! >>> >>> Lianet >>> >>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>> andrew_schofield_j...@outlook.com> wrote: >>> >>>> Hi PoAn, >>>> Thanks for the KIP. I have a few comments. >>>> >>>> AS1: It seems to me that these new additions are most useful to people >>>> trying to understand >>>> the progress of rebalancing in quite some detail. The information is >>>> really not understandable >>>> for most users who do not have deep knowledge of KIP-848/932. >>>> >>>> As a result, I suggest for kafka-share-groups.sh that you add a >> --members >>>> --verbose option >>>> and only include the new information in the output for that option, >> rather >>>> than changing the >>>> non-verbose --members output. >>>> >>>> I also make a similar suggestion for kafka-consumer-groups.sh --members >>>> and only add the >>>> new information for the --verbose output. >>>> >>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of >>>> CONSUMER-EPOCH. >>>> I think it's more confusing not following the terminology of the KIPs. >> For >>>> one thing, >>>> we already have "member" in the admin client such as MemberDescription. >>>> It's not a >>>> new term. >>>> >>>> >>>> Thanks, >>>> Andrew >>>> ________________________________________ >>>> From: PoAn Yang <pay...@apache.org> >>>> Sent: 25 October 2024 13:55 >>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command >> line >>>> tool to support new consumer group >>>> >>>> Hi Lucas, >>>> >>>> Thanks for the review! >>>> >>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could >>>> you take a look? Thanks for the suggestion. >>>> >>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users >> may >>>> confuse what is different between CONSUMER and MEMBER. >>>> >>>> Thanks, >>>> PoAn >>>> >>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>> Hi Frank, >>>>> >>>>> thanks for the KIP! >>>>> >>>>> 1) For consistency, should we do the same for >>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not >>>>> implement it right now if the share group implementation may still be >>>>> incomplete, it may make sense to include it in the KIP. >>>>> >>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more >>>>> consistent. >>>>> >>>>> Cheers, >>>>> Lucas >>>>> >>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <yangp...@gmail.com> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I would like to kick off the discussion of KIP-1099. This KIP enhances >>>> the >>>>>> kafka-consumer-groups tools to include state which is introduced by >>>> KIP-848. >>>>>> >>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support >> new >>>>>> consumer group - Apache Kafka - Apache Software Foundation >>>>>> < >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>> >>>>>> cwiki.apache.org >>>>>> < >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>> >>>>>> [image: favicon.ico] >>>>>> < >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>> >>>>>> < >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>> >>>>>> >>>>>> Thank you, >>>>>> PoAn >>>>>> >>>>> >>>> >> >>
