Hi David / Andrew, Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh implementation. I will handle kafka-consumer-groups.sh.
DJ3: After discussing with @Chia-Ping Tsai, we think that using new format is more clear. The new format will be like <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> Using colon(:) to concat topic name and partition IDs. Using comma(,) to concat partition IDs within same topic name. Using semicolon(;) to concat topic strings. AS3: Fix it with kafka-share-groups.sh. Thanks. If there is no further suggestion, I will start a vote thread next Monday. Thanks, PoAn > On Nov 14, 2024, at 9:05 PM, Andrew Schofield > <andrew_schofield_j...@outlook.com> wrote: > > Hi PoAn, > DJ2: I was just going to comment that "-" would be a more appropriate missing > value, but > you got there first. > > AS3: The examples for kafka-share-groups.sh include kafka-consumer-groups.sh > in the > command line. > > If this is accepted in time, I'm happy to pick up the implementation of the > share groups > part of this if it helps. > > Thanks, > Andrew > > ________________________________________ > From: Frank Yang <yangp...@gmail.com> > Sent: 14 November 2024 10:48 > To: dev@kafka.apache.org <dev@kafka.apache.org> > Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line > tool to support new consumer group > > Hi David, > > Thanks for the review and suggestion! I would like to get this in AK 4.0 as > well. I will do my best. > > DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before > #MEMBERS. > > DJ2: I prefer to follow current missing column value “-“. (reference > <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92>) > > DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH > TARGET-ASSIGNMENT. > Remove GROUP-EPOCH. > > For assignment value, it follows current output (reference > <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>). > I think the form of `topicid-partitionid` is more clear. > If we would like to use this form, I will update both output in > kafka-consumer-groups.sh and kafka-share-groups.sh. > > DJ4: It looks like DescribeGroupsResponseData only has protocol type at group > level. > Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData don’t > have protocol at member level. > Could we use a followup to add it? > > DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET. > > Thanks, > PoAn > >> On Nov 14, 2024, at 3:43 PM, David Jacot <dja...@confluent.io.INVALID> wrote: >> >> Hi PoAn, >> >> Thanks for the KIP! I have a few minor comments/suggestions: >> >> DJ1: In the output of `--describe --verbose`, I would suggest putting >> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`. >> DJ2: Continuing on the above, I assume that we will print out N/A for the >> fields not supported by classic groups. Is this correct? >> DJ3: In the output of `--describe --members --verbose`, I wonder if we >> should use the following order and terms: CURRENT-EPOCH CURRENT-ASSIGNMENT >> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because it >> is already in the group description. The value `(0)` used for the >> assignment is incorrect. Here I suppose that we will print out the list of >> partitions in the form of `topicid-partitionid`. >> DJ4: Continuing on the above, I wonder if we should also add the protocol >> used `classic` or `consumer`. For context, it is possible to have `classic` >> members and `consumer` members in a `consumer` group during an online >> upgrade from the classic protocol to the consumer protocol. Having this >> information may be handy for administrators. What do you think? >> DJ5: In the output of `--describe --offsets --verbose`, I would suggest >> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. >> >> It would be great if we could get this one in AK 4.0 as it changes the >> output of the command. >> >> Thanks, >> DJ >> >> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <yangp...@gmail.com> wrote: >> >>> Hi Sean / Andrew / Lianet, >>> >>> Thanks for all your review and suggestions. >>> >>> AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose >>> option. >>> —describe —verbose: shows group epoch and target assignment epoch. >>> —describe —members —verbose: shows above information, member epoch, and >>> target assignment. >>> >>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition. >>> >>> LM2: For classic group, it doesn’t have epoch, so I use Optional fields in >>> ConsumerGroupDescription. >>> For share group, it relies on KIP-848. It must have epoch, so I use int >>> fields in ShareGroupDescription. >>> >>> LM3: Remove —state change. We can get group level information by —describe >>> —verbose. >>> >>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose. >>> >>> Thanks. >>> PoAn >>> >>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <liane...@gmail.com> wrote: >>>> >>>> Hello Frank, thanks for the KIP! A few comments: >>>> >>>> LM1. I strongly agree with Andrew's suggestion of moving this into a >>>> --verbose option. I expect someone would only attempt to make sense of >>> the >>>> epochs while debugging an issue, not while trying to view the group or >>>> member's info. (Also member-epoch makes more sense to me, even more if we >>>> end up agreeing in a --verbose). Related to both issues, my take is that >>>> whoever is close to the protocol will expect member-epoch, whoever is not >>>> will probably won't even need to see the epochs at all. >>>> >>>> LM2. Why are the epochs defined as Optional (don't we expect to always >>> have >>>> them? with 0 initially, defined on the broker side for the group ones, >>> and >>>> client side for the member) >>>> >>>> LM3. Why is the KIP including the –state option in the proposed changes? >>> ( >>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>>> ). >>>> I get that the output would change in that example, but it’s not because >>> of >>>> any change to the –state option itself. It's because of the change >>> proposed >>>> to the –described (required with the --state), and the changes to >>>> --describe are already explained above (seems confusing, got me looking >>> for >>>> a change to the state filter which seemed unrelated). >>>> >>>> LM4. Group epoch and target assignment epoch are conceptually at the >>> group >>>> level. vs member epoch that lands at a member level. So wonder if we >>> should >>>> show them accordingly (ex. using the --verbose option) >>>> –describe –verbose => shows group epoch and target assignment epoch >>>> –describe –members –verbose => shows member epoch (along >>>> with group epoch and target assignment epoch) >>>> >>>> Thanks! >>>> >>>> Lianet >>>> >>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>>> andrew_schofield_j...@outlook.com> wrote: >>>> >>>>> Hi PoAn, >>>>> Thanks for the KIP. I have a few comments. >>>>> >>>>> AS1: It seems to me that these new additions are most useful to people >>>>> trying to understand >>>>> the progress of rebalancing in quite some detail. The information is >>>>> really not understandable >>>>> for most users who do not have deep knowledge of KIP-848/932. >>>>> >>>>> As a result, I suggest for kafka-share-groups.sh that you add a >>> --members >>>>> --verbose option >>>>> and only include the new information in the output for that option, >>> rather >>>>> than changing the >>>>> non-verbose --members output. >>>>> >>>>> I also make a similar suggestion for kafka-consumer-groups.sh --members >>>>> and only add the >>>>> new information for the --verbose output. >>>>> >>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of >>>>> CONSUMER-EPOCH. >>>>> I think it's more confusing not following the terminology of the KIPs. >>> For >>>>> one thing, >>>>> we already have "member" in the admin client such as MemberDescription. >>>>> It's not a >>>>> new term. >>>>> >>>>> >>>>> Thanks, >>>>> Andrew >>>>> ________________________________________ >>>>> From: PoAn Yang <pay...@apache.org> >>>>> Sent: 25 October 2024 13:55 >>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command >>> line >>>>> tool to support new consumer group >>>>> >>>>> Hi Lucas, >>>>> >>>>> Thanks for the review! >>>>> >>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could >>>>> you take a look? Thanks for the suggestion. >>>>> >>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users >>> may >>>>> confuse what is different between CONSUMER and MEMBER. >>>>> >>>>> Thanks, >>>>> PoAn >>>>> >>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>>> Hi Frank, >>>>>> >>>>>> thanks for the KIP! >>>>>> >>>>>> 1) For consistency, should we do the same for >>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not >>>>>> implement it right now if the share group implementation may still be >>>>>> incomplete, it may make sense to include it in the KIP. >>>>>> >>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more >>>>>> consistent. >>>>>> >>>>>> Cheers, >>>>>> Lucas >>>>>> >>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <yangp...@gmail.com> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I would like to kick off the discussion of KIP-1099. This KIP enhances >>>>> the >>>>>>> kafka-consumer-groups tools to include state which is introduced by >>>>> KIP-848. >>>>>>> >>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support >>> new >>>>>>> consumer group - Apache Kafka - Apache Software Foundation >>>>>>> < >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>> >>>>>>> cwiki.apache.org >>>>>>> < >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>> >>>>>>> [image: favicon.ico] >>>>>>> < >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>> >>>>>>> < >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> PoAn >>>>>>> >>>>>> >>>>> >>> >>>
