Hi David, Yes, it would be an option to have a ShareMemberDescription. Because ShareGroupDescription is still an evolving interface, it's acceptable to change its definition in a minor release, so no rush to get this done in the next 2 weeks.
I suggest that I change KIP-932 to introduce ShareMemberDescription to replace the use of MemberDescription in ShareGroupDescription. While I'm at it, it's probably prudent to have a ShareMemberAssignment also because I have plans in mind which take assignments beyond what MemberAssignment can represent. I'll start just by defining new classes which simply remove the bit of cruft which does not apply to share groups. Thanks, Andrew ________________________________________ From: David Jacot <dja...@confluent.io.INVALID> Sent: 25 November 2024 15:08 To: dev@kafka.apache.org <dev@kafka.apache.org> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group Hi PoAn, Thanks for the update. Regarding the CLI, I wonder whether we should rather have a column called "Upgraded" which is only displayed when some members use the classic protocol. The displayed value would be True for consumer members and False for classic members. I think that this may be more useful for end users. What do you think? It is a bit unfortunate that we reused MemberDescription for share groups in the admin client because they are unrelated fields now. I wonder whether we should have a ShareMemberDescription. Andrew, would it be an option? Best, David On Mon, Nov 25, 2024 at 3:50 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi PoAn, > I've taken another pass over the KIP and have some more comments around > the share > group support. > > AS4: Share groups do not have the concept of a target assignment. As a > result, > the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my > point of view. ASSIGNMENT-EPOCH is better. > > AS5: Similarly for the verbose members output, the TARGET-EPOCH and > TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer > just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields. > > Thanks, > Andrew > > ________________________________________ > From: PoAn Yang <yangp...@gmail.com> > Sent: 24 November 2024 15:08 > To: dev@kafka.apache.org <dev@kafka.apache.org> > Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line > tool to support new consumer group > > Hi Chia-Ping, > > Thanks for the suggestion. > > The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, > so we can’t use it as the protocol string. > I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh. > It returns “true” if a member is in a classic group or it’s a classic > member in a consumer group. > In other cases, it returns “false”. > > I have a draft implementation in > https://github.com/apache/kafka/pull/17872. > > Thanks, > PoAn > > > On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <chia7...@gmail.com> wrote: > > > > hi PoAn > > > > `isClassic` is good to me :) > > > > Best, > > Chia-Ping > > > > PoAn Yang <yangp...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道： > > > >> Hi Chia-Ping / David, > >> > >> Thanks for the great suggestion. > >> > >> How about using “isClassic”? > >> The term like “isLegacy” may be equivocal in the future. > >> For example, if we have a newer version of consumer / share consumer, > the > >> meaning will be different for different cases. > >> If we use “isClassic”, for MemberDescription in ClassicGroupDescription, > >> the value can be always “true”. > >> In ConsumerGroupDescription, it’s “true” when it’s a classic member. > >> In ShareGroupDescription, it’s always “false”. > >> > >> Thanks, > >> PoAn > >> > >>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <chia7...@gmail.com> > wrote: > >>> > >>> hi DJ > >>> > >>>> My goal was to have a way to tell the user that his (new) consumer > group > >>> may have non-upgraded members yet. I wonder if we could use a boolean > to > >>> signal this, e.g. isLegacy. Then, in the CLI we could have a column > which > >>> is only displayed when they are non upgraded members. What do you > think? > >>> > >>> Yes, this is a good approach, but the downside is that the new isLegacy > >>> field in MemberDescription would be irrelevant for classic and shared > >>> groups. This would require thorough documentation to clarify its > purpose. > >>> > >>> Additionally, we could use a boolean instead of a string for the new > >> field > >>> in the RPC to reduce the request size. > >>> Best, > >>> Chia-Ping > >>> > >>> > >>> > >>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道： > >>> > >>>> Hi both, > >>>> > >>>> Ah, I understand what you mean now. You're saying that we have two > >> protocol > >>>> fields. One at the group level and one at the member level (the new > >> one). > >>>> The two fields are actually two different things. The former is the > >> name of > >>>> the embedded protocol used in the classic protocol whereas the latter > is > >>>> the name of the protocol/api used. This is indeed confusing. > >>>> > >>>> My goal was to have a way to tell the user that his (new) consumer > group > >>>> may have non-upgraded members yet. I wonder if we could use a boolean > to > >>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column > >> which > >>>> is only displayed when they are non upgraded members. What do you > think? > >>>> > >>>> If we cannot find a good way to represent this, we could drop it from > >> this > >>>> KIP too. > >>>> > >>>> Best, > >>>> David > >>>> > >>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <yangp...@gmail.com> wrote: > >>>> > >>>>> Hi Chia-Ping / David, > >>>>> > >>>>> Thanks for the review and suggestions. > >>>>> > >>>>>>> I don't fully follow your comment so please excuse me if I say > >>>> something > >>>>>>> wrong. > >>>>> > >>>>> I think Chia-Ping wants to say: originally, the default protocol in > >>>>> classic group is “consumer”. > >>>>> After we have a new consumer group, the protocol of a classic member > in > >>>>> consumer group is “classic”. > >>>>> The MemberDescription is used by both ClassicGroupDescription and > >>>>> ConsumerGroupDescription. > >>>>> If we add the protocol field to MemberDescription, the protocol of a > >>>>> classic member in classic group is “consumer”, but it’s “classic” in > >>>>> consumer group. > >>>>> To avoid misleading word, we can add a new collection of member > >> protocols > >>>>> to ConsumerGroupDescription only. > >>>>> The original goal is to show member protocol during consumer > migration, > >>>> so > >>>>> we don’t need to add this information to ClassicGroupDescription. > >>>>> > >>>>> Please correct me if I misunderstood anything. > >>>>> > >>>>> Thanks, > >>>>> PoAn > >>>>> > >>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <chia7...@gmail.com> > >>>> wrote: > >>>>>> > >>>>>> hi DJ > >>>>>> > >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and > >>>> change > >>>>>> return type of ModernGroup#protocolType? > >>>>>> > >>>>>> This is not my comment ... sorry for the incorrect quote :( > >>>>>> > >>>>>>> A member in a classic group should report "classic". > >>>>>> > >>>>>> I completely agree. However, in this case, > >>>>> ClassicGroupDescription#protocol > >>>>>> returns 'consumer', while > ClassicGroupDescription#members[x].protocol > >>>>>> returns 'classic'. This inconsistency feels a bit strange to me. > >>>>>> > >>>>>> Best, > >>>>>> Chia-Ping > >>>>>> > >>>>>> > >>>>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月22日 週五 > 下午6:51寫道： > >>>>>> > >>>>>>> Hi Chia, > >>>>>>> > >>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, > but > >>>> I > >>>>>>> would like to align the format with kafka-groups.sh. > >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and > >>>> change > >>>>>>> return type of ModernGroup#protocolType? > >>>>>>> > >>>>>>> My understanding is that `GroupProtocol` is used to drive the > config > >>>> of > >>>>> the > >>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you > >>>>> referring to > >>>>>>> another GroupProtocol enum? Perhaps GroupType enum? > >>>>>>> > >>>>>>>> In fact, I'm wondering if we should even add 'protocol' to > >>>>>>> MemberDescription, > >>>>>>> since the field has different values between classic and consumer > >>>>> groups—it > >>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in > >>>>> consumer > >>>>>>> groups. Yes, it seems like a typo, but it's true. This > inconsistency > >>>>> feels > >>>>>>> confusing to me. For example, users call > Admin#describeConsumerGroups > >>>>> and > >>>>>>> see that a member has the 'consumer' protocol when it's in a > classic > >>>>> group. > >>>>>>> Then, if the group is upgraded to a consumer group, calling > >>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' > >> for > >>>>> the > >>>>>>> same member. > >>>>>>> > >>>>>>> I don't fully follow your comment so please excuse me if I say > >>>> something > >>>>>>> wrong. > >>>>>>> > >>>>>>> I think that it should work as follows: > >>>>>>> * A member in a classic group should report "classic". > >>>>>>> * A member using the consumer protocol in a consumer group should > >>>> report > >>>>>>> "consumer". > >>>>>>> * A member using the classic protocol in a consumer group should > >>>> report > >>>>>>> "classic". > >>>>>>> * A member in a share group should report "share". > >>>>>>> > >>>>>>>> If all we want to do is distinguish classic members in the > >>>>>>> ConsumerGroupDescription, then we shouldn't modify the common > >>>>>>> MemberDescription structure. Instead, we could add a collection to > >>>>>>> ConsumerGroupDescription to trace members using the classic > protocol, > >>>>> which > >>>>>>> would allow us to enrich the output of command-line tools. > >>>>>>> > >>>>>>> Personally, I believe that using different collections per protocol > >>>>> types > >>>>>>> is not necessary. A field in the member is more than enough. > >>>>>>> > >>>>>>> Best, > >>>>>>> DJ > >>>>>>> > >>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai < > chia7...@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> hi PoAn > >>>>>>>> > >>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, > but > >>>> I > >>>>>>>> would like to align the format with kafka-groups.sh. > >>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and > >>>>> change > >>>>>>>> return type of ModernGroup#protocolType? > >>>>>>>> > >>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in > >>>>> classic > >>>>>>>> groups, the protocol can be an arbitrary string, so we can't use > an > >>>>> enum. > >>>>>>>> > >>>>>>>> In fact, I'm wondering if we should even add 'protocol' to > >>>>>>>> MemberDescription, > >>>>>>>> since the field has different values between classic and consumer > >>>>>>> groups—it > >>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in > >>>>>>> consumer > >>>>>>>> groups. Yes, it seems like a typo, but it's true. This > inconsistency > >>>>>>> feels > >>>>>>>> confusing to me. For example, users call > >> Admin#describeConsumerGroups > >>>>> and > >>>>>>>> see that a member has the 'consumer' protocol when it's in a > classic > >>>>>>> group. > >>>>>>>> Then, if the group is upgraded to a consumer group, calling > >>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' > >>>> for > >>>>>>> the > >>>>>>>> same member. > >>>>>>>> > >>>>>>>> If all we want to do is distinguish classic members in the > >>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common > >>>>>>>> MemberDescription structure. Instead, we could add a collection to > >>>>>>>> ConsumerGroupDescription to trace members using the classic > >> protocol, > >>>>>>> which > >>>>>>>> would allow us to enrich the output of command-line tools. > >>>>>>>> Best, > >>>>>>>> Chia-Ping > >>>>>>>> > >>>>>>>> PoAn Yang <yangp...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道： > >>>>>>>> > >>>>>>>>> Hi Lianet / Jeff, > >>>>>>>>> > >>>>>>>>> Thanks for the review. > >>>>>>>>> > >>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show > >> offsets > >>>>>>>>> information. > >>>>>>>>> Change to use “—describe —state —verbose” to show group level > >>>>>>> information > >>>>>>>>> in kafka-share-groups.sh. > >>>>>>>>> > >>>>>>>>> LM6: Update the example. > >>>>>>>>> > >>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in > >>>>>>>>> ShareGroupCommandOptions. > >>>>>>>>> > >>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT > >>>> value. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> PoAn > >>>>>>>>> > >>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <jeffkb...@apache.org> > >>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi PoAn, > >>>>>>>>>> > >>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing > >>>>>>>>>> classic group's `--members --verbose` as well, specifically the > >>>>>>>>>> "ASSIGNMENT" column to include the topic name? > >>>>>>>>>> > >>>>>>>>>> Thanks! > >>>>>>>>>> > >>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote: > >>>>>>>>>>> Thanks for the updates PoAn! > >>>>>>>>>>> > >>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the > >> header > >>>>>>> is > >>>>>>>>> fine > >>>>>>>>>>> but the example is still missing the --state argument. > >>>>>>>>>>> > >>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, > should > >>>> we > >>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I > think > >>>>>>> it's > >>>>>>>>>>> confusing because it's presented as if the option already > exists > >>>> and > >>>>>>>>> we're > >>>>>>>>>>> only changing the output (which is the case for > >>>>>>> kafka-consumer-groups > >>>>>>>>> but > >>>>>>>>>>> not for kafka-share-groups) > >>>>>>>>>>> > >>>>>>>>>>> That's all on my side. Thanks! > >>>>>>>>>>> > >>>>>>>>>>> Lianet > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangp...@gmail.com > > > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Lianet / Jeff, > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks for review and suggestions. > >>>>>>>>>>>> > >>>>>>>>>>>> LM4: At group level, the protocol field is from > >>>>>>>>> ModernGroup#protocolType. > >>>>>>>>>>>> It’s a string. > >>>>>>>>>>>> The value is defined in different places like > >>>>>>>> ShareGroup#PROTOCOL_TYPE > >>>>>>>>> and > >>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE. > >>>>>>>>>>>> In KIP-1043 < > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh > >>>>>>>>>> , > >>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” / > >>>>>>> “consumer” / > >>>>>>>>>>>> “share”. > >>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, > but I > >>>>>>>> would > >>>>>>>>>>>> like to align the format with kafka-groups.sh. > >>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol > >> and > >>>>>>>>> change > >>>>>>>>>>>> return type of ModernGroup#protocolType? > >>>>>>>>>>>> > >>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in > >> the > >>>>>>>>> title. > >>>>>>>>>>>> You’re correct. I would like to use `—describe —state > —verbose` > >>>> to > >>>>>>>> show > >>>>>>>>>>>> group level information. > >>>>>>>>>>>> For `—describe —verbose`, it will show output as same as > >>>> `—describe > >>>>>>>>>>>> —offsets —verbose`, > >>>>>>>>>>>> because `—describe` also shows output as same as `—describe > >>>>>>>> —offsets`. > >>>>>>>>>>>> > >>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but > >> the > >>>>>>>>> classic > >>>>>>>>>>>> group will use the same format as well. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> PoAn > >>>>>>>>>>>> > >>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <jeffkb...@apache.org> > >>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for the KIP. I have some questions: > >>>>>>>>>>>>> > >>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe > >> --verbose` > >>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to > >>>> printing > >>>>>>>> the > >>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument? > >>>> Also, > >>>>>>>>> would > >>>>>>>>>>>> that mean the default `--describe $group --verbose` command > >> would > >>>>>>> not > >>>>>>>>> print > >>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would > >>>> it? > >>>>>>>>>>>>> > >>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing > >>>>>>> "ASSIGNMENT" > >>>>>>>>>>>> column under `--members --verbose` does group the topic > >>>> partitions > >>>>>>> by > >>>>>>>>> topic > >>>>>>>>>>>> but does not prefix the grouping with the topic name like > you're > >>>>>>>>> proposal: > >>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same > format > >>>>>>> for > >>>>>>>>> the > >>>>>>>>>>>> classic group as well? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote: > >>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to > >>>>>>> MemberDescription. > >>>>>>>>>>>> Should > >>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead > of > >>>>>>>>> String? > >>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the > >>>>>>>> protocol > >>>>>>>>> in > >>>>>>>>>>>>>> use, just missing Share I notice). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> LM5. Regarding the change to the output for > >>>>>>> kafka-consumer-groups, > >>>>>>>>> the > >>>>>>>>>>>>>> command shown does not include the —state option, but the > >>>> output > >>>>>>>>> shows > >>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we > >>>> want > >>>>>>> to > >>>>>>>>>>>> modify > >>>>>>>>>>>>>> the output only when we describe a group with the —state > >>>> —verbose > >>>>>>>>>>>> option, > >>>>>>>>>>>>>> is my understanding right? If my understanding is right > we’re > >>>>>>> just > >>>>>>>>>>>> missing > >>>>>>>>>>>>>> adding the —state in the example, and the KIP does not > >>>> introduce > >>>>>>>> any > >>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would > >>>> mean > >>>>>>> a > >>>>>>>>>>>> bigger > >>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is > >>>> not > >>>>>>>> the > >>>>>>>>>>>>>> intention?) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Lianet > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang < > yangp...@gmail.com > >>> > >>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Chia-Ping, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks for the review and suggestions. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in > >>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to > >>>>>>> check > >>>>>>>>>>>> whether > >>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer” > >>>>>>> protocol. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share” > >>>> protocol. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, > >> and > >>>>>>>>> “share” > >>>>>>>>>>>>>>> protocol. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> BTW, the vote thread is in > >>>>>>>>>>>>>>> > >>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai < > >>>>>>>> chia7...@apache.org> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> hi PoAn > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> CI0: We have to bump the version of > >>>>>>> ConsumerGroupDescribeRequest > >>>>>>>>> as > >>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan > to > >>>>>>> use > >>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return > >>>>>>> either > >>>>>>>>>>>>>>> "classic" or "consumer", right? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by > `ShareGroupDescription` > >>>>>>> too, > >>>>>>>> so > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", > right? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value > of > >>>> the > >>>>>>>>>>>> protocol > >>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in > KIP-1043 > >>>>>>> that > >>>>>>>>>>>> lists all > >>>>>>>>>>>>>>> possible protocol names. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> [0] > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote: > >>>>>>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks for the reminder. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP. > >>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to > >>>> “0-1”. > >>>>>>>>>>>>>>>>> Add a new field “Protocol” to > >> ConsumerGroupDescribeResponse. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot > >>>>>>>>>>>> <dja...@confluent.io.INVALID> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes > to > >>>>>>> the > >>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface > >> section? > >>>>>>> We > >>>>>>>>>>>>>>> basically > >>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>> David > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang < > >>>>>>> yangp...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription, > >>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information > >>>> when > >>>>>>>>> users > >>>>>>>>>>>>>>> describe > >>>>>>>>>>>>>>>>>>> members. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote > >>>>>>> thread > >>>>>>>>>>>> today. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot > >>>>>>>>>>>>>>> <dja...@confluent.io.INVALID> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change > >> the > >>>>>>>>> output > >>>>>>>>>>>> of > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> tool too many times. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>> David > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang < > >>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Hi David / Andrew, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up > >>>>>>>>>>>>>>> kafka-share-groups.sh > >>>>>>>>>>>>>>>>>>>>> implementation. > >>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think > >>>> that > >>>>>>>>> using > >>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>> format > >>>>>>>>>>>>>>>>>>>>> is more clear. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> The new format will be like > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition > IDs. > >>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same > >> topic > >>>>>>>> name. > >>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a > vote > >>>>>>>> thread > >>>>>>>>>>>> next > >>>>>>>>>>>>>>>>>>> Monday. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < > >>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a > >>>> more > >>>>>>>>>>>>>>> appropriate > >>>>>>>>>>>>>>>>>>>>> missing value, but > >>>>>>>>>>>>>>>>>>>>>> you got there first. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include > >>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the > >>>>>>>>>>>>>>>>>>>>>> command line. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up > the > >>>>>>>>>>>>>>> implementation of > >>>>>>>>>>>>>>>>>>>>> the share groups > >>>>>>>>>>>>>>>>>>>>>> part of this if it helps. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <yangp...@gmail.com> > >>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48 > >>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> > >>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend > >>>>>>>> kafka-consumer-groups > >>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like > to > >>>> get > >>>>>>>>> this > >>>>>>>>>>>> in > >>>>>>>>>>>>>>> AK > >>>>>>>>>>>>>>>>>>> 4.0 > >>>>>>>>>>>>>>>>>>>>> as well. I will do my best. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and > >>>>>>>>> TARGET-ASSIGNMENT-EPOCH > >>>>>>>>>>>>>>> before > >>>>>>>>>>>>>>>>>>>>> #MEMBERS. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value > >>>> “-“. > >>>>>>>>>>>>>>> (reference < > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 > >>>>>>>>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH > >> CURRENT-ASSIGNMENT > >>>>>>>>>>>>>>> TARGET-EPOCH > >>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT. > >>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output > >>>>>>> (reference > >>>>>>>> < > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 > >>>>>>>>>>>>>>>>>>>> ). > >>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more > >> clear. > >>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update > both > >>>>>>>> output > >>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only > has > >>>>>>>>> protocol > >>>>>>>>>>>>>>> type at > >>>>>>>>>>>>>>>>>>>>> group level. > >>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and > >>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData > >>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level. > >>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before > >>>>>>> CURRENT-OFFSET. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot > >>>>>>>>>>>>>>> <dja...@confluent.io.INVALID > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor > >>>>>>>> comments/suggestions: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I > would > >>>>>>>>> suggest > >>>>>>>>>>>>>>> putting > >>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before > >>>>>>>>> `#MEMBERS`. > >>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will > >>>>>>> print > >>>>>>>>> out > >>>>>>>>>>>> N/A > >>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this > >>>> correct? > >>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members > >>>> --verbose`, > >>>>>>> I > >>>>>>>>>>>> wonder > >>>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms: > >>>> CURRENT-EPOCH > >>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT > >>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the > >>>>>>>>> GROUP-EPOCH > >>>>>>>>>>>>>>>>>>> because > >>>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value > `(0)` > >>>>>>> used > >>>>>>>>> for > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we > will > >>>>>>> print > >>>>>>>>> out > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> list > >>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`. > >>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should > >>>> also > >>>>>>>> add > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> protocol > >>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is > >>>>>>> possible > >>>>>>>> to > >>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>>> `classic` > >>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` > group > >>>>>>>> during > >>>>>>>>> an > >>>>>>>>>>>>>>> online > >>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer > >>>>>>>> protocol. > >>>>>>>>>>>> Having > >>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What > do > >>>> you > >>>>>>>>> think? > >>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets > >>>> --verbose`, > >>>>>>> I > >>>>>>>>> would > >>>>>>>>>>>>>>>>>>> suggest > >>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK > 4.0 > >>>> as > >>>>>>> it > >>>>>>>>>>>>>>> changes the > >>>>>>>>>>>>>>>>>>>>>>> output of the command. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>> DJ > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang < > >>>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information > >> when > >>>>>>>> users > >>>>>>>>>>>> give > >>>>>>>>>>>>>>>>>>>>> —verbose > >>>>>>>>>>>>>>>>>>>>>>>> option. > >>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target > >>>>>>>> assignment > >>>>>>>>>>>>>>> epoch. > >>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above > >> information, > >>>>>>>>> member > >>>>>>>>>>>>>>> epoch, > >>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>> target assignment. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with > >> KIP-848 > >>>>>>>>>>>> definition. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so > I > >>>> use > >>>>>>>>>>>> Optional > >>>>>>>>>>>>>>>>>>>>> fields in > >>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription. > >>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must > have > >>>>>>>> epoch, > >>>>>>>>> so > >>>>>>>>>>>> I > >>>>>>>>>>>>>>> use > >>>>>>>>>>>>>>>>>>> int > >>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level > >>>>>>>>> information > >>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>> —describe > >>>>>>>>>>>>>>>>>>>>>>>> —verbose. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe > >>>>>>> —offsets > >>>>>>>>>>>>>>> —verbose. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks. > >>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. < > >>>>>>>> liane...@gmail.com > >>>>>>>>>> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of > >>>>>>> moving > >>>>>>>>> this > >>>>>>>>>>>>>>> into a > >>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only > >>>> attempt > >>>>>>> to > >>>>>>>>> make > >>>>>>>>>>>>>>> sense > >>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying > >> to > >>>>>>>> view > >>>>>>>>> the > >>>>>>>>>>>>>>> group > >>>>>>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more > sense > >>>> to > >>>>>>>> me, > >>>>>>>>>>>> even > >>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both > >>>>>>> issues, > >>>>>>>>> my > >>>>>>>>>>>>>>> take is > >>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect > >>>>>>>> member-epoch, > >>>>>>>>>>>>>>> whoever > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs > at > >>>>>>> all. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional > (don't > >>>> we > >>>>>>>>> expect > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> always > >>>>>>>>>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker > side > >>>> for > >>>>>>>> the > >>>>>>>>>>>> group > >>>>>>>>>>>>>>>>>>> ones, > >>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>> client side for the member) > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option > in > >>>> the > >>>>>>>>>>>> proposed > >>>>>>>>>>>>>>>>>>>>> changes? > >>>>>>>>>>>>>>>>>>>>>>>> ( > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state > >>>>>>>>>>>>>>>>>>>>>>>>> ). > >>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that > example, > >>>>>>> but > >>>>>>>>> it’s > >>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>> because > >>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's > >> because > >>>>>>> of > >>>>>>>>> the > >>>>>>>>>>>>>>> change > >>>>>>>>>>>>>>>>>>>>>>>> proposed > >>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), > and > >>>> the > >>>>>>>>>>>> changes > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems > >>>>>>> confusing, > >>>>>>>>> got > >>>>>>>>>>>> me > >>>>>>>>>>>>>>>>>>>>> looking > >>>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed > >>>> unrelated). > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are > >>>>>>>>> conceptually > >>>>>>>>>>>>>>> at the > >>>>>>>>>>>>>>>>>>>>>>>> group > >>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member > >> level. > >>>>>>> So > >>>>>>>>>>>> wonder > >>>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose > >>>> option) > >>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and > target > >>>>>>>>> assignment > >>>>>>>>>>>>>>> epoch > >>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member > >>>>>>>> epoch > >>>>>>>>>>>>>>> (along > >>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch) > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Lianet > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield > < > >>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are > >>>> most > >>>>>>>>>>>> useful to > >>>>>>>>>>>>>>>>>>>>> people > >>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand > >>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some > detail. > >>>> The > >>>>>>>>>>>>>>> information > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable > >>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of > >>>>>>>>>>>> KIP-848/932. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh > >>>> that > >>>>>>>> you > >>>>>>>>>>>> add a > >>>>>>>>>>>>>>>>>>>>>>>> --members > >>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option > >>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the > output > >>>>>>> for > >>>>>>>>> that > >>>>>>>>>>>>>>> option, > >>>>>>>>>>>>>>>>>>>>>>>> rather > >>>>>>>>>>>>>>>>>>>>>>>>>> than changing the > >>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for > >>>>>>>>>>>> kafka-consumer-groups.sh > >>>>>>>>>>>>>>>>>>>>> --members > >>>>>>>>>>>>>>>>>>>>>>>>>> and only add the > >>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use > MEMBER-EPOCH > >>>>>>>>> instead of > >>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH. > >>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the > >>>>>>>>> terminology of > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> KIPs. > >>>>>>>>>>>>>>>>>>>>>>>> For > >>>>>>>>>>>>>>>>>>>>>>>>>> one thing, > >>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client > such > >>>> as > >>>>>>>>>>>>>>>>>>>>> MemberDescription. > >>>>>>>>>>>>>>>>>>>>>>>>>> It's not a > >>>>>>>>>>>>>>>>>>>>>>>>>> new term. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pay...@apache.org> > >>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55 > >>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> > >>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend > >>>>>>>>>>>> kafka-consumer-groups > >>>>>>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>>>>>>>>> line > >>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review! > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for > >>>>>>> kafka-share-groups.sh > >>>>>>>> to > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> KIP. > >>>>>>>>>>>>>>>>>>>>> Could > >>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use > >>>>>>>>> MEMBER-EPOCH > >>>>>>>>>>>>>>> here, > >>>>>>>>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>>>>>>>>> may > >>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and > >>>>>>> MEMBER. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank, > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP! > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for > >>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, > >>>> etc. ? > >>>>>>>>> Even > >>>>>>>>>>>> if > >>>>>>>>>>>>>>> we do > >>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group > >>>>>>>> implementation > >>>>>>>>>>>> may > >>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in > >> the > >>>>>>>> KIP. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not > MEMBER-EPOCH? > >>>>>>> That > >>>>>>>>> would > >>>>>>>>>>>>>>> seem > >>>>>>>>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>>>>>>>>>>> consistent. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang < > >>>>>>>>>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of > >>>>>>> KIP-1099. > >>>>>>>>> This > >>>>>>>>>>>> KIP > >>>>>>>>>>>>>>>>>>>>> enhances > >>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state > >>>> which > >>>>>>> is > >>>>>>>>>>>>>>> introduced > >>>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command > >>>> line > >>>>>>>>> tool > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> support > >>>>>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache > Software > >>>>>>>>> Foundation > >>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico] > >>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>> > >>>> > >> > >> > >> >
