Hello Kirk ! Thank you for your comments !
KT1: Yes, you are correct. The issue is not unique to the initial heartbeat; there can always be cases where the broker might lose connection with a member. KT2: Currently, if the client doesn't have a member ID and the memberEpoch equals 0, the coordinator will generate a UUID as the member ID for the client. However, at the RPC level, the member ID is sent as a literal string, meaning there are no restrictions on the format at this level. This also reminds me that we haven't reached a final conclusion on how to enforce the use of UUIDs. >From our previous discussions, I recall two possible approaches: The first is to validate the UUID on the server side, and if it's not valid, throw an exception to the client. The second is to document a recommendation for clients to use UUIDs as member IDs, without strictly enforcing it. I think it's time to decide on the approach we want to take. KT3: Yes, "session" can be considered synonymous with "membership" in this context. KT4: Thank you for pointing that out. I will update the wording to specifically say this behavior is for consumers. Thanks again for your comments. Best regards, TengYao Kirk True <k...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道： > Hi TengYao! > > Sorry for being late to the discussion... > > After reading the thread and then the KIP, I had a few questions/comments: > > KT1: In Motivation, it states: "This scenario can result in the broker > registering a new member for which it will never receive a proper leave > request.” Just to be clear, the broker will always have cases where it > might lose connection with a member. That’s not unique to the initial > heartbeat, right? > > KT2: There was a bit of back and forth about format of the member ID. From > what I gathered in the thread, the member ID is still defined in the RPC as > a string and not a UUID, right? The KIP states that the “client must > generate a UUID as the member ID” and that the “server will validate that a > valid UUID is provided.” Is that a change for the server, or is it already > enforced as a UUID? > > KT3: Lianet mentioned some confusion over the use of the word “session.” > Isn’t “session” synonymous with “membership?” > > KT4: Under “Member ID Lifecycle,” it states: "The client should reuse the > same UUID as the member ID for all heartbeats and rejoin attempts to > maintain continuity within the group.” Could we change the first part of > that to “The Consumer instance should…” We do have lifetimes that extend > past the lifetime of a client instance (such as the transaction ID). > > Thanks, > Kirk > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <kiting...@gmail.com> wrote: > > > > Hi David, > > > > Thank you for pointing that out. > > I have updated the content of the KIP based on Lianet's and your > feedback. > > Please take a look and let me know your thoughts. > > > > Best regards, > > TengYao > > > > David Jacot <dja...@confluent.io.invalid> 於 2024年8月29日 週四 下午3:20寫道： > > > >> Hi TengYao, > >> > >> Thanks for the update. I haven't fully read it yet but I will soon. > >> > >> LM4: This is incorrect. The consumer must keep its member id during its > >> entire lifetime (until the process stops or dies). The protocol > stipulates > >> that a member must rejoin with the same member id and the member epoch > set > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member to > >> resynchronize itself. We should not change this behavior. I think that > we > >> should see the client side generation id as an incarnation id of the > >> application. It is generated once and kept until it stops or dies. > >> > >> Best, > >> David > >> > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <kiting...@gmail.com> > wrote: > >> > >>> Hello Lianet ! > >>> > >>> Thanks for the reviews and suggestions! > >>> > >>> LM1: Indeed, we plan to enforce client-side ID generation in the > future, > >>> and it is not an alternative. I will change the title accordingly. > >>> > >>> LM2: Yes, that's the expectation. I will add that statement to the > public > >>> interface section. > >>> > >>> LM3: Thank you for the high-level perspective review. I think you're > >> right; > >>> our intention isn't very clear since it was placed at the end of the > >>> section. I will try to rephrase that section to make it more obvious. > >>> > >>> LM4: Regarding the definition of "session" in this KIP, I believe it > >> refers > >>> to the period between the *first-time heartbeat* and when the *consumer > >>> leaves the group* (whether through a graceful shutdown or a heartbeat > >>> timeout). The consumer should reuse its UUID if it has been generated > >>> before. The only situation in which it will regenerate the UUID is if > the > >>> coordinator finds that there is already a consumer with the same UUID. > >>> IIRC, the coordinator should compare the member epochs, and the > >>> later-joined consumer should be fenced off by the coordinator due to > >> having > >>> a lower member epoch. Once the consumer receives a > `FENCED_MEMBER_EPOCH` > >>> error, it will generate a new UUID and attempt to rejoin. I will > clarify > >>> this in the KIP. > >>> > >>> Thanks again for your reviews, I really appreciate it. > >>> > >>> Best regards, > >>> TengYao > >>> > >>> Lianet M. <liane...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道： > >>> > >>>> Hello TengYao! Thanks for taking on this issue, we've been going > around > >>> it > >>>> for a while. > >>>> > >>>> LM1: About the title of the KIP: "Enable ID Generation for Clients > over > >>> the > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it hints that > >>>> we're adding it as an alternative (which was discussed and discarded, > >> in > >>>> favour of really enforcing it). It's also missing the core change imo, > >>>> which is "where" the generation happens. So, maybe more to the point > >> with > >>>> something along the lines of "Client-side generated ID for clients > over > >>>> ConsumerGroupHeartbeat RPC"? > >>>> > >>>> LM2: On the public interfaces section, the KIP states that "the server > >>> will > >>>> reject the request", but we should agree on the specific error type. I > >>>> expect it should fail with an InvalidRequestException, is that the > >>>> intention? (This was also suggested in the discussion thread before > but > >>> is > >>>> not in the KIP). > >>>> > >>>> LM3. Related to my previous point, I find that to be the true > >>> public-facing > >>>> change (member ID mandatory at the protocol level), but it's only at > >> the > >>>> end of the Public interfaces changes, kind of lost among details of > how > >>>> we're going to do it. Should we rephrase that section with the actual > >>>> change first, and the hows after (ex. Bumping the version is not the > >>>> public-facing change in this case, it's just the mechanism to properly > >>>> introduce our change) > >>>> > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we will > "Verify > >>>> that the UUID remains consistent across all subsequent heartbeats > >> during > >>>> the session". What is this "session" referring to here? I would expect > >>> that > >>>> the UUID is associated to a consumer instance (generated for the > >> consumer > >>>> the first time it needs to send a HB if it doesn't have the UUID yet. > >>> From > >>>> there on, every time it needs to send a "first HB" again, it will > reuse > >>> its > >>>> UUID, is that the intention? Note that we should consider that the > same > >>>> consumer instance may have many "first heartbeats", meaning heartbeats > >> to > >>>> join the group when it's not part of it (ex. consumer unsubscribe + > >>>> subscribe, fenced, stale). Is this the intention or are you > considering > >>> the > >>>> lifetime differently? We should clarify it in the KIP. > >>>> > >>>> Thanks! > >>>> > >>>> Lianet > >>>> > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <kiting...@gmail.com> > >> wrote: > >>>> > >>>>> Hi everyone, > >>>>> > >>>>> I have revised this KIP multiple times based on the feedback from our > >>>>> discussions. > >>>>> I would greatly appreciate it if you could review it when you have > >> the > >>>>> time. > >>>>> If there are no further comments or suggestions, I plan to proceed > >> with > >>>>> initiating a vote soon. > >>>>> > >>>>> Best regards, > >>>>> TengYao > >>>>> > >>>>> TengYao Chi <kiting...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道： > >>>>> > >>>>>> Hi Andrew, > >>>>>> Thank you for your previous feedback and insights. > >>>>>> Your contributions have added valuable perspectives to the > >>> discussions. > >>>>>> And we also benefit from the comparison of different solutions. > >>>>>> I’m also looking forward to seeing an initial version in KIP-932, > >> as > >>> it > >>>>>> will provide a good reference for future implementations. > >>>>>> > >>>>>> Regarding your comment on AS2, I wanted to clarify that my > >>>> specification > >>>>>> references org.apache.kafka.common.Uuid. > >>>>>> I believe we’re referring to the same class, and it might just be a > >>>> small > >>>>>> oversight due to the busy schedule. > >>>>>> > >>>>>> I want to express my gratitude once again for your many insightful > >>>>>> comments, which have helped the discussion progress smoothly. > >>>>>> > >>>>>> Best regards, > >>>>>> TengYao > >>>>>> > >>>>>> > >>>>>> Andrew Schofield <andrew_schofi...@live.com> 於 2024年8月22日 週四 > >>>> 下午11:28寫道： > >>>>>> > >>>>>>> Hi TengYao, > >>>>>>> I’ve been reading through the comments and I’m happy that the > >> lobby > >>>>>>> approach has not gained support. > >>>>>>> > >>>>>>> Assuming that this KIP is voted, I will be happy to change KIP-932 > >>> so > >>>>>>> that it only supports client-side member ID generation. Because > >> that > >>>> KIP > >>>>>>> is still > >>>>>>> under development, I can do this in the first version of > >>>>>>> ShareGroupHeartbeat. > >>>>>>> > >>>>>>> AS2: For the encoding section, I suppose the specific encoding > >> which > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Andrew > >>>>>>> > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <kiting...@gmail.com> > >>> wrote: > >>>>>>>> > >>>>>>>> Hello Apoorv, > >>>>>>>> Thank you for your feedback. > >>>>>>>> Regarding the questions you raised, unfortunately, this KIP > >> cannot > >>>>>>>> guarantee the order of heartbeats. As with many classic > >>> distributed > >>>>>>> system > >>>>>>>> challenges, what we can do is make our best effort to ensure > >> that > >>>>> there > >>>>>>> are > >>>>>>>> no idle members or stale assignments under normal circumstances. > >>>>>>>> > >>>>>>>> As for the lobby approach, I’m not a fan of it because it > >> requires > >>>>>>> adding a > >>>>>>>> mechanism to maintain client state within the ConsumerGroup, > >>> which, > >>>> in > >>>>>>> my > >>>>>>>> view, resembles something like a two-phase commit. This would > >>>>> introduce > >>>>>>>> more complexity than the proposal in this KIP, which is > >> something > >>> we > >>>>>>> want > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol, and > >>> while > >>>>> the > >>>>>>>> lobby approach is a good one, I believe it is not the right fit > >>> for > >>>>> this > >>>>>>>> particular situation. > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> TengYao > >>>>>>>> > >>>>>>>> TengYao Chi <kiting...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道： > >>>>>>>> > >>>>>>>>> Hi David, > >>>>>>>>> > >>>>>>>>> I really appreciate your review and suggestions. As I am still > >>>>> gaining > >>>>>>>>> experience in writing KIPs, your input has been incredibly > >>>> helpful. I > >>>>>>> am > >>>>>>>>> currently applying your suggestions to the KIP and will > >> complete > >>> it > >>>>> as > >>>>>>> soon > >>>>>>>>> as possible. > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a > >> conclusion > >>>>>>> yet.(So > >>>>>>>>> far according to this thread) > >>>>>>>>> However, I will review the current implementation in the Kafka > >>>> `Uuid` > >>>>>>>>> class and include a brief specification in the KIP. > >>>>>>>>> > >>>>>>>>> Once again, thank you so much for your help. > >>>>>>>>> > >>>>>>>>> Best regards, > >>>>>>>>> TengYao > >>>>>>>>> > >>>>>>>>> Chia-Ping Tsai <chia7...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道： > >>>>>>>>> > >>>>>>>>>> hi Apoorv > >>>>>>>>>> > >>>>>>>>>>> As the memberId is now known to the client, and client might > >>> send > >>>>> the > >>>>>>>>>> leave > >>>>>>>>>> group heartbeat on shutdown prior to receiving the initial > >>>> heartbeat > >>>>>>>>>> response. If that's true then how do we guarantee that the 2 > >>>>> requests > >>>>>>> to > >>>>>>>>>> join and leave will be processed in order, which could still > >>> leave > >>>>>>> stale > >>>>>>>>>> members or throw unknown member id exceptions? > >>>>>>>>>> > >>>>>>>>>> This is definitely a good question. the short answer: no > >>> guarantee > >>>>> but > >>>>>>>>>> best > >>>>>>>>>> efforts > >>>>>>>>>> > >>>>>>>>>> Please notice the root cause is "we have no enough time to > >> wait > >>>>>>> member id > >>>>>>>>>> (response) when closing consumer". Sadly, we can' guarantee > >> the > >>>>>>> request > >>>>>>>>>> order due to the same reason. > >>>>>>>>>> > >>>>>>>>>> However, in contrast to previous behavior, there is one big > >>>> benefit > >>>>>>> of new > >>>>>>>>>> approach - we can try STONITH because we know the member id > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Chia-Ping > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Apoorv Mittal <apoorvmitta...@gmail.com> 於 2024年8月14日 週三 > >>>> 下午8:55寫道： > >>>>>>>>>> > >>>>>>>>>>> Hi TengYao, > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew > >>>> mentioned > >>>>> as > >>>>>>>>>> AS1. > >>>>>>>>>>> > >>>>>>>>>>> As the memberId is now known to the client, and client might > >>> send > >>>>> the > >>>>>>>>>> leave > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the initial > >>>>> heartbeat > >>>>>>>>>>> response. If that's true then how do we guarantee that the 2 > >>>>>>> requests to > >>>>>>>>>>> join and leave will be processed in order, which could still > >>>> leave > >>>>>>> stale > >>>>>>>>>>> members or throw unknown member id exceptions? > >>>>>>>>>>> > >>>>>>>>>>> Though the client side member id generation is helpful which > >>> will > >>>>>>>>>> represent > >>>>>>>>>>> the same group perspective as from client and broker's end. > >>> But I > >>>>>>> think > >>>>>>>>>> the > >>>>>>>>>>> major concern we want to solve here is Stale Partition > >>>> Assignments > >>>>>>> which > >>>>>>>>>>> might still exist with the new approach. I am leaning towards > >>> the > >>>>>>>>>>> suggestion mentioned by Andrew where partition assignment > >>>> triggers > >>>>> on > >>>>>>>>>>> subsequent heartbeat when client acknowledges the initial > >>>>> heartbeat, > >>>>>>>>>>> delayed partition assignment. > >>>>>>>>>>> > >>>>>>>>>>> Though on a separate note, I have a different question. What > >>>>> happens > >>>>>>>>>> when > >>>>>>>>>>> there is an issue with the client which sends the initial > >>>> heartbeat > >>>>>>>>>> without > >>>>>>>>>>> memberId, then crashes and restarts? I think we must be > >>>>> re-triggering > >>>>>>>>>>> assignments and expiring members only after the heartbeat > >>> session > >>>>>>>>>> timeout? > >>>>>>>>>>> If that's true then shall delayed partition assignment can > >> help > >>>>>>> benefit > >>>>>>>>>> us > >>>>>>>>>>> from this situation as well? > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Apoorv Mittal > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot > >>>>>>>>>> <dja...@confluent.io.invalid> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Andrew, > >>>>>>>>>>>> > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes things > >>>> more > >>>>>>>>>>>> complicated and it would require changing the records on the > >>>>> server > >>>>>>>>>> too. > >>>>>>>>>>>> This is why I initially suggested the rejected alternative > >> #2 > >>>>> which > >>>>>>> is > >>>>>>>>>>>> pretty close but also not perfect. > >>>>>>>>>>>> > >>>>>>>>>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat > >> API > >>>>>>> already > >>>>>>>>>>>> supports generating the member id on the client so we don't > >>> need > >>>>> any > >>>>>>>>>>>> conditional logic on the client side. This is actually what > >> we > >>>>>>> wanted > >>>>>>>>>> to > >>>>>>>>>>> do > >>>>>>>>>>>> in the first place but the idea got pushed back by Magnus > >> back > >>>>> then > >>>>>>>>>>> because > >>>>>>>>>>>> generating uuid from librdkafka required a new dependency. > >> It > >>>>> turns > >>>>>>>>>> out > >>>>>>>>>>>> that librdkafka has that dependency today. In retrospect, we > >>>>> should > >>>>>>>>>> have > >>>>>>>>>>>> pushed back on this. Long story short, we can just do it. > >> The > >>>>>>>>>> proposal in > >>>>>>>>>>>> this KIP is to make the member id required in future > >> versions. > >>>> We > >>>>>>>>>> could > >>>>>>>>>>>> also decide not to do it and to keep supporting both > >>>> approaches. I > >>>>>>>>>> would > >>>>>>>>>>>> also be fine with this. > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> David > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield < > >>>>>>>>>>>> andrew_schofi...@live.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi TengYao, > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more try to > >>>>> persuade. > >>>>>>>>>>>>> I feel that I will need to follow the approach with KIP-932 > >>>> when > >>>>>>>>>> we’ve > >>>>>>>>>>>>> made a decision, so I do have more than a passing interest > >> in > >>>>> this. > >>>>>>>>>>>>> > >>>>>>>>>>>>> A group member in the lobby is in the group, but it does > >> not > >>>> have > >>>>>>>>>> any > >>>>>>>>>>>>> assignments. A member of a consumer group can have no > >>> assigned > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic > >> with 4 > >>>>>>>>>>>> partitions), > >>>>>>>>>>>>> so it’s a situation that consumer group members already > >>> expect. > >>>>>>>>>>>>> > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API > >>>>> versioning. > >>>>>>>>>>>>> But, there is a cost - the behaviour is different depending > >>> on > >>>>> the > >>>>>>>>>> RPC > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but we’re > >>>> already > >>>>>>>>>> adding > >>>>>>>>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat. > >>> That’s > >>>> a > >>>>>>>>>> pity. > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Andrew > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi < > >> kiting...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hello Andrew > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting the > >>>>>>>>>> discussion > >>>>>>>>>>>>> going. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> To AS1: > >>>>>>>>>>>>>> In the current scenario where the server generates the > >> UUID, > >>>> if > >>>>>>>>>> the > >>>>>>>>>>>>> client > >>>>>>>>>>>>>> shuts down before receiving the memberId generated by the > >> GC > >>>>>>>>>>>> (regardless > >>>>>>>>>>>>> of > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will > >> still > >>>> have > >>>>>>>>>> to > >>>>>>>>>>>> wait > >>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t know > >>> its > >>>>>>>>>>> memberId. > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the idempotency > >>>> issue, > >>>>>>>>>> but > >>>>>>>>>>> it > >>>>>>>>>>>>> can > >>>>>>>>>>>>>> better handle shutdown scenarios under normal > >> circumstances > >>>>>>>>>> because > >>>>>>>>>>> the > >>>>>>>>>>>>>> client always knows its memberId. Even if the client shuts > >>>> down > >>>>>>>>>>>>> immediately > >>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a > >>> graceful > >>>>>>>>>>> shutdown > >>>>>>>>>>>>> and > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the situation > >> and > >>>>>>>>>> remove > >>>>>>>>>>> the > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address the > >>>> issue > >>>>>>>>>> where > >>>>>>>>>>>> the > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the client > >>>>> leaving > >>>>>>>>>>>>> without > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced throughput > >> and > >>>>>>>>>> limited > >>>>>>>>>>>>>> scalability. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The solution you suggest has also been proposed by David. > >>> The > >>>>>>>>>> concern > >>>>>>>>>>>>> with > >>>>>>>>>>>>>> this approach is that it introduces additional complexity > >>> for > >>>>>>>>>>>>>> compatibility, as the new server would not immediately add > >>> the > >>>>>>>>>> member > >>>>>>>>>>>> to > >>>>>>>>>>>>>> the group, while the old server would. This requires > >> clients > >>>> to > >>>>>>>>>>>>>> differentiate whether their memberId has been added to the > >>>> group > >>>>>>>>>> or > >>>>>>>>>>>> not, > >>>>>>>>>>>>>> which could result in unexpected logs. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best Regards, > >>>>>>>>>>>>>> TengYao > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Andrew Schofield <andrew_schofi...@live.com> 於 2024年8月14日 > >>> 週三 > >>>>>>>>>>>> 上午12:29寫道： > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi TengYao, > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different way > >> to > >>>>> close > >>>>>>>>>>> what > >>>>>>>>>>>>>>> is quite a small window. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not > >>> idempotent, > >>>>> but > >>>>>>>>>>> this > >>>>>>>>>>>>>>> remains > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not idempotent. > >>> If > >>>>> the > >>>>>>>>>>>> client > >>>>>>>>>>>>>>> makes its > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will > >> still > >>>> have > >>>>>>>>>>> added > >>>>>>>>>>>>>>> the member to the group and it will hang around until the > >>>>> session > >>>>>>>>>>>>> expires. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member ID in > >>>>>>>>>> response to > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> first > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING state > >>> with > >>>>> no > >>>>>>>>>>>>>>> assignments until the client sends the next heartbeat, > >> thus > >>>>>>>>>>> confirming > >>>>>>>>>>>>> it > >>>>>>>>>>>>>>> has received the member ID. This would not be a protocol > >>>> change > >>>>>>>>>> at > >>>>>>>>>>>> all, > >>>>>>>>>>>>>>> just > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby > >> until > >>>> it’s > >>>>>>>>>>>>> comfirmed > >>>>>>>>>>>>>>> it knows its member ID. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi < > >>> kiting...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Chia-Ping, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for review and suggestions. > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly. > >>>>>>>>>>>>>>>> Please take a look. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>> TengYao > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Chia-Ping Tsai <chia7...@apache.org> 於 2024年8月13日 週二 > >>>>> 下午9:45寫道： > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> hi TengYao > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> thanks for this KIP. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after behavior > >> in > >>>> the > >>>>>>>>>>>>> "Proposed > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB having > >>> member > >>>>> id > >>>>>>>>>>>>>>> generated by > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will > >>> generate > >>>>> one > >>>>>>>>>>> and > >>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must" have > >>> member > >>>>> ID. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2) could you please write the version number explicitly > >>> in > >>>>> the > >>>>>>>>>> KIP > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it > >> always > >>>>>>>>>> generate > >>>>>>>>>>>>> member > >>>>>>>>>>>>>>>>> ID on client-side even though that is not restricted? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote: > >>>>>>>>>>>>>>>>>> Hello everyone, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on KIP-1082, > >>>> which > >>>>>>>>>>>> proposes > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the > >>>>>>>>>>> ConsumerGroupHeartbeat > >>>>>>>>>>>>> RPC. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082 > >>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you think, > >> and I > >>>>> would > >>>>>>>>>>>>>>> appreciate > >>>>>>>>>>>>>>>>>> any suggestions and feedback. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>>>> TengYao > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>> > >>> > >> > >
