Hi David > I tend to believe that generating a uuid on the client should not be that difficult so I am not too concerned about it. How do you see this,
You're right. It's not that difficult, and we should have more confidence in third-party libraries. hi TengYao Could you please summarize the discussion on "duplicate member ID" and add it to KIP-1082? Also, please remove "The member is also removed from the group." from KIP-848 Best, Chia-Ping David Jacot <dja...@confluent.io.invalid> 於 2024年9月23日 週一 下午10:43寫道： > Hi Chia-Ping, > > > However, my concern is that consumers are now required to generate the > member ID, so as a responsible server, we need to handle the incorrect > client behavior. At least, based on David's comment, we can create another > member id when it is already used and it is rejoining the group (member > epoch = 0). > > I don't disagree with the concern. However, I don't see how we could make > it work in practice. Let's say that a member XYZ sends a heartbeat with > epoch zero. The server cannot know whether it is a new member joining for > the first time or an existing member rejoining because it was fenced (e.g. > wrong epoch, session timed out, etc.). We cannot differentiate the two > cases. Hence we cannot handle it on the server side. The only viable > alternative is to keep the member id generation on the server side if we > are concerned by clients not able to generate unique uuids on their side. I > tend to believe that generating a uuid on the client should not be that > difficult so I am not too concerned about it. How do you see > this, Chia-Ping? > > Cheers, > David > > On Mon, Sep 23, 2024 at 4:13 PM Chia-Ping Tsai <chia7...@gmail.com> wrote: > > > hi David > > > > > For instance, we could check whether the member id is already used and > we > > could regenerate another one if it is. However, I tend to believe that > the > > risk of collision within a group is neglectable. > > > > I agree the risk is negligible when the member ID is generated by the > > server, as we can always ensure the server behaves responsibly. 😀 > > Therefore, I don't believe the additional check for the member ID is > > necessary if the member id is created by server. > > > > However, my concern is that consumers are now required to generate the > > member ID, so as a responsible server, we need to handle the incorrect > > client behavior. At least, based on David's comment, we can create > another > > member id when it is already used and it is rejoining the group (member > > epoch = 0). > > > > By the way, another related issue is that a duplicate consumer may > > inadvertently terminate another consumer, as it will always send a > > LEAVE_GROUP request upon closing, even if it hasn't received a heartbeat > > response. However, I tend to believe the risk is negligible 😀 > > > > Best, > > Chia-Ping > > > > > > David Jacot <dja...@confluent.io.invalid> 於 2024年9月23日 週一 下午9:07寫道： > > > > > Hi Chia-Ping, > > > > > > > Checks whether the member epoch matches the member epoch in its > current > > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is > also > > > removed from the group. > > > > > > We did not implement it like this in the end. We keep the member and > only > > > request him to re-synchronise himself by returning the > > FENCED_MEMBER_EPOCH > > > error. We can remove it from the KIP. > > > > > > > Given that member id is the incarnation of consumer, it is hard to > > detect > > > the scenario of "two consumers have the same member id" (it may be > caused > > > by the lost response). I guess the best approach is the later -created > > > consumer should be idle with fenced error, and we should add this to > the > > > KIP (maybe we should revise the error message carried by > > > `FencedMemberEpochException`). BUT, we need to rethink the approach if > > > above description "The member is also removed from the group" is true > > > > > > I don't think that we have a reliable way to detect whether two members > > end > > > up using the same member id. I don't think that it is possible if we > > > generate the member id on the client side. By the way, we don't handle > > this > > > case either on the server side today. We blindly generate a member id > and > > > use it. However, we could do better on the server side. For instance, > we > > > could check whether the member id is already used and we could > regenerate > > > another one if it is. However, I tend to believe that the risk of > > collision > > > within a group is neglectable. > > > > > > Best, > > > David > > > > > > On Fri, Sep 20, 2024 at 8:11 PM Chia-Ping Tsai <chia7...@gmail.com> > > wrote: > > > > > > > wait a minute. There is a description about fenced error in the > KIP-848 > > > > > > > > > Checks whether the member epoch matches the member epoch in its > > current > > > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is > > also > > > > removed from the group. > > > > > > > > I failed to find the code of implementing "The member is also removed > > > from > > > > the group.". I will appreciate it if someone can share the code link > to > > > me. > > > > > > > > Let's assume it will be implemented eventually, and I feel that will > > > cause > > > > a serious issue as the consumers having the same member id will > remove > > > the > > > > existing member and then create itself member? > > > > > > > > > Once the client receives this exception, it should consider the > > > exception > > > > as a serious error and stop the process, and the user will need to > > > assign a > > > > new memberId. > > > > > > > > Given that member id is the incarnation of consumer, it is hard to > > detect > > > > the scenario of "two consumers have the same member id" (it may be > > caused > > > > by the lost response). I guess the best approach is the later > -created > > > > consumer should be idle with fenced error, and we should add this to > > the > > > > KIP (maybe we should revise the error message carried by > > > > `FencedMemberEpochException`). BUT, we need to rethink the approach > if > > > > above description "The member is also removed from the group" is true > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > > > > > > > > > TengYao Chi <kiting...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道： > > > > > > > > > Hi Chia-Ping, > > > > > > > > > > Thanks for pointing out this issue. > > > > > > > > > > I’m thinking that maybe we might need to define a new Exception to > > > handle > > > > > this scenario. > > > > > Once the client receives this exception, it should consider the > > > exception > > > > > as a serious error and stop the process, and the user will need to > > > > assign a > > > > > new memberId. > > > > > What do you think? > > > > > > > > > > Best Regards, > > > > > TengYao > > > > > > > > > > Chia-Ping Tsai <chia7...@apache.org> 於 2024年9月20日 週五 下午7:00寫道： > > > > > > > > > > > > This part is not clear either. It basically says that if a > member > > > > joins > > > > > > with an existing member id but a different epoch, it will be > > fenced. > > > > Then > > > > > > it must rejoin with the same member id and epoch zero. This is > > > already > > > > > the > > > > > > current behavior and it does not help with detecting duplicates, > > > right? > > > > > > > > > > > > The duplicates issue is interesting. The member id is generated > on > > > > > > server-side by UUID before, so it seems to me the fenced member > > epoch > > > > > > happens only if the client miss the response with bumped epoch, > and > > > > this > > > > > > case is recoverable. > > > > > > > > > > > > however, v1 brings a different fenced epoch scenario. Users, now, > > > have > > > > > > responsibility to generate unique member id, hence clients may > > > > encounter > > > > > > infinite fenced error if there are >1 clients having same member > id > > > due > > > > > to > > > > > > config error (or other engineer error) > > > > > > > > > > > > Maybe we can highlight this scenario if we start to requires > > clients > > > to > > > > > > generate unique member id. > > > > > > > > > > > > Best, > > > > > > Chia-Ping > > > > > > > > > > > > On 2024/09/19 18:36:48 David Jacot wrote: > > > > > > > Hi, > > > > > > > > > > > > > > Thanks for the update. I have a few nits: > > > > > > > > > > > > > > > If the member ID is null or empty, the server will reject the > > > > request > > > > > > > with an InvalidRequestException. > > > > > > > We should clarify that this should only apply to version >= 1. > > > > > > > > > > > > > > > The consumer instance must generate a member ID, and this ID > > > should > > > > > > > remain consistent for the duration of the consumer's session. > > > Here, a > > > > > > > "session" is defined as the period from the consumer's first > > > > heartbeat > > > > > > > until it leaves the group, either through a graceful shutdown, > a > > > > > > heartbeat > > > > > > > timeout, or the process stopping or dying. The consumer > instance > > > > should > > > > > > > reuse the same member ID for all heartbeats and rejoin attempts > > to > > > > > > maintain > > > > > > > continuity within the group. > > > > > > > > > > > > > > This part is not clear to me. When the member leaves the group, > > it > > > > > should > > > > > > > not reset the member id. I would rather say that the member > must > > > > > generate > > > > > > > its member id when it starts and it must keep it until the > > process > > > > > stops. > > > > > > > It is basically an incarnation of the process. > > > > > > > > > > > > > > > If a conflict arises where the member ID generated by the > > client > > > is > > > > > > > detected to be a duplicate within the same group (for example, > > the > > > > same > > > > > > > member ID is associated with another active member in the > group), > > > the > > > > > > > server will handle this by comparing the memberEpoch values of > > the > > > > > > > conflicting members. The member with the lower memberEpoch is > > > > > considered > > > > > > > outdated and will be fenced off by the server. When this > occurs, > > > the > > > > > > server > > > > > > > responds with a FENCED_MEMBER_EPOCH error to the client, > > signaling > > > it > > > > > to > > > > > > > rejoin the group with the same member ID while resetting the > > > > > memberEpoch > > > > > > to > > > > > > > zero. This ensures that the client properly resynchronizes and > > > > > maintains > > > > > > > the continuity and consistency of the group membership. > > > > > > > > > > > > > > This part is not clear either. It basically says that if a > member > > > > joins > > > > > > > with an existing member id but a different epoch, it will be > > > fenced. > > > > > Then > > > > > > > it must rejoin with the same member id and epoch zero. This is > > > > already > > > > > > the > > > > > > > current behavior and it does not help with detecting > duplicates, > > > > right? > > > > > > > Should we just remove the paragraph? > > > > > > > > > > > > > > > A member ID mismatch occurs within a session: If the server > > > > detects a > > > > > > > mismatch between the provided member ID and the expected member > > ID > > > > for > > > > > an > > > > > > > ongoing session, it should return a UNKNOWN_MEMBER_ID error. > > > > > > > > > > > > > > How could we detect a mismatch between the provided and the > > > expected > > > > > > member > > > > > > > id? My understanding is that we can only know whether the > > provided > > > > > member > > > > > > > id exists or not. This is already implemented. > > > > > > > > > > > > > > Thanks, > > > > > > > David > > > > > > > > > > > > > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi < > kiting...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > Since this KIP has been fully discussed, I will initiate a > vote > > > for > > > > > it > > > > > > next > > > > > > > > Monday. > > > > > > > > Thank you and have a nice weekend. > > > > > > > > > > > > > > > > Best regards, > > > > > > > > TengYao > > > > > > > > > > > > > > > > TengYao Chi <kiting...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道： > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > KT2: It looks like everyone who has expressed an opinion > > > supports > > > > > the > > > > > > > > > second option: “Document a recommendation for clients to > use > > > > UUIDs > > > > > as > > > > > > > > > member IDs, without strictly enforcing it.” > > > > > > > > > I have updated the KIP accordingly. > > > > > > > > > Please take a look, and let me know if you have any > thoughts > > or > > > > > > feedback. > > > > > > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > TengYao > > > > > > > > > > > > > > > > > > Chia-Ping Tsai <chia7...@gmail.com> 於 2024年8月30日 週五 > > 下午9:56寫道： > > > > > > > > > > > > > > > > > >> hi TengYao > > > > > > > > >> > > > > > > > > >> KT2: +1 to second approach > > > > > > > > >> > > > > > > > > >> Best, > > > > > > > > >> Chia-Ping > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> David Jacot <dja...@confluent.io.invalid> 於 2024年8月30日 週五 > > > > > 下午9:15寫道： > > > > > > > > >> > > > > > > > > >> > Hi TengYao, > > > > > > > > >> > > > > > > > > > >> > KT2: I don't think that we can realistically validate > the > > > uuid > > > > > on > > > > > > the > > > > > > > > >> > server. It is basically a string of chars. So I lean > > towards > > > > > > having a > > > > > > > > >> good > > > > > > > > >> > recommendation in the KIP and in the document of the > field > > > in > > > > > the > > > > > > > > RPC's > > > > > > > > >> > definition. > > > > > > > > >> > > > > > > > > > >> > Best, > > > > > > > > >> > David > > > > > > > > >> > > > > > > > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi < > > > > > kiting...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > >> > > Hello Kirk ! > > > > > > > > >> > > > > > > > > > > >> > > Thank you for your comments ! > > > > > > > > >> > > > > > > > > > > >> > > KT1: Yes, you are correct. The issue is not unique to > > the > > > > > > initial > > > > > > > > >> > > heartbeat; there can always be cases where the broker > > > might > > > > > lose > > > > > > > > >> > connection > > > > > > > > >> > > with a member. > > > > > > > > >> > > > > > > > > > > >> > > KT2: Currently, if the client doesn't have a member ID > > and > > > > the > > > > > > > > >> > memberEpoch > > > > > > > > >> > > equals 0, the coordinator will generate a UUID as the > > > member > > > > > ID > > > > > > for > > > > > > > > >> the > > > > > > > > >> > > client. However, at the RPC level, the member ID is > sent > > > as > > > > a > > > > > > > > literal > > > > > > > > >> > > string, meaning there are no restrictions on the > format > > at > > > > > this > > > > > > > > level. > > > > > > > > >> > > This also reminds me that we haven't reached a final > > > > > conclusion > > > > > > on > > > > > > > > >> how to > > > > > > > > >> > > enforce the use of UUIDs. > > > > > > > > >> > > From our previous discussions, I recall two possible > > > > > approaches: > > > > > > > > >> > > The first is to validate the UUID on the server side, > > and > > > if > > > > > > it's > > > > > > > > not > > > > > > > > >> > > valid, throw an exception to the client. > > > > > > > > >> > > The second is to document a recommendation for clients > > to > > > > use > > > > > > UUIDs > > > > > > > > as > > > > > > > > >> > > member IDs, without strictly enforcing it. > > > > > > > > >> > > I think it's time to decide on the approach we want to > > > take. > > > > > > > > >> > > > > > > > > > > >> > > KT3: Yes, "session" can be considered synonymous with > > > > > > "membership" > > > > > > > > in > > > > > > > > >> > this > > > > > > > > >> > > context. > > > > > > > > >> > > > > > > > > > > >> > > KT4: Thank you for pointing that out. I will update > the > > > > > wording > > > > > > to > > > > > > > > >> > > specifically say this behavior is for consumers. > > > > > > > > >> > > > > > > > > > > >> > > Thanks again for your comments. > > > > > > > > >> > > > > > > > > > > >> > > Best regards, > > > > > > > > >> > > TengYao > > > > > > > > >> > > > > > > > > > > >> > > Kirk True <k...@kirktrue.pro> 於 2024年8月30日 週五 > > 上午12:39寫道： > > > > > > > > >> > > > > > > > > > > >> > > > Hi TengYao! > > > > > > > > >> > > > > > > > > > > > >> > > > Sorry for being late to the discussion... > > > > > > > > >> > > > > > > > > > > > >> > > > After reading the thread and then the KIP, I had a > few > > > > > > > > >> > > questions/comments: > > > > > > > > >> > > > > > > > > > > > >> > > > KT1: In Motivation, it states: "This scenario can > > result > > > > in > > > > > > the > > > > > > > > >> broker > > > > > > > > >> > > > registering a new member for which it will never > > > receive a > > > > > > proper > > > > > > > > >> leave > > > > > > > > >> > > > request.” Just to be clear, the broker will always > > have > > > > > cases > > > > > > > > where > > > > > > > > >> it > > > > > > > > >> > > > might lose connection with a member. That’s not > unique > > > to > > > > > the > > > > > > > > >> initial > > > > > > > > >> > > > heartbeat, right? > > > > > > > > >> > > > > > > > > > > > >> > > > KT2: There was a bit of back and forth about format > of > > > the > > > > > > member > > > > > > > > >> ID. > > > > > > > > >> > > From > > > > > > > > >> > > > what I gathered in the thread, the member ID is > still > > > > > defined > > > > > > in > > > > > > > > the > > > > > > > > >> > RPC > > > > > > > > >> > > as > > > > > > > > >> > > > a string and not a UUID, right? The KIP states that > > the > > > > > > “client > > > > > > > > must > > > > > > > > >> > > > generate a UUID as the member ID” and that the > “server > > > > will > > > > > > > > validate > > > > > > > > >> > > that a > > > > > > > > >> > > > valid UUID is provided.” Is that a change for the > > > server, > > > > or > > > > > > is it > > > > > > > > >> > > already > > > > > > > > >> > > > enforced as a UUID? > > > > > > > > >> > > > > > > > > > > > >> > > > KT3: Lianet mentioned some confusion over the use of > > the > > > > > word > > > > > > > > >> > “session.” > > > > > > > > >> > > > Isn’t “session” synonymous with “membership?” > > > > > > > > >> > > > > > > > > > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The > > client > > > > > > should > > > > > > > > >> reuse > > > > > > > > >> > the > > > > > > > > >> > > > same UUID as the member ID for all heartbeats and > > rejoin > > > > > > attempts > > > > > > > > to > > > > > > > > >> > > > maintain continuity within the group.” Could we > change > > > the > > > > > > first > > > > > > > > >> part > > > > > > > > >> > of > > > > > > > > >> > > > that to “The Consumer instance should…” We do have > > > > lifetimes > > > > > > that > > > > > > > > >> > extend > > > > > > > > >> > > > past the lifetime of a client instance (such as the > > > > > > transaction > > > > > > > > ID). > > > > > > > > >> > > > > > > > > > > > >> > > > Thanks, > > > > > > > > >> > > > Kirk > > > > > > > > >> > > > > > > > > > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi < > > > > > > kiting...@gmail.com> > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > Hi David, > > > > > > > > >> > > > > > > > > > > > > >> > > > > Thank you for pointing that out. > > > > > > > > >> > > > > I have updated the content of the KIP based on > > > Lianet's > > > > > and > > > > > > your > > > > > > > > >> > > > feedback. > > > > > > > > >> > > > > Please take a look and let me know your thoughts. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Best regards, > > > > > > > > >> > > > > TengYao > > > > > > > > >> > > > > > > > > > > > > >> > > > > David Jacot <dja...@confluent.io.invalid> 於 > > > 2024年8月29日 > > > > 週四 > > > > > > > > >> 下午3:20寫道： > > > > > > > > >> > > > > > > > > > > > > >> > > > >> Hi TengYao, > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Thanks for the update. I haven't fully read it > yet > > > but > > > > I > > > > > > will > > > > > > > > >> soon. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> LM4: This is incorrect. The consumer must keep > its > > > > member > > > > > > id > > > > > > > > >> during > > > > > > > > >> > > its > > > > > > > > >> > > > >> entire lifetime (until the process stops or > dies). > > > The > > > > > > protocol > > > > > > > > >> > > > stipulates > > > > > > > > >> > > > >> that a member must rejoin with the same member id > > and > > > > the > > > > > > > > member > > > > > > > > >> > epoch > > > > > > > > >> > > > set > > > > > > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This > > > allows > > > > > the > > > > > > > > >> member > > > > > > > > >> > to > > > > > > > > >> > > > >> resynchronize itself. We should not change this > > > > > behavior. I > > > > > > > > think > > > > > > > > >> > that > > > > > > > > >> > > > we > > > > > > > > >> > > > >> should see the client side generation id as an > > > > > incarnation > > > > > > id > > > > > > > > of > > > > > > > > >> the > > > > > > > > >> > > > >> application. It is generated once and kept until > it > > > > stops > > > > > > or > > > > > > > > >> dies. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Best, > > > > > > > > >> > > > >> David > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi < > > > > > > > > kiting...@gmail.com > > > > > > > > >> > > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > >> > > > > > > > > >> > > > >>> Hello Lianet ! > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Thanks for the reviews and suggestions! > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID > > > > > generation > > > > > > in > > > > > > > > the > > > > > > > > >> > > > future, > > > > > > > > >> > > > >>> and it is not an alternative. I will change the > > > title > > > > > > > > >> accordingly. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> LM2: Yes, that's the expectation. I will add > that > > > > > > statement to > > > > > > > > >> the > > > > > > > > >> > > > public > > > > > > > > >> > > > >>> interface section. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> LM3: Thank you for the high-level perspective > > > review. > > > > I > > > > > > think > > > > > > > > >> > you're > > > > > > > > >> > > > >> right; > > > > > > > > >> > > > >>> our intention isn't very clear since it was > placed > > > at > > > > > the > > > > > > end > > > > > > > > of > > > > > > > > >> > the > > > > > > > > >> > > > >>> section. I will try to rephrase that section to > > make > > > > it > > > > > > more > > > > > > > > >> > obvious. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> LM4: Regarding the definition of "session" in > this > > > > KIP, > > > > > I > > > > > > > > >> believe > > > > > > > > >> > it > > > > > > > > >> > > > >> refers > > > > > > > > >> > > > >>> to the period between the *first-time heartbeat* > > and > > > > > when > > > > > > the > > > > > > > > >> > > *consumer > > > > > > > > >> > > > >>> leaves the group* (whether through a graceful > > > shutdown > > > > > or > > > > > > a > > > > > > > > >> > heartbeat > > > > > > > > >> > > > >>> timeout). The consumer should reuse its UUID if > it > > > has > > > > > > been > > > > > > > > >> > generated > > > > > > > > >> > > > >>> before. The only situation in which it will > > > regenerate > > > > > the > > > > > > > > UUID > > > > > > > > >> is > > > > > > > > >> > if > > > > > > > > >> > > > the > > > > > > > > >> > > > >>> coordinator finds that there is already a > consumer > > > > with > > > > > > the > > > > > > > > same > > > > > > > > >> > > UUID. > > > > > > > > >> > > > >>> IIRC, the coordinator should compare the member > > > > epochs, > > > > > > and > > > > > > > > the > > > > > > > > >> > > > >>> later-joined consumer should be fenced off by > the > > > > > > coordinator > > > > > > > > >> due > > > > > > > > >> > to > > > > > > > > >> > > > >> having > > > > > > > > >> > > > >>> a lower member epoch. Once the consumer > receives a > > > > > > > > >> > > > `FENCED_MEMBER_EPOCH` > > > > > > > > >> > > > >>> error, it will generate a new UUID and attempt > to > > > > > rejoin. > > > > > > I > > > > > > > > will > > > > > > > > >> > > > clarify > > > > > > > > >> > > > >>> this in the KIP. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Thanks again for your reviews, I really > appreciate > > > it. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Best regards, > > > > > > > > >> > > > >>> TengYao > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Lianet M. <liane...@gmail.com> 於 2024年8月28日 週三 > > > > > 下午7:12寫道： > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue, > > > we've > > > > > > been > > > > > > > > >> going > > > > > > > > >> > > > around > > > > > > > > >> > > > >>> it > > > > > > > > >> > > > >>>> for a while. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID > > > > Generation > > > > > > for > > > > > > > > >> Clients > > > > > > > > >> > > > over > > > > > > > > >> > > > >>> the > > > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it > confusing > > > > > because > > > > > > it > > > > > > > > >> hints > > > > > > > > >> > > that > > > > > > > > >> > > > >>>> we're adding it as an alternative (which was > > > > discussed > > > > > > and > > > > > > > > >> > > discarded, > > > > > > > > >> > > > >> in > > > > > > > > >> > > > >>>> favour of really enforcing it). It's also > missing > > > the > > > > > > core > > > > > > > > >> change > > > > > > > > >> > > imo, > > > > > > > > >> > > > >>>> which is "where" the generation happens. So, > > maybe > > > > more > > > > > > to > > > > > > > > the > > > > > > > > >> > point > > > > > > > > >> > > > >> with > > > > > > > > >> > > > >>>> something along the lines of "Client-side > > generated > > > > ID > > > > > > for > > > > > > > > >> clients > > > > > > > > >> > > > over > > > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"? > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> LM2: On the public interfaces section, the KIP > > > states > > > > > > that > > > > > > > > "the > > > > > > > > >> > > server > > > > > > > > >> > > > >>> will > > > > > > > > >> > > > >>>> reject the request", but we should agree on the > > > > > specific > > > > > > > > error > > > > > > > > >> > > type. I > > > > > > > > >> > > > >>>> expect it should fail with an > > > > InvalidRequestException, > > > > > is > > > > > > > > that > > > > > > > > >> the > > > > > > > > >> > > > >>>> intention? (This was also suggested in the > > > discussion > > > > > > thread > > > > > > > > >> > before > > > > > > > > >> > > > but > > > > > > > > >> > > > >>> is > > > > > > > > >> > > > >>>> not in the KIP). > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> LM3. Related to my previous point, I find that > to > > > be > > > > > the > > > > > > true > > > > > > > > >> > > > >>> public-facing > > > > > > > > >> > > > >>>> change (member ID mandatory at the protocol > > level), > > > > but > > > > > > it's > > > > > > > > >> only > > > > > > > > >> > at > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >>>> end of the Public interfaces changes, kind of > > lost > > > > > among > > > > > > > > >> details > > > > > > > > >> > of > > > > > > > > >> > > > how > > > > > > > > >> > > > >>>> we're going to do it. Should we rephrase that > > > section > > > > > > with > > > > > > > > the > > > > > > > > >> > > actual > > > > > > > > >> > > > >>>> change first, and the hows after (ex. Bumping > the > > > > > > version is > > > > > > > > >> not > > > > > > > > >> > the > > > > > > > > >> > > > >>>> public-facing change in this case, it's just > the > > > > > > mechanism to > > > > > > > > >> > > properly > > > > > > > > >> > > > >>>> introduce our change) > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the > KIP > > > > states > > > > > > we > > > > > > > > will > > > > > > > > >> > > > "Verify > > > > > > > > >> > > > >>>> that the UUID remains consistent across all > > > > subsequent > > > > > > > > >> heartbeats > > > > > > > > >> > > > >> during > > > > > > > > >> > > > >>>> the session". What is this "session" referring > to > > > > > here? I > > > > > > > > would > > > > > > > > >> > > expect > > > > > > > > >> > > > >>> that > > > > > > > > >> > > > >>>> the UUID is associated to a consumer instance > > > > > (generated > > > > > > for > > > > > > > > >> the > > > > > > > > >> > > > >> consumer > > > > > > > > >> > > > >>>> the first time it needs to send a HB if it > > doesn't > > > > have > > > > > > the > > > > > > > > >> UUID > > > > > > > > >> > > yet. > > > > > > > > >> > > > >>> From > > > > > > > > >> > > > >>>> there on, every time it needs to send a "first > > HB" > > > > > > again, it > > > > > > > > >> will > > > > > > > > >> > > > reuse > > > > > > > > >> > > > >>> its > > > > > > > > >> > > > >>>> UUID, is that the intention? Note that we > should > > > > > consider > > > > > > > > that > > > > > > > > >> the > > > > > > > > >> > > > same > > > > > > > > >> > > > >>>> consumer instance may have many "first > > heartbeats", > > > > > > meaning > > > > > > > > >> > > heartbeats > > > > > > > > >> > > > >> to > > > > > > > > >> > > > >>>> join the group when it's not part of it (ex. > > > consumer > > > > > > > > >> unsubscribe > > > > > > > > >> > + > > > > > > > > >> > > > >>>> subscribe, fenced, stale). Is this the > intention > > or > > > > are > > > > > > you > > > > > > > > >> > > > considering > > > > > > > > >> > > > >>> the > > > > > > > > >> > > > >>>> lifetime differently? We should clarify it in > the > > > > KIP. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Thanks! > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Lianet > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi < > > > > > > > > >> kiting...@gmail.com> > > > > > > > > >> > > > >> wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>>> Hi everyone, > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>>> I have revised this KIP multiple times based > on > > > the > > > > > > feedback > > > > > > > > >> from > > > > > > > > >> > > our > > > > > > > > >> > > > >>>>> discussions. > > > > > > > > >> > > > >>>>> I would greatly appreciate it if you could > > review > > > it > > > > > > when > > > > > > > > you > > > > > > > > >> > have > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >>>>> time. > > > > > > > > >> > > > >>>>> If there are no further comments or > > suggestions, I > > > > > plan > > > > > > to > > > > > > > > >> > proceed > > > > > > > > >> > > > >> with > > > > > > > > >> > > > >>>>> initiating a vote soon. > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>>> Best regards, > > > > > > > > >> > > > >>>>> TengYao > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>>> TengYao Chi <kiting...@gmail.com> 於 > 2024年8月23日 > > 週五 > > > > > > 下午2:43寫道： > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>>>> Hi Andrew, > > > > > > > > >> > > > >>>>>> Thank you for your previous feedback and > > > insights. > > > > > > > > >> > > > >>>>>> Your contributions have added valuable > > > perspectives > > > > > to > > > > > > the > > > > > > > > >> > > > >>> discussions. > > > > > > > > >> > > > >>>>>> And we also benefit from the comparison of > > > > different > > > > > > > > >> solutions. > > > > > > > > >> > > > >>>>>> I’m also looking forward to seeing an initial > > > > version > > > > > > in > > > > > > > > >> > KIP-932, > > > > > > > > >> > > > >> as > > > > > > > > >> > > > >>> it > > > > > > > > >> > > > >>>>>> will provide a good reference for future > > > > > > implementations. > > > > > > > > >> > > > >>>>>> > > > > > > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to > > > clarify > > > > > > that my > > > > > > > > >> > > > >>>> specification > > > > > > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid. > > > > > > > > >> > > > >>>>>> I believe we’re referring to the same class, > > and > > > it > > > > > > might > > > > > > > > >> just > > > > > > > > >> > be > > > > > > > > >> > > a > > > > > > > > >> > > > >>>> small > > > > > > > > >> > > > >>>>>> oversight due to the busy schedule. > > > > > > > > >> > > > >>>>>> > > > > > > > > >> > > > >>>>>> I want to express my gratitude once again for > > > your > > > > > many > > > > > > > > >> > insightful > > > > > > > > >> > > > >>>>>> comments, which have helped the discussion > > > progress > > > > > > > > smoothly. > > > > > > > > >> > > > >>>>>> > > > > > > > > >> > > > >>>>>> Best regards, > > > > > > > > >> > > > >>>>>> TengYao > > > > > > > > >> > > > >>>>>> > > > > > > > > >> > > > >>>>>> > > > > > > > > >> > > > >>>>>> Andrew Schofield <andrew_schofi...@live.com> > 於 > > > > > > 2024年8月22日 > > > > > > > > 週四 > > > > > > > > >> > > > >>>> 下午11:28寫道： > > > > > > > > >> > > > >>>>>> > > > > > > > > >> > > > >>>>>>> Hi TengYao, > > > > > > > > >> > > > >>>>>>> I’ve been reading through the comments and > I’m > > > > happy > > > > > > that > > > > > > > > >> the > > > > > > > > >> > > > >> lobby > > > > > > > > >> > > > >>>>>>> approach has not gained support. > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be > > happy > > > > to > > > > > > change > > > > > > > > >> > > KIP-932 > > > > > > > > >> > > > >>> so > > > > > > > > >> > > > >>>>>>> that it only supports client-side member ID > > > > > > generation. > > > > > > > > >> Because > > > > > > > > >> > > > >> that > > > > > > > > >> > > > >>>> KIP > > > > > > > > >> > > > >>>>>>> is still > > > > > > > > >> > > > >>>>>>> under development, I can do this in the > first > > > > > version > > > > > > of > > > > > > > > >> > > > >>>>>>> ShareGroupHeartbeat. > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the > > > > > specific > > > > > > > > >> encoding > > > > > > > > >> > > > >> which > > > > > > > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid > > > uses. > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>>>> Thanks, > > > > > > > > >> > > > >>>>>>> Andrew > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi < > > > > > > > > kiting...@gmail.com > > > > > > > > >> > > > > > > > > > >> > > > >>> wrote: > > > > > > > > >> > > > >>>>>>>> > > > > > > > > >> > > > >>>>>>>> Hello Apoorv, > > > > > > > > >> > > > >>>>>>>> Thank you for your feedback. > > > > > > > > >> > > > >>>>>>>> Regarding the questions you raised, > > > > unfortunately, > > > > > > this > > > > > > > > KIP > > > > > > > > >> > > > >> cannot > > > > > > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with > > many > > > > > > classic > > > > > > > > >> > > > >>> distributed > > > > > > > > >> > > > >>>>>>> system > > > > > > > > >> > > > >>>>>>>> challenges, what we can do is make our best > > > > effort > > > > > to > > > > > > > > >> ensure > > > > > > > > >> > > > >> that > > > > > > > > >> > > > >>>>> there > > > > > > > > >> > > > >>>>>>> are > > > > > > > > >> > > > >>>>>>>> no idle members or stale assignments under > > > normal > > > > > > > > >> > circumstances. > > > > > > > > >> > > > >>>>>>>> > > > > > > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of > > it > > > > > > because it > > > > > > > > >> > > > >> requires > > > > > > > > >> > > > >>>>>>> adding a > > > > > > > > >> > > > >>>>>>>> mechanism to maintain client state within > the > > > > > > > > >> ConsumerGroup, > > > > > > > > >> > > > >>> which, > > > > > > > > >> > > > >>>> in > > > > > > > > >> > > > >>>>>>> my > > > > > > > > >> > > > >>>>>>>> view, resembles something like a two-phase > > > > commit. > > > > > > This > > > > > > > > >> would > > > > > > > > >> > > > >>>>> introduce > > > > > > > > >> > > > >>>>>>>> more complexity than the proposal in this > > KIP, > > > > > which > > > > > > is > > > > > > > > >> > > > >> something > > > > > > > > >> > > > >>> we > > > > > > > > >> > > > >>>>>>> want > > > > > > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the > > existing > > > > > > protocol, > > > > > > > > >> and > > > > > > > > >> > > > >>> while > > > > > > > > >> > > > >>>>> the > > > > > > > > >> > > > >>>>>>>> lobby approach is a good one, I believe it > is > > > not > > > > > the > > > > > > > > right > > > > > > > > >> > fit > > > > > > > > >> > > > >>> for > > > > > > > > >> > > > >>>>> this > > > > > > > > >> > > > >>>>>>>> particular situation. > > > > > > > > >> > > > >>>>>>>> > > > > > > > > >> > > > >>>>>>>> Best regards, > > > > > > > > >> > > > >>>>>>>> TengYao > > > > > > > > >> > > > >>>>>>>> > > > > > > > > >> > > > >>>>>>>> TengYao Chi <kiting...@gmail.com> 於 > > 2024年8月14日 > > > > 週三 > > > > > > > > >> 下午11:45寫道： > > > > > > > > >> > > > >>>>>>>> > > > > > > > > >> > > > >>>>>>>>> Hi David, > > > > > > > > >> > > > >>>>>>>>> > > > > > > > > >> > > > >>>>>>>>> I really appreciate your review and > > > suggestions. > > > > > As > > > > > > I am > > > > > > > > >> > still > > > > > > > > >> > > > >>>>> gaining > > > > > > > > >> > > > >>>>>>>>> experience in writing KIPs, your input has > > > been > > > > > > > > incredibly > > > > > > > > >> > > > >>>> helpful. I > > > > > > > > >> > > > >>>>>>> am > > > > > > > > >> > > > >>>>>>>>> currently applying your suggestions to the > > KIP > > > > and > > > > > > will > > > > > > > > >> > > > >> complete > > > > > > > > >> > > > >>> it > > > > > > > > >> > > > >>>>> as > > > > > > > > >> > > > >>>>>>> soon > > > > > > > > >> > > > >>>>>>>>> as possible. > > > > > > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we > haven’t > > > > > reached > > > > > > a > > > > > > > > >> > > > >> conclusion > > > > > > > > >> > > > >>>>>>> yet.(So > > > > > > > > >> > > > >>>>>>>>> far according to this thread) > > > > > > > > >> > > > >>>>>>>>> However, I will review the current > > > > implementation > > > > > > in the > > > > > > > > >> > Kafka > > > > > > > > >> > > > >>>> `Uuid` > > > > > > > > >> > > > >>>>>>>>> class and include a brief specification in > > the > > > > > KIP. > > > > > > > > >> > > > >>>>>>>>> > > > > > > > > >> > > > >>>>>>>>> Once again, thank you so much for your > help. > > > > > > > > >> > > > >>>>>>>>> > > > > > > > > >> > > > >>>>>>>>> Best regards, > > > > > > > > >> > > > >>>>>>>>> TengYao > > > > > > > > >> > > > >>>>>>>>> > > > > > > > > >> > > > >>>>>>>>> Chia-Ping Tsai <chia7...@gmail.com> 於 > > > > 2024年8月14日 > > > > > 週三 > > > > > > > > >> > 下午11:14寫道： > > > > > > > > >> > > > >>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> hi Apoorv > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the > > client, > > > > and > > > > > > client > > > > > > > > >> > might > > > > > > > > >> > > > >>> send > > > > > > > > >> > > > >>>>> the > > > > > > > > >> > > > >>>>>>>>>> leave > > > > > > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to > > > receiving > > > > > the > > > > > > > > >> initial > > > > > > > > >> > > > >>>> heartbeat > > > > > > > > >> > > > >>>>>>>>>> response. If that's true then how do we > > > > guarantee > > > > > > that > > > > > > > > >> the 2 > > > > > > > > >> > > > >>>>> requests > > > > > > > > >> > > > >>>>>>> to > > > > > > > > >> > > > >>>>>>>>>> join and leave will be processed in > order, > > > > which > > > > > > could > > > > > > > > >> still > > > > > > > > >> > > > >>> leave > > > > > > > > >> > > > >>>>>>> stale > > > > > > > > >> > > > >>>>>>>>>> members or throw unknown member id > > > exceptions? > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> This is definitely a good question. the > > short > > > > > > answer: > > > > > > > > no > > > > > > > > >> > > > >>> guarantee > > > > > > > > >> > > > >>>>> but > > > > > > > > >> > > > >>>>>>>>>> best > > > > > > > > >> > > > >>>>>>>>>> efforts > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> Please notice the root cause is "we have > no > > > > > enough > > > > > > time > > > > > > > > >> to > > > > > > > > >> > > > >> wait > > > > > > > > >> > > > >>>>>>> member id > > > > > > > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, > > we > > > > can' > > > > > > > > >> guarantee > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >>>>>>> request > > > > > > > > >> > > > >>>>>>>>>> order due to the same reason. > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> However, in contrast to previous > behavior, > > > > there > > > > > > is one > > > > > > > > >> big > > > > > > > > >> > > > >>>> benefit > > > > > > > > >> > > > >>>>>>> of new > > > > > > > > >> > > > >>>>>>>>>> approach - we can try STONITH because we > > know > > > > the > > > > > > > > member > > > > > > > > >> id > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> Best, > > > > > > > > >> > > > >>>>>>>>>> Chia-Ping > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> Apoorv Mittal <apoorvmitta...@gmail.com> > 於 > > > > > > 2024年8月14日 > > > > > > > > 週三 > > > > > > > > >> > > > >>>> 下午8:55寫道： > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> Hi TengYao, > > > > > > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the > > point > > > > > which > > > > > > > > Andrew > > > > > > > > >> > > > >>>> mentioned > > > > > > > > >> > > > >>>>> as > > > > > > > > >> > > > >>>>>>>>>> AS1. > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the > > client, > > > > and > > > > > > client > > > > > > > > >> > might > > > > > > > > >> > > > >>> send > > > > > > > > >> > > > >>>>> the > > > > > > > > >> > > > >>>>>>>>>> leave > > > > > > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to > > > receiving > > > > > the > > > > > > > > >> initial > > > > > > > > >> > > > >>>>> heartbeat > > > > > > > > >> > > > >>>>>>>>>>> response. If that's true then how do we > > > > > guarantee > > > > > > that > > > > > > > > >> the > > > > > > > > >> > 2 > > > > > > > > >> > > > >>>>>>> requests to > > > > > > > > >> > > > >>>>>>>>>>> join and leave will be processed in > order, > > > > which > > > > > > could > > > > > > > > >> > still > > > > > > > > >> > > > >>>> leave > > > > > > > > >> > > > >>>>>>> stale > > > > > > > > >> > > > >>>>>>>>>>> members or throw unknown member id > > > exceptions? > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> Though the client side member id > > generation > > > is > > > > > > helpful > > > > > > > > >> > which > > > > > > > > >> > > > >>> will > > > > > > > > >> > > > >>>>>>>>>> represent > > > > > > > > >> > > > >>>>>>>>>>> the same group perspective as from > client > > > and > > > > > > broker's > > > > > > > > >> end. > > > > > > > > >> > > > >>> But I > > > > > > > > >> > > > >>>>>>> think > > > > > > > > >> > > > >>>>>>>>>> the > > > > > > > > >> > > > >>>>>>>>>>> major concern we want to solve here is > > Stale > > > > > > Partition > > > > > > > > >> > > > >>>> Assignments > > > > > > > > >> > > > >>>>>>> which > > > > > > > > >> > > > >>>>>>>>>>> might still exist with the new > approach. I > > > am > > > > > > leaning > > > > > > > > >> > towards > > > > > > > > >> > > > >>> the > > > > > > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where > > > partition > > > > > > > > >> assignment > > > > > > > > >> > > > >>>> triggers > > > > > > > > >> > > > >>>>> on > > > > > > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client > > > acknowledges > > > > > the > > > > > > > > >> initial > > > > > > > > >> > > > >>>>> heartbeat, > > > > > > > > >> > > > >>>>>>>>>>> delayed partition assignment. > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a > > > different > > > > > > > > question. > > > > > > > > >> > What > > > > > > > > >> > > > >>>>> happens > > > > > > > > >> > > > >>>>>>>>>> when > > > > > > > > >> > > > >>>>>>>>>>> there is an issue with the client which > > > sends > > > > > the > > > > > > > > >> initial > > > > > > > > >> > > > >>>> heartbeat > > > > > > > > >> > > > >>>>>>>>>> without > > > > > > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I > > think > > > > we > > > > > > must > > > > > > > > be > > > > > > > > >> > > > >>>>> re-triggering > > > > > > > > >> > > > >>>>>>>>>>> assignments and expiring members only > > after > > > > the > > > > > > > > >> heartbeat > > > > > > > > >> > > > >>> session > > > > > > > > >> > > > >>>>>>>>>> timeout? > > > > > > > > >> > > > >>>>>>>>>>> If that's true then shall delayed > > partition > > > > > > assignment > > > > > > > > >> can > > > > > > > > >> > > > >> help > > > > > > > > >> > > > >>>>>>> benefit > > > > > > > > >> > > > >>>>>>>>>> us > > > > > > > > >> > > > >>>>>>>>>>> from this situation as well? > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> Regards, > > > > > > > > >> > > > >>>>>>>>>>> Apoorv Mittal > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David > > Jacot > > > > > > > > >> > > > >>>>>>>>>> <dja...@confluent.io.invalid> > > > > > > > > >> > > > >>>>>>>>>>> wrote: > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> Hi Andrew, > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby > > > approach. > > > > It > > > > > > makes > > > > > > > > >> > things > > > > > > > > >> > > > >>>> more > > > > > > > > >> > > > >>>>>>>>>>>> complicated and it would require > changing > > > the > > > > > > records > > > > > > > > >> on > > > > > > > > >> > the > > > > > > > > >> > > > >>>>> server > > > > > > > > >> > > > >>>>>>>>>> too. > > > > > > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the > > > > rejected > > > > > > > > >> alternative > > > > > > > > >> > > > >> #2 > > > > > > > > >> > > > >>>>> which > > > > > > > > >> > > > >>>>>>> is > > > > > > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect. > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The > > > > > > > > >> ConsumerGroupHeartbeat > > > > > > > > >> > > > >> API > > > > > > > > >> > > > >>>>>>> already > > > > > > > > >> > > > >>>>>>>>>>>> supports generating the member id on > the > > > > client > > > > > > so we > > > > > > > > >> > don't > > > > > > > > >> > > > >>> need > > > > > > > > >> > > > >>>>> any > > > > > > > > >> > > > >>>>>>>>>>>> conditional logic on the client side. > > This > > > is > > > > > > > > actually > > > > > > > > >> > what > > > > > > > > >> > > > >> we > > > > > > > > >> > > > >>>>>>> wanted > > > > > > > > >> > > > >>>>>>>>>> to > > > > > > > > >> > > > >>>>>>>>>>> do > > > > > > > > >> > > > >>>>>>>>>>>> in the first place but the idea got > > pushed > > > > back > > > > > > by > > > > > > > > >> Magnus > > > > > > > > >> > > > >> back > > > > > > > > >> > > > >>>>> then > > > > > > > > >> > > > >>>>>>>>>>> because > > > > > > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka > required > > a > > > > new > > > > > > > > >> dependency. > > > > > > > > >> > > > >> It > > > > > > > > >> > > > >>>>> turns > > > > > > > > >> > > > >>>>>>>>>> out > > > > > > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency > > today. > > > In > > > > > > > > >> retrospect, > > > > > > > > >> > we > > > > > > > > >> > > > >>>>> should > > > > > > > > >> > > > >>>>>>>>>> have > > > > > > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, > we > > > can > > > > > > just do > > > > > > > > >> it. > > > > > > > > >> > > > >> The > > > > > > > > >> > > > >>>>>>>>>> proposal in > > > > > > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id > > required > > > in > > > > > > future > > > > > > > > >> > > > >> versions. > > > > > > > > >> > > > >>>> We > > > > > > > > >> > > > >>>>>>>>>> could > > > > > > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep > > > > supporting > > > > > > both > > > > > > > > >> > > > >>>> approaches. I > > > > > > > > >> > > > >>>>>>>>>> would > > > > > > > > >> > > > >>>>>>>>>>>> also be fine with this. > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> Best, > > > > > > > > >> > > > >>>>>>>>>>>> David > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew > > > > > > Schofield < > > > > > > > > >> > > > >>>>>>>>>>>> andrew_schofi...@live.com> > > > > > > > > >> > > > >>>>>>>>>>>> wrote: > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> Hi TengYao, > > > > > > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have > just > > > one > > > > > > more > > > > > > > > try > > > > > > > > >> to > > > > > > > > >> > > > >>>>> persuade. > > > > > > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the > > > > approach > > > > > > with > > > > > > > > >> > KIP-932 > > > > > > > > >> > > > >>>> when > > > > > > > > >> > > > >>>>>>>>>> we’ve > > > > > > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more > than > > a > > > > > > passing > > > > > > > > >> > interest > > > > > > > > >> > > > >> in > > > > > > > > >> > > > >>>>> this. > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the > > > group, > > > > > > but it > > > > > > > > >> does > > > > > > > > >> > > > >> not > > > > > > > > >> > > > >>>> have > > > > > > > > >> > > > >>>>>>>>>> any > > > > > > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer > > group > > > > can > > > > > > have > > > > > > > > no > > > > > > > > >> > > > >>> assigned > > > > > > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members > > > subscribed > > > > > to a > > > > > > > > topic > > > > > > > > >> > > > >> with 4 > > > > > > > > >> > > > >>>>>>>>>>>> partitions), > > > > > > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer > group > > > > > members > > > > > > > > >> already > > > > > > > > >> > > > >>> expect. > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way > that > > > we > > > > > > handle > > > > > > > > API > > > > > > > > >> > > > >>>>> versioning. > > > > > > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour > is > > > > > > different > > > > > > > > >> > depending > > > > > > > > >> > > > >>> on > > > > > > > > >> > > > >>>>> the > > > > > > > > >> > > > >>>>>>>>>> RPC > > > > > > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of > > > > completion, > > > > > > but > > > > > > > > >> we’re > > > > > > > > >> > > > >>>> already > > > > > > > > >> > > > >>>>>>>>>> adding > > > > > > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for > > > > > > > > >> ConsumerGroupHeartbeat. > > > > > > > > >> > > > >>> That’s > > > > > > > > >> > > > >>>> a > > > > > > > > >> > > > >>>>>>>>>> pity. > > > > > > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s > > unfortunate. > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > >>>>>>>>>>>>> Andrew > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao > Chi < > > > > > > > > >> > > > >> kiting...@gmail.com> > > > > > > > > >> > > > >>>>>>>>>> wrote: > > > > > > > > >> > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>> Hello Andrew > > > > > > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful > > suggestions > > > > and > > > > > > > > getting > > > > > > > > >> > the > > > > > > > > >> > > > >>>>>>>>>> discussion > > > > > > > > >> > > > >>>>>>>>>>>>> going. > > > > > > > > >> > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>> To AS1: > > > > > > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the > > server > > > > > > generates > > > > > > > > >> the > > > > > > > > >> > > > >> UUID, > > > > > > > > >> > > > >>>> if > > > > > > > > >> > > > >>>>>>>>>> the > > > > > > > > >> > > > >>>>>>>>>>>>> client > > > > > > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the > > memberId > > > > > > generated > > > > > > > > by > > > > > > > > >> > the > > > > > > > > >> > > > >> GC > > > > > > > > >> > > > >>>>>>>>>>>> (regardless > > > > > > > > >> > > > >>>>>>>>>>>>> of > > > > > > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or > > not), > > > > the > > > > > > GC > > > > > > > > will > > > > > > > > >> > > > >> still > > > > > > > > >> > > > >>>> have > > > > > > > > >> > > > >>>>>>>>>> to > > > > > > > > >> > > > >>>>>>>>>>>> wait > > > > > > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the > > > > client > > > > > > > > doesn’t > > > > > > > > >> > know > > > > > > > > >> > > > >>> its > > > > > > > > >> > > > >>>>>>>>>>> memberId. > > > > > > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely > > resolve > > > > the > > > > > > > > >> > idempotency > > > > > > > > >> > > > >>>> issue, > > > > > > > > >> > > > >>>>>>>>>> but > > > > > > > > >> > > > >>>>>>>>>>> it > > > > > > > > >> > > > >>>>>>>>>>>>> can > > > > > > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios > under > > > > normal > > > > > > > > >> > > > >> circumstances > > > > > > > > >> > > > >>>>>>>>>> because > > > > > > > > >> > > > >>>>>>>>>>> the > > > > > > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId. > Even > > if > > > > the > > > > > > > > client > > > > > > > > >> > shuts > > > > > > > > >> > > > >>>> down > > > > > > > > >> > > > >>>>>>>>>>>>> immediately > > > > > > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long > as > > > it > > > > > > > > performs a > > > > > > > > >> > > > >>> graceful > > > > > > > > >> > > > >>>>>>>>>>> shutdown > > > > > > > > >> > > > >>>>>>>>>>>>> and > > > > > > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can > > > manage > > > > > the > > > > > > > > >> situation > > > > > > > > >> > > > >> and > > > > > > > > >> > > > >>>>>>>>>> remove > > > > > > > > >> > > > >>>>>>>>>>> the > > > > > > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this > KIP > > > is > > > > to > > > > > > > > address > > > > > > > > >> > the > > > > > > > > >> > > > >>>> issue > > > > > > > > >> > > > >>>>>>>>>> where > > > > > > > > >> > > > >>>>>>>>>>>> the > > > > > > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat > > timeout > > > > due > > > > > > to the > > > > > > > > >> > client > > > > > > > > >> > > > >>>>> leaving > > > > > > > > >> > > > >>>>>>>>>>>>> without > > > > > > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to > > > > reduced > > > > > > > > >> throughput > > > > > > > > >> > > > >> and > > > > > > > > >> > > > >>>>>>>>>> limited > > > > > > > > >> > > > >>>>>>>>>>>>>> scalability. > > > > > > > > >> > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also > been > > > > > > proposed by > > > > > > > > >> > David. > > > > > > > > >> > > > >>> The > > > > > > > > >> > > > >>>>>>>>>> concern > > > > > > > > >> > > > >>>>>>>>>>>>> with > > > > > > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces > > > > > additional > > > > > > > > >> > complexity > > > > > > > > >> > > > >>> for > > > > > > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server > would > > > not > > > > > > > > >> immediately > > > > > > > > >> > add > > > > > > > > >> > > > >>> the > > > > > > > > >> > > > >>>>>>>>>> member > > > > > > > > >> > > > >>>>>>>>>>>> to > > > > > > > > >> > > > >>>>>>>>>>>>>> the group, while the old server > would. > > > This > > > > > > > > requires > > > > > > > > >> > > > >> clients > > > > > > > > >> > > > >>>> to > > > > > > > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId > > has > > > > been > > > > > > added > > > > > > > > >> to > > > > > > > > >> > the > > > > > > > > >> > > > >>>> group > > > > > > > > >> > > > >>>>>>>>>> or > > > > > > > > >> > > > >>>>>>>>>>>> not, > > > > > > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected > logs. > > > > > > > > >> > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>> Best Regards, > > > > > > > > >> > > > >>>>>>>>>>>>>> TengYao > > > > > > > > >> > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield < > > > > andrew_schofi...@live.com> > > > > > 於 > > > > > > > > >> > 2024年8月14日 > > > > > > > > >> > > > >>> 週三 > > > > > > > > >> > > > >>>>>>>>>>>> 上午12:29寫道： > > > > > > > > >> > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao, > > > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if > > there’s > > > a > > > > > > > > different > > > > > > > > >> way > > > > > > > > >> > > > >> to > > > > > > > > >> > > > >>>>> close > > > > > > > > >> > > > >>>>>>>>>>> what > > > > > > > > >> > > > >>>>>>>>>>>>>>> is quite a small window. > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial > > > heartbeat > > > > > is > > > > > > not > > > > > > > > >> > > > >>> idempotent, > > > > > > > > >> > > > >>>>> but > > > > > > > > >> > > > >>>>>>>>>>> this > > > > > > > > >> > > > >>>>>>>>>>>>>>> remains > > > > > > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just > > > differently > > > > > not > > > > > > > > >> > idempotent. > > > > > > > > >> > > > >>> If > > > > > > > > >> > > > >>>>> the > > > > > > > > >> > > > >>>>>>>>>>>> client > > > > > > > > >> > > > >>>>>>>>>>>>>>> makes its > > > > > > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and > > dies, > > > > the > > > > > > GC > > > > > > > > will > > > > > > > > >> > > > >> still > > > > > > > > >> > > > >>>> have > > > > > > > > >> > > > >>>>>>>>>>> added > > > > > > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will > > hang > > > > > > around > > > > > > > > >> until > > > > > > > > >> > the > > > > > > > > >> > > > >>>>> session > > > > > > > > >> > > > >>>>>>>>>>>>> expires. > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still > > generate > > > > the > > > > > > member > > > > > > > > >> ID > > > > > > > > >> > in > > > > > > > > >> > > > >>>>>>>>>> response to > > > > > > > > >> > > > >>>>>>>>>>>> the > > > > > > > > >> > > > >>>>>>>>>>>>>>> first > > > > > > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a > > > special > > > > > > PENDING > > > > > > > > >> > state > > > > > > > > >> > > > >>> with > > > > > > > > >> > > > >>>>> no > > > > > > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends > the > > > > next > > > > > > > > >> heartbeat, > > > > > > > > >> > > > >> thus > > > > > > > > >> > > > >>>>>>>>>>> confirming > > > > > > > > >> > > > >>>>>>>>>>>>> it > > > > > > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This > would > > > not > > > > > be > > > > > > a > > > > > > > > >> > protocol > > > > > > > > >> > > > >>>> change > > > > > > > > >> > > > >>>>>>>>>> at > > > > > > > > >> > > > >>>>>>>>>>>> all, > > > > > > > > >> > > > >>>>>>>>>>>>>>> just > > > > > > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new > > member > > > in > > > > > the > > > > > > > > lobby > > > > > > > > >> > > > >> until > > > > > > > > >> > > > >>>> it’s > > > > > > > > >> > > > >>>>>>>>>>>>> comfirmed > > > > > > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID. > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > >>>>>>>>>>>>>>> Andrew > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao > > Chi < > > > > > > > > >> > > > >>> kiting...@gmail.com> > > > > > > > > >> > > > >>>>>>>>>> wrote: > > > > > > > > >> > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping, > > > > > > > > >> > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions. > > > > > > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP > > > > > > accordingly. > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Please take a look. > > > > > > > > >> > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Best regards, > > > > > > > > >> > > > >>>>>>>>>>>>>>>> TengYao > > > > > > > > >> > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai < > chia7...@apache.org> > > 於 > > > > > > > > 2024年8月13日 > > > > > > > > >> 週二 > > > > > > > > >> > > > >>>>> 下午9:45寫道： > > > > > > > > >> > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP. > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the > > > > > > before/after > > > > > > > > >> > behavior > > > > > > > > >> > > > >> in > > > > > > > > >> > > > >>>> the > > > > > > > > >> > > > >>>>>>>>>>>>> "Proposed > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current > RPC > > > > allows > > > > > > HB > > > > > > > > >> having > > > > > > > > >> > > > >>> member > > > > > > > > >> > > > >>>>> id > > > > > > > > >> > > > >>>>>>>>>>>>>>> generated by > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member > > ID, > > > > > > server > > > > > > > > will > > > > > > > > >> > > > >>> generate > > > > > > > > >> > > > >>>>> one > > > > > > > > >> > > > >>>>>>>>>>> and > > > > > > > > >> > > > >>>>>>>>>>>>> then > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will > > enforce > > > HB > > > > > > "must" > > > > > > > > >> have > > > > > > > > >> > > > >>> member > > > > > > > > >> > > > >>>>> ID. > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the > > version > > > > > number > > > > > > > > >> > explicitly > > > > > > > > >> > > > >>> in > > > > > > > > >> > > > >>>>> the > > > > > > > > >> > > > >>>>>>>>>> KIP > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the > > old > > > > HB? > > > > > > Does > > > > > > > > it > > > > > > > > >> > > > >> always > > > > > > > > >> > > > >>>>>>>>>> generate > > > > > > > > >> > > > >>>>>>>>>>>>> member > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that > > is > > > > not > > > > > > > > >> restricted? > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Best, > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi > > > > wrote: > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone, > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a > discussion > > > > thread > > > > > > on > > > > > > > > >> > KIP-1082, > > > > > > > > >> > > > >>>> which > > > > > > > > >> > > > >>>>>>>>>>>> proposes > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for > clients > > > over > > > > > the > > > > > > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat > > > > > > > > >> > > > >>>>>>>>>>>>> RPC. > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082 > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> < > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me > know > > > what > > > > > you > > > > > > > > >> think, > > > > > > > > >> > > > >> and I > > > > > > > > >> > > > >>>>> would > > > > > > > > >> > > > >>>>>>>>>>>>>>> appreciate > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback. > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards, > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>>> > > > > > > > > >> > > > >>>>>>>>> > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>>>> > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
