Hi Mickael, I agree that the logical client concept would apply to Connect as well as Streams. My thinking at the moment is to decouple that from this KIP, and then have a separate KIP about logical clients, which I'm happy to lead.
Thanks, Andrew On 2026/05/18 09:52:33 Mickael Maison wrote: > Hi, > > About the logical client concept, that would apply to Connect as well. > > I wonder if the logical client (Streams, Connect or any application) > could set generate its own 64bit number and then each client will > reuse that when building their UUID. That way each client would have a > unique UUID and all UUIDs from clients from the same logical > application would share the same prefix. > > Thanks, > Mickael > > On Mon, May 18, 2026 at 10:50 AM Andrew Schofield <[email protected]> > wrote: > > > > Hi Matthias, > > Thanks for your message. > > > > It does seem useful to have some kind of "logical client" concept, but I > > would say that it builds upon this KIP, rather than having to be an > > intrinsic part of it. For example, a follow-on could introduce a new > > version of GetTelemetrySubscriptions/PushTelemetry with an optional > > additional UUID where we want to indicate a logical relationship between > > client connections. There are a lot of details to thrash out I think, so a > > separate KIP is my strong preference. > > > > Thanks, > > Andrew > > > > On 2026/05/13 06:14:18 "Matthias J. Sax" wrote: > > > Thanks for this KIP Andrew. > > > > > > I am following the discussion for a while already, and it seems there is > > > a lot of details to consider. So I feel a little bit bad here, as I > > > might make matters worse. > > > > > > The KIP does touch on Kafka Streams briefly, but I am wondering if we > > > should make a larger change? Kafka Streams at this point, treats the > > > producer/consumer/admin clients as black-boxes, and I would describe > > > Kafka Streams as "logical client" as it does not maintain its own > > > network connections but re-used the network connections from the > > > underlying clients. > > > > > > To make KIP-714 work for Kafka Streams, we extended the client APIs to > > > allow registering additional custom metrics (KIP-1076), to be send to > > > the broker. However, these metrics are reported using the same > > > `clientInstanceId` as the physical client. As a matter of fact, Kafka > > > Streams uses a mix of admin client and consumer client (per thread), and > > > thus reports it's metrics using multiple different `clientInstanceIds` > > > for the same Kafka Streams instance. > > > > > > With KIP-1324 and KIP-1331, we also want to collect client side configs, > > > including Kafka Streams configs, and Topology information, and face a > > > similar challenge. > > > > > > Thus, I was thinking if we should assign Kafka Streams clients it's own > > > clientInstanceId, even if it's not a physical client. If > > > clientInstanceIds are generated by clients themselves anyway, a Kafka > > > Streams instance can just generate its own ID (as a matter of face, we > > > already have a `processId` which is also a UUID). > > > > > > Having such a UUID at hand, we could use it to send all these different > > > things (metrics, config, topology) using the same UUID, what would make > > > it much simpler to correlate broker side. Of course, such a logical > > > client UUID could not go into the request header, as the underlying > > > physical client has its own UUID. This might still work, as the KIP > > > already says SHOULD, not MUST. > > > > > > To make this work for KIP-714/KIP-1076, we would need to change the > > > client APIs a little bit, and allow Kafka Streams to pass it's own UUID > > > into the consumer/admin client to be use to push the Streams metrics. We > > > would also need to expose the Streams clientInstanceID via the > > > `ClientInstanceIds` interface. > > > > > > For KIP-1324 and KIP-1331, we would need similar APIs. > > > > > > I don't think we can (atm) unify this with KIP-848/1071 member-ID > > > though, as a Kafka Streams client has multiple threads, and each thread > > > has its own consumer, ie, there is multiple members per instance. > > > However, we actually do have plans to refactor Kafka Streams threading > > > model, and would like to reduce the number of consumers/group-members > > > per instance to one. When we make this change, we could also unify with > > > member-id. > > > > > > > > > Thoughts? > > > > > > > > > -Matthias > > > > > > > > > > > > > > > On 5/12/26 10:48 AM, Andrew Schofield wrote: > > > > Hi Jun, > > > > Thanks for the reply and digging into the details. > > > > > > > > JR23: Correct. The client telemetry component will use UUID-B as the > > > > client instance ID. > > > > > > > > JR23.1: Yes, I agree. It's not ideal. When I was drawing up the tables, > > > > I was thinking that this might be a possibility, but I'm less convinced > > > > now. I think that I should mandate that if a client specifies > > > > header.ClientInstanceId on GetTelemetrySubscriptions request, then > > > > request.ClientInstanceId must either be zero or equal to > > > > header.ClientInstanceId. > > > > > > > > JR23.2: This is perhaps the interesting one. From its original intent, > > > > it should be UUID-B (the telemetry UUID), but then that contradicts the > > > > change in signature to remove the timeout. Unless I make the change > > > > above, in which case it will be UUID-H. > > > > > > > > Thanks, > > > > Andrew > > > > > > > > On 2026/05/12 17:23:58 Jun Rao via dev wrote: > > > >> Hi, Andrew, > > > >> > > > >> Thanks for the reply. > > > >> > > > >> JR23. In the new client -> old broker case, we have > > > >> header.ClientInstanceId=UUID-H > > > >> request.ClientInstanceId=UUID-B > > > >> response.ClientInstanceId=0 > > > >> > > > >> On the server side, I guess the telemetry component will use UUID-B as > > > >> the > > > >> clientInstanceId? This has a couple of implications. > > > >> JR23.1 On the server side, we have two different clientInstanceIds > > > >> used in > > > >> different places, UUID-H for request logging and UUID-B in telemetry. > > > >> This > > > >> seems confusing since we can't uniquely identify a client on the server > > > >> side. > > > >> JR23.2 On the client side. what uuid does clientInstanceId(Duration > > > >> timeout) return? If it returns UUID-H, it will be confusing since it > > > >> doesn't match the ID used for telemetry on the server. > > > >> > > > >> Jun > > > >> > > > >> > > > >> > > > >> On Tue, May 12, 2026 at 12:58 AM Andrew Schofield > > > >> <[email protected]> > > > >> wrote: > > > >> > > > >>> Hi Jun, > > > >>> Thanks for your response. > > > >>> > > > >>> JR20: I have improved (I hope) the wording. The client sends > > > >>> request.clientInstanceId = 0 and header.clientInstanceId = UUID-H, > > > >>> and the > > > >>> broker responds response.clientInstanceId=UUID-H. In this way, the > > > >>> broker > > > >>> will have taken the UUID-H from the header, and told the client to > > > >>> use it > > > >>> for client telemetry also. > > > >>> > > > >>> JR21: Done. Look for "henceforth". > > > >>> > > > >>> JR22: Summary table added. > > > >>> > > > >>> Thanks, > > > >>> Andrew > > > >>> > > > >>> On 2026/05/11 19:18:24 Jun Rao via dev wrote: > > > >>>> Hi, Andrew, > > > >>>> > > > >>>> Thanks for the reply. > > > >>>> > > > >>>> JR20. "If the client requests a new client instance ID on its initial > > > >>>> GetTelemetrySubscriptions request and it sends a client instance ID > > > >>>> in > > > >>> the > > > >>>> request header, the broker will send back that client instance ID > > > >>>> rather > > > >>>> than generating a new UUID. This will automatically align the UUID > > > >>>> in the > > > >>>> request headers and client telemetry." > > > >>>> > > > >>>> This seems inconsistent with what's in the table. In the table, for > > > >>>> example, if the client has the following: > > > >>>> GetTelemetrySubscriptions v0 > > > >>>> header.ClientInstanceId = UUID-H > > > >>>> request.ClientInstanceId = UUID-H > > > >>>> > > > >>>> or > > > >>>> > > > >>>> GetTelemetrySubscriptions v0 > > > >>>> header.ClientInstanceId = UUID-H > > > >>>> request.ClientInstanceId = UUID-R > > > >>>> > > > >>>> the broker returns > > > >>>> response.ClientInstanceId = 0. > > > >>>> > > > >>>> JR21. It will be useful to document what the new client does with the > > > >>>> returned response.ClientInstanceId. Note that return value may or > > > >>>> may not > > > >>>> be 0. > > > >>>> > > > >>>> JR22. It's probably clearer if we could populate the table with 4 > > > >>>> combinations: old/new clients with old/new brokers. > > > >>>> > > > >>>> Jun > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Fri, May 8, 2026 at 2:49 AM Andrew Schofield > > > >>>> <[email protected]> > > > >>>> wrote: > > > >>>> > > > >>>>> Hi Jun and Chia-Ping, > > > >>>>> I've overhauled part of the KIP to do with alignment of the request > > > >>> header > > > >>>>> client instance ID, client telemetry client instance ID and group > > > >>> protocol > > > >>>>> member IDs. The alignment is by convention, not mandate (SHOULD not > > > >>> MUST). > > > >>>>> > > > >>>>> It would be possible to go around the existing RPCs such as > > > >>>>> ConsumerGroupHeartbeat and GetTelemetrySubscriptions, and remove the > > > >>> fields > > > >>>>> containing the existing identifiers which are intended to be > > > >>>>> aligned. > > > >>> Doing > > > >>>>> so would be a bad idea though, because we would then have RPC > > > >>>>> versions > > > >>>>> which essentially depend upon the presence of a tagged field in the > > > >>> request > > > >>>>> header. This is a protocol-compatibility nightmare. > > > >>>>> > > > >>>>> I have removed the new versions of GetTelemetrySubscriptions and > > > >>>>> PushTelemetry. I have also explained the behavior of > > > >>>>> GetTelemetrySubscriptions in the presence and absence of a client > > > >>> instance > > > >>>>> ID in the request header. > > > >>>>> > > > >>>>> Let me know what you think. > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Andrew > > > >>>>> > > > >>>>> On 2026/05/07 15:09:31 Andrew Schofield wrote: > > > >>>>>> Hi Jun and Chia-Ping, > > > >>>>>> I've been thinking and discussing the changes to the KIP-714 RPCs. > > > >>> There > > > >>>>> are too many combinations for my liking at the moment. I want to > > > >>>>> take > > > >>>>> another pass at this area and will make an update in a few days. > > > >>>>>> > > > >>>>>> I intend to start a new vote once we have consensus because the > > > >>>>>> spec > > > >>> has > > > >>>>> changed somewhat since the earliest votes. > > > >>>>>> > > > >>>>>> Thanks, > > > >>>>>> Andrew > > > >>>>>> > > > >>>>>> On 2026/05/06 17:28:27 Chia-Ping Tsai wrote: > > > >>>>>>> hi Andrew > > > >>>>>>> > > > >>>>>>> chia_0: If the consensus is to remove the "duplicate" field from > > > >>> the > > > >>>>> RPC payloads, the tagged field in the header will essentially > > > >>>>> become a > > > >>>>> required field. This means the broker needs to handle the edge case > > > >>> where > > > >>>>> both the header and the request body have no ClientInstanceId, > > > >>>>> right? > > > >>> If > > > >>>>> so, would you mind clarifying the expected broker behavior in the > > > >>>>> KIP? > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Chia-Ping > > > >>>>>>> > > > >>>>>>> On 2026/04/03 16:17:37 Andrew Schofield wrote: > > > >>>>>>>> Hi, > > > >>>>>>>> I would like to start the discussion on KIP-1313. This adds a > > > >>> unique > > > >>>>> client instance ID to the request header of all Kafka protocol > > > >>> requests to > > > >>>>> give a unique identifier which can be used to correlate the requests > > > >>> from > > > >>>>> each client for the purposes of problem determination. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>> > > > >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!uqWf0-b_X82WmpmCYImD2W2rht_s_q5vHcqB9ToMV4IaeQbZF42eMJyS5XC5b5qE_qJJUj3KTCXcqEvYbwYS$ > > > >>>>>>>> > > > >>>>>>>> Thanks, > > > >>>>>>>> Andrew > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > >
