Hi, About the logical client concept, that would apply to Connect as well.
I wonder if the logical client (Streams, Connect or any application) could set generate its own 64bit number and then each client will reuse that when building their UUID. That way each client would have a unique UUID and all UUIDs from clients from the same logical application would share the same prefix. Thanks, Mickael On Mon, May 18, 2026 at 10:50 AM Andrew Schofield <[email protected]> wrote: > > Hi Matthias, > Thanks for your message. > > It does seem useful to have some kind of "logical client" concept, but I > would say that it builds upon this KIP, rather than having to be an intrinsic > part of it. For example, a follow-on could introduce a new version of > GetTelemetrySubscriptions/PushTelemetry with an optional additional UUID > where we want to indicate a logical relationship between client connections. > There are a lot of details to thrash out I think, so a separate KIP is my > strong preference. > > Thanks, > Andrew > > On 2026/05/13 06:14:18 "Matthias J. Sax" wrote: > > Thanks for this KIP Andrew. > > > > I am following the discussion for a while already, and it seems there is > > a lot of details to consider. So I feel a little bit bad here, as I > > might make matters worse. > > > > The KIP does touch on Kafka Streams briefly, but I am wondering if we > > should make a larger change? Kafka Streams at this point, treats the > > producer/consumer/admin clients as black-boxes, and I would describe > > Kafka Streams as "logical client" as it does not maintain its own > > network connections but re-used the network connections from the > > underlying clients. > > > > To make KIP-714 work for Kafka Streams, we extended the client APIs to > > allow registering additional custom metrics (KIP-1076), to be send to > > the broker. However, these metrics are reported using the same > > `clientInstanceId` as the physical client. As a matter of fact, Kafka > > Streams uses a mix of admin client and consumer client (per thread), and > > thus reports it's metrics using multiple different `clientInstanceIds` > > for the same Kafka Streams instance. > > > > With KIP-1324 and KIP-1331, we also want to collect client side configs, > > including Kafka Streams configs, and Topology information, and face a > > similar challenge. > > > > Thus, I was thinking if we should assign Kafka Streams clients it's own > > clientInstanceId, even if it's not a physical client. If > > clientInstanceIds are generated by clients themselves anyway, a Kafka > > Streams instance can just generate its own ID (as a matter of face, we > > already have a `processId` which is also a UUID). > > > > Having such a UUID at hand, we could use it to send all these different > > things (metrics, config, topology) using the same UUID, what would make > > it much simpler to correlate broker side. Of course, such a logical > > client UUID could not go into the request header, as the underlying > > physical client has its own UUID. This might still work, as the KIP > > already says SHOULD, not MUST. > > > > To make this work for KIP-714/KIP-1076, we would need to change the > > client APIs a little bit, and allow Kafka Streams to pass it's own UUID > > into the consumer/admin client to be use to push the Streams metrics. We > > would also need to expose the Streams clientInstanceID via the > > `ClientInstanceIds` interface. > > > > For KIP-1324 and KIP-1331, we would need similar APIs. > > > > I don't think we can (atm) unify this with KIP-848/1071 member-ID > > though, as a Kafka Streams client has multiple threads, and each thread > > has its own consumer, ie, there is multiple members per instance. > > However, we actually do have plans to refactor Kafka Streams threading > > model, and would like to reduce the number of consumers/group-members > > per instance to one. When we make this change, we could also unify with > > member-id. > > > > > > Thoughts? > > > > > > -Matthias > > > > > > > > > > On 5/12/26 10:48 AM, Andrew Schofield wrote: > > > Hi Jun, > > > Thanks for the reply and digging into the details. > > > > > > JR23: Correct. The client telemetry component will use UUID-B as the > > > client instance ID. > > > > > > JR23.1: Yes, I agree. It's not ideal. When I was drawing up the tables, I > > > was thinking that this might be a possibility, but I'm less convinced > > > now. I think that I should mandate that if a client specifies > > > header.ClientInstanceId on GetTelemetrySubscriptions request, then > > > request.ClientInstanceId must either be zero or equal to > > > header.ClientInstanceId. > > > > > > JR23.2: This is perhaps the interesting one. From its original intent, it > > > should be UUID-B (the telemetry UUID), but then that contradicts the > > > change in signature to remove the timeout. Unless I make the change > > > above, in which case it will be UUID-H. > > > > > > Thanks, > > > Andrew > > > > > > On 2026/05/12 17:23:58 Jun Rao via dev wrote: > > >> Hi, Andrew, > > >> > > >> Thanks for the reply. > > >> > > >> JR23. In the new client -> old broker case, we have > > >> header.ClientInstanceId=UUID-H > > >> request.ClientInstanceId=UUID-B > > >> response.ClientInstanceId=0 > > >> > > >> On the server side, I guess the telemetry component will use UUID-B as > > >> the > > >> clientInstanceId? This has a couple of implications. > > >> JR23.1 On the server side, we have two different clientInstanceIds used > > >> in > > >> different places, UUID-H for request logging and UUID-B in telemetry. > > >> This > > >> seems confusing since we can't uniquely identify a client on the server > > >> side. > > >> JR23.2 On the client side. what uuid does clientInstanceId(Duration > > >> timeout) return? If it returns UUID-H, it will be confusing since it > > >> doesn't match the ID used for telemetry on the server. > > >> > > >> Jun > > >> > > >> > > >> > > >> On Tue, May 12, 2026 at 12:58 AM Andrew Schofield <[email protected]> > > >> wrote: > > >> > > >>> Hi Jun, > > >>> Thanks for your response. > > >>> > > >>> JR20: I have improved (I hope) the wording. The client sends > > >>> request.clientInstanceId = 0 and header.clientInstanceId = UUID-H, and > > >>> the > > >>> broker responds response.clientInstanceId=UUID-H. In this way, the > > >>> broker > > >>> will have taken the UUID-H from the header, and told the client to use > > >>> it > > >>> for client telemetry also. > > >>> > > >>> JR21: Done. Look for "henceforth". > > >>> > > >>> JR22: Summary table added. > > >>> > > >>> Thanks, > > >>> Andrew > > >>> > > >>> On 2026/05/11 19:18:24 Jun Rao via dev wrote: > > >>>> Hi, Andrew, > > >>>> > > >>>> Thanks for the reply. > > >>>> > > >>>> JR20. "If the client requests a new client instance ID on its initial > > >>>> GetTelemetrySubscriptions request and it sends a client instance ID in > > >>> the > > >>>> request header, the broker will send back that client instance ID > > >>>> rather > > >>>> than generating a new UUID. This will automatically align the UUID in > > >>>> the > > >>>> request headers and client telemetry." > > >>>> > > >>>> This seems inconsistent with what's in the table. In the table, for > > >>>> example, if the client has the following: > > >>>> GetTelemetrySubscriptions v0 > > >>>> header.ClientInstanceId = UUID-H > > >>>> request.ClientInstanceId = UUID-H > > >>>> > > >>>> or > > >>>> > > >>>> GetTelemetrySubscriptions v0 > > >>>> header.ClientInstanceId = UUID-H > > >>>> request.ClientInstanceId = UUID-R > > >>>> > > >>>> the broker returns > > >>>> response.ClientInstanceId = 0. > > >>>> > > >>>> JR21. It will be useful to document what the new client does with the > > >>>> returned response.ClientInstanceId. Note that return value may or may > > >>>> not > > >>>> be 0. > > >>>> > > >>>> JR22. It's probably clearer if we could populate the table with 4 > > >>>> combinations: old/new clients with old/new brokers. > > >>>> > > >>>> Jun > > >>>> > > >>>> > > >>>> > > >>>> On Fri, May 8, 2026 at 2:49 AM Andrew Schofield <[email protected]> > > >>>> wrote: > > >>>> > > >>>>> Hi Jun and Chia-Ping, > > >>>>> I've overhauled part of the KIP to do with alignment of the request > > >>> header > > >>>>> client instance ID, client telemetry client instance ID and group > > >>> protocol > > >>>>> member IDs. The alignment is by convention, not mandate (SHOULD not > > >>> MUST). > > >>>>> > > >>>>> It would be possible to go around the existing RPCs such as > > >>>>> ConsumerGroupHeartbeat and GetTelemetrySubscriptions, and remove the > > >>> fields > > >>>>> containing the existing identifiers which are intended to be aligned. > > >>> Doing > > >>>>> so would be a bad idea though, because we would then have RPC versions > > >>>>> which essentially depend upon the presence of a tagged field in the > > >>> request > > >>>>> header. This is a protocol-compatibility nightmare. > > >>>>> > > >>>>> I have removed the new versions of GetTelemetrySubscriptions and > > >>>>> PushTelemetry. I have also explained the behavior of > > >>>>> GetTelemetrySubscriptions in the presence and absence of a client > > >>> instance > > >>>>> ID in the request header. > > >>>>> > > >>>>> Let me know what you think. > > >>>>> > > >>>>> Thanks, > > >>>>> Andrew > > >>>>> > > >>>>> On 2026/05/07 15:09:31 Andrew Schofield wrote: > > >>>>>> Hi Jun and Chia-Ping, > > >>>>>> I've been thinking and discussing the changes to the KIP-714 RPCs. > > >>> There > > >>>>> are too many combinations for my liking at the moment. I want to take > > >>>>> another pass at this area and will make an update in a few days. > > >>>>>> > > >>>>>> I intend to start a new vote once we have consensus because the spec > > >>> has > > >>>>> changed somewhat since the earliest votes. > > >>>>>> > > >>>>>> Thanks, > > >>>>>> Andrew > > >>>>>> > > >>>>>> On 2026/05/06 17:28:27 Chia-Ping Tsai wrote: > > >>>>>>> hi Andrew > > >>>>>>> > > >>>>>>> chia_0: If the consensus is to remove the "duplicate" field from > > >>> the > > >>>>> RPC payloads, the tagged field in the header will essentially become a > > >>>>> required field. This means the broker needs to handle the edge case > > >>> where > > >>>>> both the header and the request body have no ClientInstanceId, right? > > >>> If > > >>>>> so, would you mind clarifying the expected broker behavior in the KIP? > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> Chia-Ping > > >>>>>>> > > >>>>>>> On 2026/04/03 16:17:37 Andrew Schofield wrote: > > >>>>>>>> Hi, > > >>>>>>>> I would like to start the discussion on KIP-1313. This adds a > > >>> unique > > >>>>> client instance ID to the request header of all Kafka protocol > > >>> requests to > > >>>>> give a unique identifier which can be used to correlate the requests > > >>> from > > >>>>> each client for the purposes of problem determination. > > >>>>>>>> > > >>>>>>>> > > >>>>> > > >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!uqWf0-b_X82WmpmCYImD2W2rht_s_q5vHcqB9ToMV4IaeQbZF42eMJyS5XC5b5qE_qJJUj3KTCXcqEvYbwYS$ > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> Andrew > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > >
