Bumping up on this thread. In a forked Kafka repo, if we want to introduce a new tagged field. Which approach should be taken? Is it fine to relax the constraint in StructSpec <https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76> to start with a very large tag-id like 10000 for custom tagged fields to not clash with future tagged fields being introduced upstream? TIA.
Thanks, Kamal On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > In KIP-482 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields#KIP482:TheKafkaProtocolshouldSupportOptionalTaggedFields-FlexibleVersions>, > we have introduced support for optional tagged fields when the > API-message supports "flexibleVersions" feature. One of the motivations of > the KIP is to support attaching extra information to a message without > requiring a version bump for everyone who has to read that message. > > In StructSpec > <https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76>, > there is a constraint to start the tagId of the tagged field from 0 and > should be contiguous. Once the user introduces a custom taggedField that > starts from 0, there is a risk of protocol divergence between the server > and client when the upstream/Apache introduces a new taggedField in future > that starts from 0. > > Once a tag-id gets introduced, it cannot be used for anything else. We > want to propagate a custom trace-id in the API-message between > server/client and want to use a large tagId like 10001 (2 bytes varint) to > avoid conflict with the future API message specification. > > We would like to understand the reason behind the tagId constraint to > start from 0 and contiguous. One use-case of having this constraint ensures > that the tag-id starts from 0 and won't be random. If it is random, then > the user won't be sure in defining the unique tagged-ids for their > user-defined fields. > > I’d appreciate it if you could take a look and share your thoughts or > feedback. > > Thanks, > Kamal > > > >
