Hi Luke, Thanks for the KIP. A few things:
1. Record format changes are _extremely expensive_ for the ecosystem, so we need to have very strong motivation for them. There is a reason why we have had so few of them and the last one was in 0.11. 2. It was a conscious decision to make the record header fixed size - it would be a lot more complicated to set some of the fields after writing the actual records otherwise. If we want the record header to be variable size, then we would probably want to move some fields to a "trailer". 3. v3 of the record format should make it cheaper to make changes in the future (perhaps it could support tagged fields or similar) 4. We'd want to fix other known issues at the same time (eg log append time should always be available, there may be others) 5. We should consider whether we would want to introduce a user header that is at the batch level vs record level for efficiency reasons Ismael On Fri, May 12, 2023 at 12:04 AM Luke Chen <show...@gmail.com> wrote: > Hi all, > > I'd like to start a discussion for the KIP-931: Flag to ignore unused > message attribute field. This KIP is to add a flag in the batch header to > indicate if messages inside the batch have attribute field or not, to > reduce the message size, thus, save network traffic and storage size (and > money, of course). > > Please check the link for more detail: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > Any feedback is welcome. > > Thank you. > Luke >
