Hi Divij and Ismael, Thanks for your great comments. Yes, I know record format changes are _extremely expensive_ for the ecosystem. But on the other hand, it's not clear "what kind of change" is worth changing it. That's why I posted the KIP for discussion.
It looks like the benefit of this KIP is still not strong enough, and we have more further changes planned for message format v3. I'll move this KIP into "discarded" state and add some reasons there. Please remember to take this KIP (and Divij's proposal) into consideration when we plan to propose a new message format. Thank you. Luke On Mon, May 15, 2023 at 10:55 PM Ismael Juma <ism...@juma.me.uk> wrote: > Hi Luke, > > Thanks for the KIP. A few things: > > 1. Record format changes are _extremely expensive_ for the ecosystem, so we > need to have very strong motivation for them. There is a reason why we have > had so few of them and the last one was in 0.11. > 2. It was a conscious decision to make the record header fixed size - it > would be a lot more complicated to set some of the fields after writing the > actual records otherwise. If we want the record header to be variable size, > then we would probably want to move some fields to a "trailer". > 3. v3 of the record format should make it cheaper to make changes in the > future (perhaps it could support tagged fields or similar) > 4. We'd want to fix other known issues at the same time (eg log append time > should always be available, there may be others) > 5. We should consider whether we would want to introduce a user header that > is at the batch level vs record level for efficiency reasons > > Ismael > > On Fri, May 12, 2023 at 12:04 AM Luke Chen <show...@gmail.com> wrote: > > > Hi all, > > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused > > message attribute field. This KIP is to add a flag in the batch header to > > indicate if messages inside the batch have attribute field or not, to > > reduce the message size, thus, save network traffic and storage size (and > > money, of course). > > > > Please check the link for more detail: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > > > Any feedback is welcome. > > > > Thank you. > > Luke > > >
