Perhaps we can start a wiki page where we collect these ideas as a precursor to a KIP for record format v3?
Ismael On Mon, May 15, 2023, 8:19 PM Luke Chen <show...@gmail.com> wrote: > Hi Divij and Ismael, > > Thanks for your great comments. > Yes, I know record format changes are _extremely expensive_ for the > ecosystem. > But on the other hand, it's not clear "what kind of change" is worth > changing it. > That's why I posted the KIP for discussion. > > It looks like the benefit of this KIP is still not strong enough, and we > have more further changes planned for message format v3. > I'll move this KIP into "discarded" state and add some reasons there. > Please remember to take this KIP (and Divij's proposal) into consideration > when we plan to propose a new message format. > > Thank you. > Luke > > > On Mon, May 15, 2023 at 10:55 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Luke, > > > > Thanks for the KIP. A few things: > > > > 1. Record format changes are _extremely expensive_ for the ecosystem, so > we > > need to have very strong motivation for them. There is a reason why we > have > > had so few of them and the last one was in 0.11. > > 2. It was a conscious decision to make the record header fixed size - it > > would be a lot more complicated to set some of the fields after writing > the > > actual records otherwise. If we want the record header to be variable > size, > > then we would probably want to move some fields to a "trailer". > > 3. v3 of the record format should make it cheaper to make changes in the > > future (perhaps it could support tagged fields or similar) > > 4. We'd want to fix other known issues at the same time (eg log append > time > > should always be available, there may be others) > > 5. We should consider whether we would want to introduce a user header > that > > is at the batch level vs record level for efficiency reasons > > > > Ismael > > > > On Fri, May 12, 2023 at 12:04 AM Luke Chen <show...@gmail.com> wrote: > > > > > Hi all, > > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused > > > message attribute field. This KIP is to add a flag in the batch header > to > > > indicate if messages inside the batch have attribute field or not, to > > > reduce the message size, thus, save network traffic and storage size > (and > > > money, of course). > > > > > > Please check the link for more detail: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > > > > > Any feedback is welcome. > > > > > > Thank you. > > > Luke > > > > > >
