Good idea! I've created a wiki for the ideas for message format v.3, and added the link in this KIP. https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
Thanks. Luke On Tue, May 16, 2023 at 4:30 PM Ismael Juma <ism...@juma.me.uk> wrote: > Perhaps we can start a wiki page where we collect these ideas as a > precursor to a KIP for record format v3? > > Ismael > > On Mon, May 15, 2023, 8:19 PM Luke Chen <show...@gmail.com> wrote: > > > Hi Divij and Ismael, > > > > Thanks for your great comments. > > Yes, I know record format changes are _extremely expensive_ for the > > ecosystem. > > But on the other hand, it's not clear "what kind of change" is worth > > changing it. > > That's why I posted the KIP for discussion. > > > > It looks like the benefit of this KIP is still not strong enough, and we > > have more further changes planned for message format v3. > > I'll move this KIP into "discarded" state and add some reasons there. > > Please remember to take this KIP (and Divij's proposal) into > consideration > > when we plan to propose a new message format. > > > > Thank you. > > Luke > > > > > > On Mon, May 15, 2023 at 10:55 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi Luke, > > > > > > Thanks for the KIP. A few things: > > > > > > 1. Record format changes are _extremely expensive_ for the ecosystem, > so > > we > > > need to have very strong motivation for them. There is a reason why we > > have > > > had so few of them and the last one was in 0.11. > > > 2. It was a conscious decision to make the record header fixed size - > it > > > would be a lot more complicated to set some of the fields after writing > > the > > > actual records otherwise. If we want the record header to be variable > > size, > > > then we would probably want to move some fields to a "trailer". > > > 3. v3 of the record format should make it cheaper to make changes in > the > > > future (perhaps it could support tagged fields or similar) > > > 4. We'd want to fix other known issues at the same time (eg log append > > time > > > should always be available, there may be others) > > > 5. We should consider whether we would want to introduce a user header > > that > > > is at the batch level vs record level for efficiency reasons > > > > > > Ismael > > > > > > On Fri, May 12, 2023 at 12:04 AM Luke Chen <show...@gmail.com> wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused > > > > message attribute field. This KIP is to add a flag in the batch > header > > to > > > > indicate if messages inside the batch have attribute field or not, to > > > > reduce the message size, thus, save network traffic and storage size > > (and > > > > money, of course). > > > > > > > > Please check the link for more detail: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > > > > > > > Any feedback is welcome. > > > > > > > > Thank you. > > > > Luke > > > > > > > > > >
