Hey Luke Thank you for another great idea. Though, I agree with David's concern here. The benefits of the space savings outweigh the cost incurred for message conversion here. As an example, for a compressed workload, we will have to perform decompression -> change format -> compress in new format -> decompress to validate the batch header/data for every batch. The whole process of message conversion is very expensive in Kafka.
Having said that, we definitely can improve the current message format. 1. About deployment timeline: May I suggest that v3 message format should be kept as opt-in until we release Kafka v4.0 where it becomes default. This way, the users can choose to take the incremental hit of message conversion. 2. About changes to data layout: Since we are changing the version number, let's bundle all pending changes together. I have another change proposed that can be clubbed into this KIP, see below. 3. Note that protocol buffer encoded varlong can take upto 10 bytes (and 5 bytes for varInt). Mentioning it here so that it could be added as an FYI in the doc since we are proposing moving from int64 to varLog. The worst case of varLong uses more space. Not that it impacts the proposal, but worth keeping an eye on. 4. Since we are changing the message version, we should revisit usage of protobuf zigzag encoding for varInt. There are alternatives developed in the last few years which might be more suited to our use case such as prefixVarint. Developers of Protocol buffers recommend looking at alternatives suited for a use case [1]. More discussions here [2]. [Additional proposal] Changing the batch format proposal Problem: Currently, we perform validation of data schema (such as verifying monotonically increasing offsets etc.). To do that, we have to read the headers of each message in a batch. For a compressed batch, this means that we have to decompress the entire batch including the payload (key/value) even though we don't really require any information out of them. Solution: If we change the ordering of messages in a batch to prefix with headers of all messages first and followed by the payload of all messages, it will lead to an optimization where we have to "partially" decompress the batch (i.e. the prefix). [1] https://github.com/WebAssembly/design/issues/601#issuecomment-196045891 [2] https://news.ycombinator.com/item?id=11263378 -- Divij Vaidya On Sat, May 13, 2023 at 1:14 AM Luke Chen <show...@gmail.com> wrote: > Hi Kirk, > > Yes, the pressure in broker comes from the message format down conversion. > > Luke > > > Kirk True <k...@kirktrue.pro> 於 2023年5月13日 週六 上午1:30 寫道： > > > Hi David, > > > > For my own edification, when you refer to this change possibly putting > > "more pressure on the brokers," is that from the downconversion of the > > message format, specifically, or something else? > > > > Thanks, > > Kirk > > > > On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote: > > > Hi David, > > > > > > I know what you mean. > > > Let's hear what others' thoughts about it. :) > > > > > > Luke > > > > > > On Fri, May 12, 2023 at 4:53 PM David Jacot > <dja...@confluent.io.invalid > > > > > > wrote: > > > > > > > Thanks, Luke. > > > > > > > > > But if the producers and consumers all existed in the same > > organization, > > > > which means upgrading producers/consumers for the org's cost saving, > > should > > > > be a reasonable motivation. > > > > > > > > Yeah, that works in this case. However, Kafka is often used as a > > service > > > > (on premise or in cloud) nowadays and in this case the > > producers/consumers > > > > versions are completely out of control thus my concern. > > > > > > > > BR, > > > > David > > > > > > > > On Fri, May 12, 2023 at 10:47 AM Luke Chen <show...@gmail.com> > wrote: > > > > > > > > > Hi David, > > > > > > > > > > Yes, you're right. I've bumped the version of record batch, and > > describe > > > > > the down-conversion will happen like what we do for message format > > v1 now > > > > > when old consumers consuming records. > > > > > > > > > > > Overall, I wonder if the bandwidth saving is worth this change > > given > > > > that > > > > > it will put more pressure on the brokers. > > > > > Actually, I'm not 100% sure. So I'd also like to hear what the > > community > > > > > thought about it. > > > > > But if the producers and consumers all existed in the same > > organization, > > > > > which means upgrading producers/consumers for the org's cost > saving, > > > > should > > > > > be a reasonable motivation. > > > > > > > > > > Thanks. > > > > > Luke > > > > > > > > > > > > > > > On Fri, May 12, 2023 at 3:43 PM David Jacot > > <dja...@confluent.io.invalid > > > > > > > > > > wrote: > > > > > > > > > > > Hi Luke, > > > > > > > > > > > > Thanks for the KIP. > > > > > > > > > > > > What do we do in the case where a batch is written with > > > > > > `ignoreMessageAttributes` set to 1, which means that messages > won't > > > > have > > > > > > the `attributes`, and is consumed by a consumer which does not > > > > understand > > > > > > this new format? I suppose that we would need to introduce a new > > > > version > > > > > > for the message format (v3) and that we will have to downconvert > > > > records > > > > > > from the new format version to v2 in this case. This is not clear > > in > > > > the > > > > > > KIP. Could you elaborate a bit more on this? Overall, I wonder if > > the > > > > > > bandwidth saving is worth this change given that it will put more > > > > > pressure > > > > > > on the brokers. > > > > > > > > > > > > Best, > > > > > > David > > > > > > > > > > > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <show...@gmail.com> > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore > > unused > > > > > > > message attribute field. This KIP is to add a flag in the batch > > > > header > > > > > to > > > > > > > indicate if messages inside the batch have attribute field or > > not, to > > > > > > > reduce the message size, thus, save network traffic and storage > > size > > > > > (and > > > > > > > money, of course). > > > > > > > > > > > > > > Please check the link for more detail: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > > > > > > > > > > > > > Any feedback is welcome. > > > > > > > > > > > > > > Thank you. > > > > > > > Luke > > > > > > > > > > > > > > > > > > > > > > > > > > > >
