Hi David, For my own edification, when you refer to this change possibly putting "more pressure on the brokers," is that from the downconversion of the message format, specifically, or something else?
Thanks, Kirk On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote: > Hi David, > > I know what you mean. > Let's hear what others' thoughts about it. :) > > Luke > > On Fri, May 12, 2023 at 4:53 PM David Jacot <dja...@confluent.io.invalid> > wrote: > > > Thanks, Luke. > > > > > But if the producers and consumers all existed in the same organization, > > which means upgrading producers/consumers for the org's cost saving, should > > be a reasonable motivation. > > > > Yeah, that works in this case. However, Kafka is often used as a service > > (on premise or in cloud) nowadays and in this case the producers/consumers > > versions are completely out of control thus my concern. > > > > BR, > > David > > > > On Fri, May 12, 2023 at 10:47 AM Luke Chen <show...@gmail.com> wrote: > > > > > Hi David, > > > > > > Yes, you're right. I've bumped the version of record batch, and describe > > > the down-conversion will happen like what we do for message format v1 now > > > when old consumers consuming records. > > > > > > > Overall, I wonder if the bandwidth saving is worth this change given > > that > > > it will put more pressure on the brokers. > > > Actually, I'm not 100% sure. So I'd also like to hear what the community > > > thought about it. > > > But if the producers and consumers all existed in the same organization, > > > which means upgrading producers/consumers for the org's cost saving, > > should > > > be a reasonable motivation. > > > > > > Thanks. > > > Luke > > > > > > > > > On Fri, May 12, 2023 at 3:43 PM David Jacot <dja...@confluent.io.invalid > > > > > > wrote: > > > > > > > Hi Luke, > > > > > > > > Thanks for the KIP. > > > > > > > > What do we do in the case where a batch is written with > > > > `ignoreMessageAttributes` set to 1, which means that messages won't > > have > > > > the `attributes`, and is consumed by a consumer which does not > > understand > > > > this new format? I suppose that we would need to introduce a new > > version > > > > for the message format (v3) and that we will have to downconvert > > records > > > > from the new format version to v2 in this case. This is not clear in > > the > > > > KIP. Could you elaborate a bit more on this? Overall, I wonder if the > > > > bandwidth saving is worth this change given that it will put more > > > pressure > > > > on the brokers. > > > > > > > > Best, > > > > David > > > > > > > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <show...@gmail.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused > > > > > message attribute field. This KIP is to add a flag in the batch > > header > > > to > > > > > indicate if messages inside the batch have attribute field or not, to > > > > > reduce the message size, thus, save network traffic and storage size > > > (and > > > > > money, of course). > > > > > > > > > > Please check the link for more detail: > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > > > > > > > > > Any feedback is welcome. > > > > > > > > > > Thank you. > > > > > Luke > > > > > > > > > > > > > > >
