Good idea!
I've created a wiki for the ideas for message format v.3, and added the
link in this KIP.
https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
Thanks.
Luke
On Tue, May 16, 2023 at 4:30 PM Ismael Juma wrote:
> Perhaps we can start a wiki page where we c
Perhaps we can start a wiki page where we collect these ideas as a
precursor to a KIP for record format v3?
Ismael
On Mon, May 15, 2023, 8:19 PM Luke Chen wrote:
> Hi Divij and Ismael,
>
> Thanks for your great comments.
> Yes, I know record format changes are _extremely expensive_ for the
> ec
Hi Divij and Ismael,
Thanks for your great comments.
Yes, I know record format changes are _extremely expensive_ for the
ecosystem.
But on the other hand, it's not clear "what kind of change" is worth
changing it.
That's why I posted the KIP for discussion.
It looks like the benefit of this KIP i
Hi Luke,
Thanks for the KIP. A few things:
1. Record format changes are _extremely expensive_ for the ecosystem, so we
need to have very strong motivation for them. There is a reason why we have
had so few of them and the last one was in 0.11.
2. It was a conscious decision to make the record hea
Hey Luke
Thank you for another great idea. Though, I agree with David's concern
here. The benefits of the space savings outweigh the cost incurred for
message conversion here. As an example, for a compressed workload, we will
have to perform decompression -> change format -> compress in new format
Hi Kirk,
Yes, the pressure in broker comes from the message format down conversion.
Luke
Kirk True 於 2023年5月13日 週六 上午1:30 寫道：
> Hi David,
>
> For my own edification, when you refer to this change possibly putting
> "more pressure on the brokers," is that from the downconversion of the
> messa
Hi David,
For my own edification, when you refer to this change possibly putting "more
pressure on the brokers," is that from the downconversion of the message
format, specifically, or something else?
Thanks,
Kirk
On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote:
> Hi David,
>
> I know what
Hi David,
I know what you mean.
Let's hear what others' thoughts about it. :)
Luke
On Fri, May 12, 2023 at 4:53 PM David Jacot
wrote:
> Thanks, Luke.
>
> > But if the producers and consumers all existed in the same organization,
> which means upgrading producers/consumers for the org's cost sa
Thanks, Luke.
> But if the producers and consumers all existed in the same organization,
which means upgrading producers/consumers for the org's cost saving, should
be a reasonable motivation.
Yeah, that works in this case. However, Kafka is often used as a service
(on premise or in cloud) nowada
Hi David,
Yes, you're right. I've bumped the version of record batch, and describe
the down-conversion will happen like what we do for message format v1 now
when old consumers consuming records.
> Overall, I wonder if the bandwidth saving is worth this change given that
it will put more pressure
Hi Luke,
Thanks for the KIP.
What do we do in the case where a batch is written with
`ignoreMessageAttributes` set to 1, which means that messages won't have
the `attributes`, and is consumed by a consumer which does not understand
this new format? I suppose that we would need to introduce a new
Hi all,
I'd like to start a discussion for the KIP-931: Flag to ignore unused
message attribute field. This KIP is to add a flag in the batch header to
indicate if messages inside the batch have attribute field or not, to
reduce the message size, thus, save network traffic and storage size (and
mo
12 matches
Mail list logo
