Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Luke, sorry for late response. > IMO, the retry should be the logic inside the > "large.message.payload.store.class" > implementation. If we really want it, I think we need to make it clear in > which circumstance we will retry. For example, if it's an unknown exception > thrown from S3 API, wh

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Omnia, Thanks for the explanation and update. It's better now. Questions: > 2. It's not clear how the "retry count" comes into play in the KIP. It > needs more explanation. My initial thinking is the retry configs are a must for all blob stores, so we can provide them, and validate them for fr

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Kirk, thanks for having time to look into this, > KT1: In PayloadReferenceValue, Is "payloadStoreClass" intended to be a > fully-qualified Java class name, or something else? I'm considering the case > where a Producer and/or Consumer are using a non-Java-based client. This would be any stri

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Omnia, A very interesting KIP! Thanks for the write up and discussion thus far! A few questions: KT1: In PayloadReferenceValue, Is "payloadStoreClass" intended to be a fully-qualified Java class name, or something else? I'm considering the case where a Producer and/or Consumer are using a n

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Luke, >> 3. What does "LargeMessageFormatter" do in the process? >> I thought all we want to do is to replace the "large value data" into a >> path, and consumers will read the path via blob store class. >> All these should be done in serializer/deserializer, so why do we need the >> formatter?

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Luke, thanks for having the time to look into the KIP > 2. It's not clear how the "retry count" comes into play in the KIP. It > needs more explanation. My initial thinking is the retry configs are a must for all blob stores, so we can provide them, and validate them for free for all blob store

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Omnia, Thanks for proposing this feature that many users expected. Some comments: 1. It's quite interesting to see the idea of chained serializer/deserializer used here. I like it. 2. It's not clear how the "retry count" comes into play in the KIP. It needs more explanation. 3. What does "La

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Pump this thread up > On 11 Apr 2025, at 16:32, Omnia Ibrahim wrote: > > Thanks PoAn for having a look into the KIP >> PY_1: Does LargeMessageDeserializer need >> large.message.blob.store.blob.id_generator >> configuration? The blob id is encapsulated in Kafka event by >> LargeMessageSerialize

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Thanks PoAn for having a look into the KIP > PY_1: Does LargeMessageDeserializer need > large.message.blob.store.blob.id_generator > configuration? The blob id is encapsulated in Kafka event by > LargeMessageSerializer. > The LargeMessageDeserializer can get the id from the event. You are right,

Re: [DISCUSS] KIP-1159: Large message reference based Serializer

Hi Omnia, Thanks for the KIP. It’s good to integrate blob store for large objects. PY_1: Does LargeMessageDeserializer need large.message.blob.store.blob.id_generator configuration? The blob id is encapsulated in Kafka event by LargeMessageSerializer. The LargeMessageDeserializer can get the id

[DISCUSS] KIP-1159: Large message reference based Serializer

Hi there I would like to start discussions on https://cwiki.apache.org/confluence/display/KAFKA/KIP-1159%3A+Large+message+reference+based+Serializer Thanks Omnia