Proposal to Include Record Headers in ProducerInterceptor onAcknowledgement Method

Dear Kafka Development Community, I propose enhancing the Kafka ProducerInterceptor interface, specifically to make Record Headers available in the onAcknowledgement method. I would appreciate your feedback on this proposal. If the feedback is positive, I will follow up with a detailed discussion

Re: Proposal to Include Record Headers in ProducerInterceptor onAcknowledgement Method

d would be to write a KIP and collect > feedback from there. > > > -Matthias > > On 7/12/24 9:14 PM, Rich C. wrote: > > Dear Kafka Development Community, > > > > I propose enhancing the Kafka ProducerInterceptor interface, specifically > > to make Record He

Re: Proposal to Include Record Headers in ProducerInterceptor onAcknowledgement Method

t the motivation must be clearly explained and the > argument of "no needed" but be overcome in a convincing way, to get the > KIP accepted. The API design question seems to be of second nature, and > I am sure an agreement can be found, if people agree that a change is > needed

Getting Started: requesting permissions to contribute to Apache Kafka

Hi team, Following Getting Started #4 , I would like to request permission to contribute to Apache Kafka. I guess the edit permission for wiki (KIP). Jira: jychen7 wiki: juchen7 (note: wiki id is different from Jira id

[DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hi Everyone, I hope this email finds you well. I would like to start a discussion on KIP-512. The initial version of KIP-512 was created in 2019, and I have resurrected it in 2024 with more details about the motivation behind it. You can view the current version of the KIP here: KIP-512: Make Re

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

ere anyway). > > > -Matthias > > On 7/23/24 6:36 AM, Kevin Lam wrote: > > Hi, > > > > Thanks for starting the discussion. Latency Measurement and Tracing > > Completeness are both good reasons to support this feature, and would be > > interested to see thi

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

ducerInterceptor.onAcknowledgement > method, as opposed to the producer Callback.onCompletion method. > > Thanks, > Andrew > > > On 26 Jul 2024, at 04:54, Rich C. wrote: > > > > Hi Kevin, > > > > Thanks for your support. > > > > Hi Matthias, >

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

rs think about whether it’s a good > idea to extend RecordMetadata with the headers. > > Thanks, > Andrew > > > On 29 Jul 2024, at 05:36, Rich C. wrote: > > > > Hi all, > > > > Thank you for the positive feedback. I added proposal cha

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

> share Andrew's feeling because Headers are indeed part of the > ProducerRecord. But then headers will in practice simply contain info > related to the record, so it seems sensible to expect to find headers in > the RecordMetadata as you suggest, ok with me. > > Thanks! >

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

ng latency. The user simply > creates > > the record with a timestamp (known existing concept), and we make that > > value accessible in the RecordMetadata (where it exists already at some > > point, but it's mutated). Thoughts? > > > > LM2. Regardless of the poin

Inquiry on Compact Topic Behavior in Kafka 3.7.0

Hi, dev community, We are currently working with Kafka 3.7.0 and have configured a topic with the following settings: log.cleanup.policy=compact segment.ms=12 (2 minutes) log.roll.hours (default 7 days) This topic is used for testing purposes and has a low message volume. Specifically, it co

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

xt few days, I will start a voting thread. Regards, Rich On Wed, Jul 31, 2024 at 10:14 PM Rich C. wrote: > Hi Matthias, > > > (100) To me, Headers store "application metadata" of a record, but they > are not "Kafka native" record metadata > > fair enough

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

ack has the same problem. Only the new method > should have a default implementation. > > > Thanks, > Andrew > > > > On 7 Aug 2024, at 18:16, Rich C. wrote: > > > > Hi all, > > > > thanks for all the feedback. As most prefer the new interface appro

[VOTE] KIP-512: make Record Headers available in onAcknowledgement and onComplete

Hi all, After incorporating the feedback received , I have updated KIP-512 to reflect the majority preference for the new interface approach. The KIP is now in its final form and ready for a vote. Regards, Rich

Re: [VOTE] KIP-512: make Record Headers available in onAcknowledgement and onComplete

Sorry, forgot the link to KIP-512 https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3A+make+Record+Headers+available+in+onAcknowledgement+and+onComplete Regards, Rich On Sat, Aug 10, 2024 at 1:28 PM Rich C. wrote: > Hi all, > > After incorporating the feedback received

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

users do want to modify `Headers`, they can get inconsistent results > as `Headers` can be either readonly of modifiable. > > Best, > Chia-Ping > > On 2024/07/23 03:13:59 "Rich C." wrote: > > Hi Everyone, > > > > I hope this email finds you well. > >

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

h > > Headers is a sub class of Iterable already, so changing the type > from Headers to Iterable does not make code complicated :) > > The point is “should we pass a mutable interface to users in those > callback function?” > > IMHO, the answer is NO. > > Best, > Chia-

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hi Chia-Ping, To address your concern, how about we clone the headers as RecordHeaders and then use setReadOnly? This approach will pass immitable interface to callback and won’t affect the developer experience. Regards, Rich On Thu, Aug 22, 2024 at 23:29 Rich C. wrote: > Hi Chia-Ping, &g

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Rich, > > Thanks for sharing the case. Maybe we can add public helpers to Header to > offer “last header”? The helper can cast the Iterable to List (if ok) to > get last header effectively > > Thanks, > Chia-Ping > > > > > Rich C. 於 2024年8月23日 中午12:00 寫道： > > &

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

; At any rate, that is not related to this KIP. > > thanks for having discussion with me. I will +1 to this KIP and file a Jira > to discuss the interface changes of `Headers` > > Best, > Chia-Ping > > > > > Rich C. 於 2024年8月24日 週六 上午9:30寫道： > > > Hi Chia-

Re: [VOTE] KIP-512: make Record Headers available in onAcknowledgement and onComplete

ing) > > Thanks Rich! > > > On Fri, Aug 23, 2024, 10:32 p.m. Chia-Ping Tsai > wrote: > > > +1 (binding) > > > > On 2024/08/10 17:28:45 "Rich C." wrote: > > > Hi all, > > > > > > After incorporating the feedback received > >

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

nother way is to create a new functional interface "SendCallback", > and > > > add new `Producer.send(ProducerRecord, SendCallback)` > > > > > > > > That means both `Callback` and `Producer.send(ProducerRecord, > > Callback)` > > >