Hi Matthias, Thank you for the positive feedback. I am new to Kafka contributions and have just requested a Confluence account for creating the KIP. I am currently waiting for the account creation.
I noticed that KIP-512 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData>, which addressed a similar issue with headers, was proposed five years ago. Do you suggest that I edit and improve KIP-512 (pending agreement from the original author) or create a new KIP? My previous email focused on WHY we need this feature. This follow-up email will discuss **HOW** we can implement it. **Implementation Options** 1. **Option 1: Add Headers to RecordMetadata** (Original KIP-512 Approach) - **Prototype**: [ https://github.com/jychen7/kafka/pull/1](https://github.com/jychen7/kafka/pull/1) 2. **Option 2: Add a New onAcknowledgement Method with an Additional Parameter for Headers or Producer Record** - This option was brainstormed from the original thread <https://www.mail-archive.com/dev@kafka.apache.org/msg100650.html> - **Prototype**: [ https://github.com/jychen7/kafka/pull/2](https://github.com/jychen7/kafka/pull/2) I prefer Option 1 because it is backward compatible, does not introduce any new API, and does not require deprecating any existing APIs. In my opinion, headers are a form of metadata, and RecordMetadata is created on the client side without needing extra information from the broker. What are your thoughts on these options? If there are other suggestions, I am open to considering them as well. Best regards, Rich On Sat, Jul 13, 2024 at 1:50 PM Matthias J. Sax <mj...@apache.org> wrote: > Hi Rich, > > thanks for reaching out. I don't see any reason to object to this > proposal. The best way forward would be to write a KIP and collect > feedback from there. > > > -Matthias > > On 7/12/24 9:14 PM, Rich C. wrote: > > Dear Kafka Development Community, > > > > I propose enhancing the Kafka ProducerInterceptor interface, specifically > > to make Record Headers available in the onAcknowledgement method. I would > > appreciate your feedback on this proposal. If the feedback is positive, I > > will follow up with a detailed discussion on implementing this feature. > > > > *Current State* > > > > At present, the topic, partition, offset, and timestamp are available in > > the onAcknowledgement method. However, headers are not accessible. > > > > *Why This Feature Is Important* > > > > Two primary use cases highlight the importance of making headers > available > > in the onAcknowledgement method: > > > > 1. *Latency Measurement* > > > > Latency measurement is crucial for understanding the time taken for > > messages to travel from the producer to Kafka and back to the producer as > > an acknowledgment. The current setup does not allow for precise > measurement > > of the producer-side latency (a) in the following scenario: > > > > ``` > > producer send -> (a) -> Kafka -> (b) -> acknowledge > > ``` > > > > - If using CreateTime, the calculation is: `now - message.timestamp = > > (a) + (b)` > > > > - If using LogAppendTime, the calculation is: `now - > message.timestamp = > > (b)` > > > > By making headers available in onAcknowledgement, we can include a > > timestamp in the header when the message is sent. This allows us to > > calculate the producer-side latency (a) as `message.timestamp - > > CreateTimeFromHeader` when using LogAppendTime. > > > > 2. *Tracing Completeness* > > > > In distributed systems, tracing is essential for tracking the flow of > > messages and understanding system behavior. The trace ID is typically > > stored in the message headers. Having access to headers in the > > onAcknowledgement method would enable us to add spans indicating when the > > message arrives at Kafka and when it is acknowledged by the client. This > > would significantly enhance tracing completeness and accuracy. > > > > *Why Not Use Callback on Send* > > > > While it is possible to use the Callback mechanism provided by the send > > method, this approach has limitations: > > > > - *Injectability*: In some cases, such as with Debezium and Flink Kafka > > sink, the Callback is not injectable, making it impractical to use this > > method for the aforementioned purposes. > > > > I believe that adding Record Headers to the onAcknowledgement method in > > ProducerInterceptor would significantly enhance Kafka's capabilities in > > latency measurement and tracing, and also open up new opportunities for > > further improvements and innovations. > > > > I look forward to your feedback on this proposal. If the community is in > > favor, I will provide a detailed follow-up on how this feature can be > > implemented. While KIP-512 was considered, we can discuss other options > in > > the next follow-up. > > > > Thank you for your consideration. > > > > Best regards, > > Rich Chen > > >
