Hi Matthias, Thank you for your response. I have left a comment in JIRA to the original author about taking over KIP-512. In the meantime, I am still waiting for my Confluence account to be created to edit the KIP.
Overall, it seems that the motivation must be clearly explained and the > argument of "no needed" must be overcome in a convincing way, to get the > KIP accepted I agree completely. That’s why my initial email focused on explaining why this feature is necessary. But it seems we are already starting a discussion which should actually > take place on the KIP discuss thread Once I have edited KIP-512, I will start a new KIP discussion thread, and we can continue our discussion there. Regards, Rich On Sun, Jul 14, 2024 at 1:05 PM Matthias J. Sax <mj...@apache.org> wrote: > Hey, > > Resurrecting KIP-512 might be a good idea. Just follow up on the Jira > ticket and/or old DISCUSS thread, and ask to take over -- if the > original author does not reply in a few days, just take it on; given > that the KIP is many years old, I would be surprised if anybody objects > that you resurrect it and take over. > > > > However, I would personally not add Headers to RecordMetadata, but > follow the idea to pass Headers as additional parameter (which was you > original proposal, and was also mentioned as alternative on the KIP-512 > discuss thread). > > As pointed out on the KIP-512 discussion, RecordMetadata is "Kafka > metadata" about the records (send back by the broker) like the offset. > Headers seems to be of a different quality because it's > "user/application metadata"), and I believe it would become a convoluted > API to add them to RecordMetadata where Headers IMHO do not belong to. > > But it seems we are already starting a discussion which should actually > take place on the KIP discuss thread :) -- And it's of course just my > personal opinion. In the end, the question will be what the majority of > people believe is better. > > > > The other argument on KIP-512 was, that no change is necessary at all > because Headers exist already client side. While I see the point, I > don't really agree to it... In the end, while technically correct, it's > quite complicated and clumsy to keep a copy of the Headers in the > application code, and link them back and pass into an interceptor or > callback to be used there. So letting the producer pass them into the > interceptor and/or callback, seems to be a good improvement for user > experience and ease of use. > > > > Overall, it seem that the motivation must be clearly explained and the > argument of "no needed" but be overcome in a convincing way, to get the > KIP accepted. The API design question seems to be of second nature, and > I am sure an agreement can be found, if people agree that a change is > needed in the first place. > > > > -Matthias > > > On 7/13/24 5:15 PM, Rich C. wrote: > > Hi Matthias, > > > > Thank you for the positive feedback. I am new to Kafka contributions and > > have just requested a Confluence account for creating the KIP. I am > > currently waiting for the account creation. > > > > I noticed that KIP-512 > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData > >, > > which addressed a similar issue with headers, was proposed five years > ago. > > Do you suggest that I edit and improve KIP-512 (pending agreement from > the > > original author) or create a new KIP? > > > > My previous email focused on WHY we need this feature. This follow-up > email > > will discuss **HOW** we can implement it. > > > > **Implementation Options** > > > > 1. **Option 1: Add Headers to RecordMetadata** (Original KIP-512 > Approach) > > - **Prototype**: [ > > > https://github.com/jychen7/kafka/pull/1](https://github.com/jychen7/kafka/pull/1) > > > > 2. **Option 2: Add a New onAcknowledgement Method with an Additional > > Parameter for Headers or Producer Record** > > - This option was brainstormed from the original thread > > <https://www.mail-archive.com/dev@kafka.apache.org/msg100650.html> > > - **Prototype**: [ > > > https://github.com/jychen7/kafka/pull/2](https://github.com/jychen7/kafka/pull/2) > > > > I prefer Option 1 because it is backward compatible, does not introduce > any > > new API, and does not require deprecating any existing APIs. In my > opinion, > > headers are a form of metadata, and RecordMetadata is created on the > client > > side without needing extra information from the broker. > > > > What are your thoughts on these options? If there are other suggestions, > I > > am open to considering them as well. > > > > Best regards, > > Rich > > > > > > On Sat, Jul 13, 2024 at 1:50 PM Matthias J. Sax <mj...@apache.org> > wrote: > > > >> Hi Rich, > >> > >> thanks for reaching out. I don't see any reason to object to this > >> proposal. The best way forward would be to write a KIP and collect > >> feedback from there. > >> > >> > >> -Matthias > >> > >> On 7/12/24 9:14 PM, Rich C. wrote: > >>> Dear Kafka Development Community, > >>> > >>> I propose enhancing the Kafka ProducerInterceptor interface, > specifically > >>> to make Record Headers available in the onAcknowledgement method. I > would > >>> appreciate your feedback on this proposal. If the feedback is > positive, I > >>> will follow up with a detailed discussion on implementing this feature. > >>> > >>> *Current State* > >>> > >>> At present, the topic, partition, offset, and timestamp are available > in > >>> the onAcknowledgement method. However, headers are not accessible. > >>> > >>> *Why This Feature Is Important* > >>> > >>> Two primary use cases highlight the importance of making headers > >> available > >>> in the onAcknowledgement method: > >>> > >>> 1. *Latency Measurement* > >>> > >>> Latency measurement is crucial for understanding the time taken for > >>> messages to travel from the producer to Kafka and back to the producer > as > >>> an acknowledgment. The current setup does not allow for precise > >> measurement > >>> of the producer-side latency (a) in the following scenario: > >>> > >>> ``` > >>> producer send -> (a) -> Kafka -> (b) -> acknowledge > >>> ``` > >>> > >>> - If using CreateTime, the calculation is: `now - > message.timestamp = > >>> (a) + (b)` > >>> > >>> - If using LogAppendTime, the calculation is: `now - > >> message.timestamp = > >>> (b)` > >>> > >>> By making headers available in onAcknowledgement, we can include a > >>> timestamp in the header when the message is sent. This allows us to > >>> calculate the producer-side latency (a) as `message.timestamp - > >>> CreateTimeFromHeader` when using LogAppendTime. > >>> > >>> 2. *Tracing Completeness* > >>> > >>> In distributed systems, tracing is essential for tracking the flow of > >>> messages and understanding system behavior. The trace ID is typically > >>> stored in the message headers. Having access to headers in the > >>> onAcknowledgement method would enable us to add spans indicating when > the > >>> message arrives at Kafka and when it is acknowledged by the client. > This > >>> would significantly enhance tracing completeness and accuracy. > >>> > >>> *Why Not Use Callback on Send* > >>> > >>> While it is possible to use the Callback mechanism provided by the send > >>> method, this approach has limitations: > >>> > >>> - *Injectability*: In some cases, such as with Debezium and Flink Kafka > >>> sink, the Callback is not injectable, making it impractical to use this > >>> method for the aforementioned purposes. > >>> > >>> I believe that adding Record Headers to the onAcknowledgement method in > >>> ProducerInterceptor would significantly enhance Kafka's capabilities in > >>> latency measurement and tracing, and also open up new opportunities for > >>> further improvements and innovations. > >>> > >>> I look forward to your feedback on this proposal. If the community is > in > >>> favor, I will provide a detailed follow-up on how this feature can be > >>> implemented. While KIP-512 was considered, we can discuss other options > >> in > >>> the next follow-up. > >>> > >>> Thank you for your consideration. > >>> > >>> Best regards, > >>> Rich Chen > >>> > >> > > >
