Hey,
Resurrecting KIP-512 might be a good idea. Just follow up on the Jira
ticket and/or old DISCUSS thread, and ask to take over -- if the
original author does not reply in a few days, just take it on; given
that the KIP is many years old, I would be surprised if anybody objects
that you resurrect it and take over.
However, I would personally not add Headers to RecordMetadata, but
follow the idea to pass Headers as additional parameter (which was you
original proposal, and was also mentioned as alternative on the KIP-512
discuss thread).
As pointed out on the KIP-512 discussion, RecordMetadata is "Kafka
metadata" about the records (send back by the broker) like the offset.
Headers seems to be of a different quality because it's
"user/application metadata"), and I believe it would become a convoluted
API to add them to RecordMetadata where Headers IMHO do not belong to.
But it seems we are already starting a discussion which should actually
take place on the KIP discuss thread :) -- And it's of course just my
personal opinion. In the end, the question will be what the majority of
people believe is better.
The other argument on KIP-512 was, that no change is necessary at all
because Headers exist already client side. While I see the point, I
don't really agree to it... In the end, while technically correct, it's
quite complicated and clumsy to keep a copy of the Headers in the
application code, and link them back and pass into an interceptor or
callback to be used there. So letting the producer pass them into the
interceptor and/or callback, seems to be a good improvement for user
experience and ease of use.
Overall, it seem that the motivation must be clearly explained and the
argument of "no needed" but be overcome in a convincing way, to get the
KIP accepted. The API design question seems to be of second nature, and
I am sure an agreement can be found, if people agree that a change is
needed in the first place.
-Matthias
On 7/13/24 5:15 PM, Rich C. wrote:
Hi Matthias,
Thank you for the positive feedback. I am new to Kafka contributions and
have just requested a Confluence account for creating the KIP. I am
currently waiting for the account creation.
I noticed that KIP-512
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData>,
which addressed a similar issue with headers, was proposed five years ago.
Do you suggest that I edit and improve KIP-512 (pending agreement from the
original author) or create a new KIP?
My previous email focused on WHY we need this feature. This follow-up email
will discuss **HOW** we can implement it.
**Implementation Options**
1. **Option 1: Add Headers to RecordMetadata** (Original KIP-512 Approach)
- **Prototype**: [
https://github.com/jychen7/kafka/pull/1](https://github.com/jychen7/kafka/pull/1)
2. **Option 2: Add a New onAcknowledgement Method with an Additional
Parameter for Headers or Producer Record**
- This option was brainstormed from the original thread
<https://www.mail-archive.com/dev@kafka.apache.org/msg100650.html>
- **Prototype**: [
https://github.com/jychen7/kafka/pull/2](https://github.com/jychen7/kafka/pull/2)
I prefer Option 1 because it is backward compatible, does not introduce any
new API, and does not require deprecating any existing APIs. In my opinion,
headers are a form of metadata, and RecordMetadata is created on the client
side without needing extra information from the broker.
What are your thoughts on these options? If there are other suggestions, I
am open to considering them as well.
Best regards,
Rich
On Sat, Jul 13, 2024 at 1:50 PM Matthias J. Sax <mj...@apache.org> wrote:
Hi Rich,
thanks for reaching out. I don't see any reason to object to this
proposal. The best way forward would be to write a KIP and collect
feedback from there.
-Matthias
On 7/12/24 9:14 PM, Rich C. wrote:
Dear Kafka Development Community,
I propose enhancing the Kafka ProducerInterceptor interface, specifically
to make Record Headers available in the onAcknowledgement method. I would
appreciate your feedback on this proposal. If the feedback is positive, I
will follow up with a detailed discussion on implementing this feature.
*Current State*
At present, the topic, partition, offset, and timestamp are available in
the onAcknowledgement method. However, headers are not accessible.
*Why This Feature Is Important*
Two primary use cases highlight the importance of making headers
available
in the onAcknowledgement method:
1. *Latency Measurement*
Latency measurement is crucial for understanding the time taken for
messages to travel from the producer to Kafka and back to the producer as
an acknowledgment. The current setup does not allow for precise
measurement
of the producer-side latency (a) in the following scenario:
```
producer send -> (a) -> Kafka -> (b) -> acknowledge
```
- If using CreateTime, the calculation is: `now - message.timestamp =
(a) + (b)`
- If using LogAppendTime, the calculation is: `now -
message.timestamp =
(b)`
By making headers available in onAcknowledgement, we can include a
timestamp in the header when the message is sent. This allows us to
calculate the producer-side latency (a) as `message.timestamp -
CreateTimeFromHeader` when using LogAppendTime.
2. *Tracing Completeness*
In distributed systems, tracing is essential for tracking the flow of
messages and understanding system behavior. The trace ID is typically
stored in the message headers. Having access to headers in the
onAcknowledgement method would enable us to add spans indicating when the
message arrives at Kafka and when it is acknowledged by the client. This
would significantly enhance tracing completeness and accuracy.
*Why Not Use Callback on Send*
While it is possible to use the Callback mechanism provided by the send
method, this approach has limitations:
- *Injectability*: In some cases, such as with Debezium and Flink Kafka
sink, the Callback is not injectable, making it impractical to use this
method for the aforementioned purposes.
I believe that adding Record Headers to the onAcknowledgement method in
ProducerInterceptor would significantly enhance Kafka's capabilities in
latency measurement and tracing, and also open up new opportunities for
further improvements and innovations.
I look forward to your feedback on this proposal. If the community is in
favor, I will provide a detailed follow-up on how this feature can be
implemented. While KIP-512 was considered, we can discuss other options
in
the next follow-up.
Thank you for your consideration.
Best regards,
Rich Chen
