Have you considered adding a new onAcknowledgement method to the ProducerInterceptor with the signature onAcknowledgement(RecordMetadata metadata, Exception exception, ProducerRecord record)? I would also consider adding this to Producer Callbacks as well, since linking a Callback to a specific record currently requires creating a new Callback for every ProducerRecord sent.
This seems like a more robust strategy compared to using Headers. Headers don't necessarily contain anything that connects them to the original ProducerRecord, and forcibly including information in the Headers seems like unnecessary bloat. If your goal is to link a RecordMetadata to a specific ProducerRecord, it seems simpler to make sure the original ProducerRecord is accessible at the same time as the RecordMetadata On Mon, Aug 26, 2019 at 10:26 AM Renuka M <renumetuk...@gmail.com> wrote: > Hi Gwen, > > 1.We are not doing any changes on the broker side. This change is only on > Kafka clients library. > 2. RecordMetaData is created by client library while appending record to > ProducerBatch where offset alone returned by broker. Here we are adding > headers to RecordMetaData while creating FutureRecordMetaData to create > context between record and its metadata. I have updated the snippet in KIP > proposed changes in step 3. > 3. As we mentioned in alternatives, client side we can link record and its > metadata using callback, but Interceptors having same RecordMetadata will > not have context on for which record this MetaData belongs to. To fill that > Gap, we are proposing these changes. > Please let us know if we are not clear. > > Thanks > Renuka M > > > > > On Fri, Aug 23, 2019 at 7:08 PM Gwen Shapira <g...@confluent.io> wrote: > > > I am afraid I don't understand the proposal. The RecordMetadata is > > information returned from the broker regarding the record. The > > producer already has the record (including the headers), so why would > > the broker need to send the headers back as part of the metadata? > > > > On Fri, Aug 23, 2019 at 4:22 PM Renuka M <renumetuk...@gmail.com> wrote: > > > > > > Hi All, > > > > > > I am starting this thread to discuss > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData > > > . > > > > > > Please provide the feedback. > > > > > > Thanks > > > Renuka M > > > > > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > >
