Thanks Radai. Great to have a concrete example of the intended usage. Regarding performance, we would need to benchmark, as you said. But there would be a lot of reuse (in essence, we are copying 5 references plus a new object header), so I'd be surprised if that would be the bottleneck compared to some of the other allocations that would be happening in that path. In any case, I think we can leave this aside for now since people also felt that the mutable API would be easier to use.
About ProducerRecord reuse, my understanding is that people do sometimes retry a failed request manually due to the fact that a large retry number doesn't help if a batch is expired in the queue. I believe KIP-91 will help: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 91+Provide+Intuitive+User+Timeouts+in+The+Producer In addition, KIP-98 (Exactly-once) won't achieve its goal if people do manual retries. So, it seems like it's OK to require people to create a new ProducerRecord if they really need to do manual retries. But we should add a note to the compatibility section of the KIP. I have a few minor API suggestions. I'll send a follow-up later today, hopefully. Ismael On Mon, Mar 13, 2017 at 6:23 PM, radai <radai.rosenbl...@gmail.com> wrote: > the common "stack" we envision at linkedin would consist of (at least) the > following components that add headers to every outgoing request: > > 1. auditing/"lineage" - appends a header containing "node" (hostname etc), > time (UTC time) and destination (cluster/topic). these accumulate as > requests get mirrored between clusters > 2. serialization - sets a header containing a schema identifier to allow > deserialization > 3. client-side encryption - would probably set a header identifying the > key/scheme used > 4. internal "billing" > > there are also several other teams at linkedin that would use headers > (although its unclear yet if via interceptors or by directly manipulating > requests) > > if headers are made completely immutable (as the entire request object > currently is) we would end up copying (parts of) every msg 4 times. I > havent benchmarked but this seems like it would have an impact to me. > > looking elsewhere rabbitMQ and http components both use mutable request > objects (rabbitMW's BasicProperties object, http components' addHeader > method). > > how common is it right now for instances of ProducerRecord to actually be > reused? > do people really have things like publis static final ProducerRecord > MY_FAVORITE_REQUEST = ... ? >
