Thanks Guozhang for the context. The producer batch is either bounded by the size or the linger time. For the default 10ms linger and 100ms transaction commit time, the producer will be capped by AddPartitionToTxn 10 times in the worst case. I think the improvement here aims for the worst case scenario for users who didn't realize how the internal works, and uses the API calls in a very inefficient way as the scenario where record processing and send() happen concurrently.
Boyang On Sat, May 16, 2020 at 10:19 AM Guozhang Wang <wangg...@gmail.com> wrote: > Hello Boyang, > > Thanks for the proposed KIP, overall it makes sense to me. > > One non-public API related point that I'd like to make though, is that in > KafkaProducer.send call we can potentially defer sending > AddPartitionsToTxn request until the sender is about to send the first > batch -- this is what I observed from some soak testing investigation such > that the batching effects actually allows the first record to be sent much > later than the send() call and that can be leveraged to further reduce the > time that we would be blocked on the AddPartitionsToTxn request. > > > Guozhang > > > On Thu, May 14, 2020 at 10:26 PM Boyang Chen <reluctanthero...@gmail.com> > wrote: > > > Hey all, > > > > I would like to start the discussion for KIP-609: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-609%3A+Use+Pre-registration+and+Blocking+Calls+for+Better+Transaction+Efficiency > > > > This KIP aims to improve the current EOS semantic which makes the > > processing more efficient and consolidated. > > > > Thanks! > > > > > -- > -- Guozhang >
