This sounds like flush: https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API
which was recently implemented in trunk. Joel On Mon, Apr 27, 2015 at 08:19:40PM +0000, Roshan Naik wrote: > Been evaluating the perf of old and new Produce APIs for reliable high volume > streaming data movement. I do see one area of improvement that the new API > could use for synchronous clients. > > AFAIKT, the new API does not support batched synchronous transfers. To do > synchronous send, one needs to do a future.get() after every Producer.send(). > I changed the new o.a.k.clients.tools.ProducerPerformance tool to asses the > perf of this mode of operation. May not be surprising that it much slower > than the async mode... hard t push it beyond 4MB/s. > > The 0.8.1 Scala based producer API supported a batched sync mode via > Producer.send( List<KeyedMessage> ) . My measurements show that it was able > to approach (and sometimes exceed) the old async speeds... 266MB/s > > > Supporting this batched sync mode is very critical for streaming clients > (such as flume for example) that need delivery guarantees. Although it can be > done with Async mode, it requires additional book keeping as to which events > are delivered and which ones are not. The programming model becomes much > simpler with the batched sync mode. Client having to deal with one single > future.get() helps performance greatly too as I noted. > > Wanted to propose adding this as an enhancement to the new Producer API.
