Hi Neelesh :) The new producer has configuration for controlling the batch sizes. By default, it will batch as much as possible without delay (controlled by linger.ms) and without using too much memory (controlled by batch.size).
As mentioned in the docs, you can set batch.size to 0 to disable batching completely if you want. It is worthwhile to consider using the producer callback to avoid losing messages when the webservice crashes (for example have the webservice only consider messages as sent if the callback is triggered for a successful send). You can read more information on batching here: http://ingest.tips/2015/07/19/tips-for-improving-performance-of-kafka-producer/ And some examples on how to produce data to Kafka with the new producer - both with futures and callbacks here: https://github.com/gwenshap/kafka-examples/blob/master/SimpleCounter/src/main/java/com/shapira/examples/producer/simplecounter/DemoProducerNewJava.java Gwen On Fri, Aug 14, 2015 at 5:07 PM, Neelesh <neele...@gmail.com> wrote: > We are fronting all our Kafka requests with a simple web service (we do > some additional massaging and writing to other stores as well). The new > KafkaProducer in 0.8.2 seems very geared towards producer batching. Most of > our payload are single messages. > > Producer batching basically sets us up for lost messages if our web service > goes down with unflushed messaged in the producer. > > Another issue is when we have a batch of records. It looks like I have to > call producer.send for each record and deal with individual futures > returned. > > Are there any patterns for primarily single message requests, without > losing data? I understand the throughput will be low. > > Thanks! > -Neelesh >
