Re: Question regarding Producer and Duplicates

2016-09-05 Thread Florian Hussonnois
Thanks Guozhang, So increasing batch.size can lead to more duplicates in case of failure. Also when you said : "The broker will accept a batch of records as a whole or reject them ". For example, if a producer request contains two batches, a first one for Topic A / Partition 0 and a second one fo

Re: Question regarding Producer and Duplicates

2016-08-31 Thread Guozhang Wang
Hi Florian, The broker will accept a batch of records as a whole or reject them as a whole unless it encounters an IOException while trying to append the messages, which will be treated as a fatal error anyways. Duplicates usually happen when the whole batch is accepted but the ack was not delive

Question regarding Producer and Duplicates

2016-08-30 Thread Florian Hussonnois
Hi all, I am using kafka_2.11-0.10.0.1, my understanding is that the producer API batches records per partition to send efficient requests. We can configure batch.size to increase the throughtput. However, in case of failure all records within the batch failed ? If that is true, does that mean t