What is the performance overhead if we don't use batching? It can be very significant. I'm not sure if we have published performance numbers for the new producer yet, but you can see the 0.8 producer throughput numbers here - https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing#Performancetesting-Producerthroughput. The performance difference between batch size 1 and 50 can be upto 18x.
Thanks, Neha On Wed, Mar 26, 2014 at 7:24 PM, Kane Kane <kane.ist...@gmail.com> wrote: > I have a possibly related question: > > In a batch sending mode it's possible that messages are lost if > producer dies while accumulating the batch. What is the performance > overhead if we don't use batching? > > Thanks. > > > On Wed, Mar 26, 2014 at 9:24 AM, Jagbir <jsho...@hotmail.com> wrote: > > Hi Neha, > > > > Thank you for the numbers as this information helped me to size my > project as well. While we are on this topic, do you mind commenting on the > maximum number of topics that can be hosted on a 8-10 node cluster. We have > a need for about 100k topics and similar number of producers and consumers > and I'm looking for some rough estimates. > > > > Thanks, > > Jagbir > > > > On March 25, 2014 8:59:21 PM PDT, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > >>You shouldn't have any problem with that. We frequently have 10s of > >>thousands of producer connections to a 8-10 node cluster at all times. > >>You > >>might have to bump up the limit for the number of open file handles per > >>broker though. > >> > >>Thanks, > >>Neha > >> > >> > >>On Tue, Mar 25, 2014 at 3:41 PM, Kane Kane <kane.ist...@gmail.com> > >>wrote: > >> > >>> Is there a recommended cap for the concurrent producers threads? > >>> We plan to have around 4000 connections across cluster writing to > >>> kafka, i assume there shouldn't be any performance implications > >>> related to that? > >>> > >>> Thanks. > >>> > > > > -- > > Sent from my Android phone with K-9 Mail. Please excuse my brevity. >
