What is the performance
overhead if we don't use batching?
It can be very significant. I'm not sure if we have published performance
numbers for the new producer yet, but you can see the 0.8 producer
throughput numbers here -
https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing#P
I have a possibly related question:
In a batch sending mode it's possible that messages are lost if
producer dies while accumulating the batch. What is the performance
overhead if we don't use batching?
Thanks.
On Wed, Mar 26, 2014 at 9:24 AM, Jagbir wrote:
> Hi Neha,
>
> Thank you for the nu
Hi Neha,
Thank you for the numbers as this information helped me to size my project as
well. While we are on this topic, do you mind commenting on the maximum number
of topics that can be hosted on a 8-10 node cluster. We have a need for about
100k topics and similar number of producers and
You shouldn't have any problem with that. We frequently have 10s of
thousands of producer connections to a 8-10 node cluster at all times. You
might have to bump up the limit for the number of open file handles per
broker though.
Thanks,
Neha
On Tue, Mar 25, 2014 at 3:41 PM, Kane Kane wrote:
>
Is there a recommended cap for the concurrent producers threads?
We plan to have around 4000 connections across cluster writing to
kafka, i assume there shouldn't be any performance implications
related to that?
Thanks.
