There's no strict limit on the number of producers. If you're hitting some CPU limit, perhaps you are simply overloading the broker? 6 or 700 brokers doesn't sound that bad, but if they are producing too much data then of course eventually the broker will become overwhelmed. How much total data is being produced?
-Ewen On Thu, Jul 21, 2016 at 10:07 AM, Nazario Parsacala <dodongj...@gmail.com> wrote: > Kafka Users, > > Are there any settings that limits the number of producers per topic per > broker ..? I am experimenting with a single broker with around 500 > producers and this works fine .. but increasing this further to 600 , 700 > producers and suddenly the kafka broker stops functioning .. CPU > utilization shoots up and no data is being received by the consumers. If > we bring bring the producer back to 500 , the broker starts functioning > fine again. > > The thread dump on the broker seems to point to the threads being blocked > on the SSL side ..? > For experimentation purpose we have increased already the following : > > # The number of threads handling network requests > #num.network.threads=3 > num.network.threads=128 > > # The number of threads doing disk I/O > #num.io.threads=8 > num.io.threads=256 > > But no dice .. > > BTW I am using kafka_2.10-0.9.0.1 -- Thanks, Ewen
