We normally run over 1,000 partitions per broker, and I know of a major
company with 30+ kafka clusters that averages 1,100 partitions per broker
across all clusters.
So 300 shouldn't be an issue as long as the throughput per partition isn't
too high. Given that disk and cpu are so low, I'd guess
300 partitions is at lower side, surely that won't be the root cause.
How about usage of network bandwidth for your nodes?
Are they reachable to zookeeper?
Are you executing some partition-rebalancing jobs (or re-assigning
partitions) in parallel?
On 2 March 2017 at 10:57, Jun MA wrote:
> Hi,
>
Hi,
I’m curious what’s the recommended number of partitions running on each
individual broker? We have a 3 nodes clusters running 0.9.0.1, each one has 24
cores, 1.1T ssd, 48G ram, 10G NIC. There’s about 300 partitions running on each
broker and the resource usage is pretty low (5% cpu, 50% ram
