Stephane Maarek created KAFKA-5851:
--------------------------------------
Summary: CPU overhead to having partitions (even inactive)
Key: KAFKA-5851
URL: https://issues.apache.org/jira/browse/KAFKA-5851
Project: Kafka
Issue Type: Bug
Environment: 0.11.0.0, 0.10.x
Reporter: Stephane Maarek
We're running on three r4.xlarge, and this is our dev environment.
Some people managed to create 1200 topics with 3 partitions, so we end up at
4000 partitions per broker (we have a replication factor of 3). Even though no
data goes through the cluster (it sits at a comfortable 20 messages per
seconds), we saw CPU at 100% (out of 400%).
I went ahead today and deleted 700 topics that I knew were unused. And CPU went
drastically down. See image
!https://i.imgur.com/OIPTwDM.png!
We use the defaults for our brokers, and they use PLAINTEXT internally to
replicate.
I'm not sure of the root cause of this, threads, replication, log cleanup, etc,
and I guess it wouldn't be too hard to replicate (just create 1000 topics on a
vanilla cluster and see CPU go up).
Hope that helps
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
