We have a cluster of 32 nodes cluster, with a few topics. When we set 100 partitions for each topic, the overall CPU usage is 30%, and when we increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The # of producers are not changed.
I understand that more partitions will drive up broker CPU usage. However, this seems to be a limiting factor to parallelism (mostly for consumer perspective) because all I test is just 400 partitions. Not sure if this is being addressed in an way. Any thoughts on this? -- Chen Song
