You may need to consider replication factor as well? Here you increased 4 times of partitions but if you are having 3 replication factor then it will be (4 * 3) times of partition copies in the cluster.
On Sat, Nov 21, 2015 at 12:43 AM, Chen Song <chen.song...@gmail.com> wrote: > Any thoughts on this topic. Not sure if others have seen the same spike as > us. > > On Tue, Nov 17, 2015 at 3:51 PM, Chen Song <chen.song...@gmail.com> wrote: > > > BTW, we are running Kafka 0.8.2.2. > > > > On Tue, Nov 17, 2015 at 3:48 PM, Chen Song <chen.song...@gmail.com> > wrote: > > > >> We have a cluster of 32 nodes cluster, with a few topics. When we set > 100 > >> partitions for each topic, the overall CPU usage is 30%, and when we > >> increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. > The # > >> of producers are not changed. > >> > >> I understand that more partitions will drive up broker CPU usage. > >> However, this seems to be a limiting factor to parallelism (mostly for > >> consumer perspective) because all I test is just 400 partitions. > >> > >> Not sure if this is being addressed in an way. Any thoughts on this? > >> > >> -- > >> Chen Song > >> > >> > > > > > > -- > > Chen Song > > > > > > > -- > Chen Song >
