Hi Marc, If you are seeing high CPU usages with a large number of partitions on 0.8.2 you should definitely upgrade to 0.8.2.1 as the following issue was fixed: https://issues.apache.org/jira/browse/KAFKA-1952
Also see the 0.8.2.1 release notes for other fixes: https://archive.apache.org/dist/kafka/0.8.2.1/RELEASE_NOTES.html Mark On 13 March 2015 at 15:10, Marc Labbe <mrla...@gmail.com> wrote: > Hi, > > our cluster is deployed on AWS, we have brokers on r3.large instances, a > decent amount of topics+partitions (+600 partitions). We're not making that > many requests/sec, roughly 80 produce/sec and 240 fetch/sec (not counting > internal replication requests) and yet CPU hovers around 40%, which I > consider quite high given the nature of Kafka. I have worked on other > deployments not on AWS where we were getting much larger figures in > requests/sec, w/ much less CPU usage than that. > > There are two things I consider trying to reduce that. The first one is > obviously using 0.8.2.1 though I am not sure how much the impact is. I also > found this thread from a while ago > > http://mail-archives.apache.org/mod_mbox/kafka-users/201305.mbox/%3ceb51b84c-ad91-4a2f-b97d-5283ef079...@transpac.com%3E > about the use of the OpenJDK (which we're using, Ubuntu trusty's default) > vs Oracle JDK. > > I am planning to do both anyways but I thought it'd be interesting to know > if anyone else has experienced that before. Is there any other tuning I > should think about? > > thanks > marc >
