[ https://issues.apache.org/jira/browse/KAFKA-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322246#comment-14322246 ]
Jun Rao commented on KAFKA-1952: -------------------------------- Uploaded a new patch that addresses the review comments. Some test results. Create a 2 node cluster locally. Create a topic X with 3000 partitions and a replication factor of 2 (wait until all leaders are elected; this can take 5 minutes). Without the patch, when there is no produce load on the cluster, each broker uses 100% CPU and the localTime for each replica fetch request is about 1 sec. With the patch, each broker uses about 10% CPU and the localTime for each replica fetch request is about 9ms. Also tested the end to end latency with the patch. On the above 2 node cluster, create a topic test with 1 partition and a replication factor of 2. bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency localhost:9092 localhost:2181 test 5000 500 -1 0 61.206 1000 26.766 2000 38.58 3000 25.647 4000 26.991 Avg latency: 28.5952 ms Percentiles: 50th = 27, 99th = 41, 99.9th = 52 The latency is bounded by the message commit time. With 3000 partitions, each replica fetch request take 9ms in localTime and another 9ms in sendTime. So the time to complete a produce request (with ack=-1) is about 26ms on average. In any case, the end to end latency is way less than the maxWaitTime in the regular consumer and replica fetcher (both at 500ms), which is expected. Also measured the localTime and sendTime for the same replica fetch requests on 0.8.1.1. The numbers are comparable with 0.8.2.0 with the patch. > High CPU Usage in 0.8.2 release > ------------------------------- > > Key: KAFKA-1952 > URL: https://issues.apache.org/jira/browse/KAFKA-1952 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.8.2.0 > Reporter: Jay Kreps > Assignee: Jun Rao > Priority: Critical > Fix For: 0.8.2.0 > > Attachments: kafka-1952.patch, kafka-1952_2015-02-15_15:26:33.patch > > > Brokers with high partition count see increased CPU usage when migrating from > 0.8.1.1 to 0.8.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
