After digging more into the stack trace got from flight recorder (which is attached), it seems that Kafka (0.8.1.1) can optimize the usage of Crc32. The stack trace shows that Crc32 is invoked twice from Log.append(). First is from the line number 231:
val appendInfo = analyzeAndValidateMessageSet(messages) The second time is from line 252 in the same function: validMessages = validMessages.assignOffsets(offset, appendInfo.codec) If one of the Crc32 invocation can be eliminated, we are looking at saving at least 7% of CPU usage. Thanks, Allen On Wed, Nov 5, 2014 at 6:32 PM, Allen Wang <aw...@netflix.com> wrote: > Hi, > > Using flight recorder, we have observed high CPU usage of CRC32 > (kafka.utils.Crc32.update()) on Kafka broker. It uses as much as 25% of CPU > on an instance. Tracking down stack trace, this method is invoked by > ReplicaFetcherThread. > > Is there any tuning we can do to reduce this? > > Also on the topic of CPU utilization, we observed that overall CPU > utilization is proportional to AllTopicsBytesInPerSec metric. Does this > metric include incoming replication traffic? > > Thanks, > Allen > >
