Philip Nee created KAFKA-18376:
----------------------------------
Summary: High CPU load when AsyncKafkaConsumer uses a small max
poll value
Key: KAFKA-18376
URL: https://issues.apache.org/jira/browse/KAFKA-18376
Project: Kafka
Issue Type: Bug
Components: clients, consumer
Reporter: Philip Nee
Assignee: Philip Nee
We stress tested the AsyncConsumer using maxPoll = 5 and observed abnormally
high CPU usage. Under normal usage (with defaults), the consumers on average
use around 10% of the CPU with 20mb/s byte rate, which is aligned with the
ClassicKafkaConsumer. As we tested the consumer with a small max poll value,
we observed the CPU usage spikes to > 50% while the classic consumer stays at
around 10%.
_note: percentage of CPU usage may depend on the running pod hardware._
The profiling results shows two major contributors to the CPU cycles
# AsyncKafkaConsumer.updateFetchPosition (addAndGet & new
CheckAndUpdatePositionEvent())
# AbstractFetch.fetchablePartitions from the fetchrequestmanager
for AsyncKafkaConsumer.updateFetchPosition - It seems like
* UUID generation can become quite expensive. This is particularly noticeable
when creating large number of events
* ConsumerUtils.getResult, which uses future.get() also consumes quite a bit
of CPU cycles
for fetchablePartitions, FetchBuffer.bufferedPartitions which uses Java
ConcurrentLinkedQueue.forEach also consumes quite a bit of CPUs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
