Justinwins created KAFKA-14171:
----------------------------------
Summary: mm2-offsets topic should be set retention.ms=1h or less
as default
Key: KAFKA-14171
URL: https://issues.apache.org/jira/browse/KAFKA-14171
Project: Kafka
Issue Type: Bug
Affects Versions: 3.2.1
Reporter: Justinwins
- This seems a small bug (or improvment) ,but it really impacts perf of mm2.
- When DistributedHerder starts, it will startServices()-->
this.worker.start() -->
offsetBackingStore.start() --> offsetLog.start() ,and finally in
`KafkaBasedLog` class ,we
see `consumer.seekToBeginning(partitions)` .
Take a look at `org.apache.kafka.connect.util.KafkaBasedLog#start` ,you will
get to know it.
- Basically, mm2-offsets topic will be kept for 7 days (as defined by
'retention.ms' ) . If there are many paritions for mm2 to replicate ,then
mm2-offsets topic may be quite 'big' in 7 days. And it may take a few
minutes or more to poll unitil the consumer reaches the latest . This is a VERY
Cpu-consuming action, and it incurs cpu throttle in the k8s container.
- I think mm-offsets topic ,or to be specific , KafkaBasedLog is a special
topic .At least, we can set a much shorter ttl for it to avoid this bug .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
