Jarkko Jaakola created KAFKA-18125:
--------------------------------------
Summary: Implement MirrorCheckpointConnector max task limit
Key: KAFKA-18125
URL: https://issues.apache.org/jira/browse/KAFKA-18125
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Reporter: Jarkko Jaakola
Mirrormaker 2 MirrorCheckpointConnector currently can create as many tasks as
globally configured. Primary limiting factor for number of tasks up to the
configured maximum is the number of known consumer groups. On larger mirrored
environments where known consumers groups is large this leads to excessive
amount of checkpoint tasks.
As the offset sync topic is single partition topic there will be excessive
amount of tasks reading from and writing to this topic. The targeted Kafka
broker can and will be under heavy burden.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
