Patrik Márton created KAFKA-17200:
-------------------------------------
Summary: Enable MM2 to replicate topics ending in "internal" suffix
Key: KAFKA-17200
URL: https://issues.apache.org/jira/browse/KAFKA-17200
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Reporter: Patrik Márton
Assignee: Patrik Márton
In the current Mirror Maker 2 implementation, topics ending in ".internal" or
"-internal" cannot be replicated as they are considered connect/mm2 internal
topics.
In some cases, users have business topics ending in ".internal" or "-internal"
that are excluded from the replication for the same reason. This is because of
two things:
(1) The ReplicationPolicy interface excludes all topics ending in ".internal"
or "-internal" from the replication, as they are considered internal connect
topics:
{code:java}
/** Internal topics are never replicated. */
default boolean isInternalTopic(String topic) {
boolean isKafkaInternalTopic = topic.startsWith("__") ||
topic.startsWith(".");
boolean isDefaultConnectTopic = topic.endsWith("-internal") ||
topic.endsWith(".internal");
return isMM2InternalTopic(topic) || isKafkaInternalTopic ||
isDefaultConnectTopic;
} {code}
(2) The DefaultTopicFilter has the following default exclude regular expression:
{code:java}
".*[\\-\\.]internal, .*\\.replica, __.*" {code}
The goal of this ticket is to enable the replication of such business topics,
while making sure that kafka internal topics are not replicated.
*Solution 1:* The DefaultTopicFilter can be configured to have a different
exclude list with more specific regex to exclude all kafka internal topics, but
include business internal topics, eg.:
{code:java}
"mm2.*[\\-\\.]internal, .*\\.replica, __.*"; {code}
As these topics are explicitly excluded in the ReplicationPolicy interface too,
we can have a new replication policy from DefaultReplicationPolicy, that
preserves the same behavior, but overrides the isInternalTopic() method in a
way that topics ending with the internal suffix are not considered internal
topics. Obviously in this case it would be important to set up the topic filter
correctly to avoid replicating kafka internal topics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
