[
https://issues.apache.org/jira/browse/KAFKA-10582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Martin updated KAFKA-10582:
----------------------------------
Description:
We are using Mirror Maker 2 from the 2.5.1 release for replication on some
clusters. Replication is working as expected for existing topics. When we
create a new topic, however, Mirror Maker 2 creates the replicated topic as
expected but never starts replicating it. If we restart Mirror Maker 2 within
2-3 minutes the topic starts replicating as expected. From documentation we
haveve seen it appears this should start replicating without a restart based on
the settings we have.
*Example:*
Create topic "mytesttopic" on source cluster
MirrorMaker 2 creates "source.mytesttopioc" on target cluster with no issue
MirrorMaker 2 does not replicate "mytesttopic" -> "source.mytesttopic"
Restart MirrorMaker 2 and now replication works for "mytesttopic" ->
"source.mytesttopic"
*Example config:*
name = source->target
group.id = source-to-target
# specify any number of cluster aliases
clusters = source, target
# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
source.bootstrap.servers = sourcehosts:9092
target.bootstrap.servers = targethosts:9092
# enable and configure individual replication flows
source->target.enabled = true
# regex which defines which topics gets replicated. For eg "foo-.*"
source->target.topics = .*
target->source = false
target->source.topics = .*
# Setting replication factor of newly created remote topics
replication.factor=3
# The replication factor for mm2 internal topics "heartbeats",
"B.checkpoints.internal" and
# "mm2-offset-syncs.B.internal"
# For anything other than development testing, a value greater than 1 is
recommended to ensure availability such as 3.
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3
# The replication factor for connect internal topics "mm2-configs.B.internal",
"mm2-offsets.B.internal" and
# "mm2-status.B.internal"
# For anything other than development testing, a value greater than 1 is
recommended to ensure availability such as 3.
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3
# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5
# tasks.max results in parallelism
tasks.max = 16
refresh.topics.enabled = true
sync.topic.configs.enabled = true
# Setting the below too low can result in performance issues
refresh.topics.interval.seconds = 300
refresh.groups.interval.seconds = 300
readahead.queue.capacity = 100
emit.checkpoints.enabled = true
emit.checkpoints.interval.seconds = 5
was:
We are using Mirror Maker 2 from the 2.5.1 release for replication on some
clusters. Replication is working as expected for existing topics. When we
create a new topic, however, Mirror Maker 2 creates the replicated topic as
expected but never starts replicating it. If we restart Mirror Maker 2 within
2-3 minutes the topic starts replicating as expected. From documentation we
haveve seen it appears this should start replicating without a restart based on
the settings we have.
*Example:*
Create topic "mytesttopic" on source cluster
MirrorMaker 2 creates "source.mytesttopioc" on target cluster with no issue
MirrorMaker 2 does not replicate "mytesttopic" -> "source.mytesttopic"
Restart MirrorMaker 2 and now replication works for "mytesttopic" ->
"source.mytesttopic"
*Example config:*
name = source->target
group.id = source-to-target
# specify any number of cluster aliases
clusters = source, target
# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
source.bootstrap.servers = sourcehosts:9092
target.bootstrap.servers = targethosts:9092
# enable and configure individual replication flows
source->target.enabled = true
# regex which defines which topics gets replicated. For eg "foo-.*"
source->target.topics = .*
target->source = false
target->source.topics = .*
# Setting replication factor of newly created remote topics
replication.factor=3
############################# Internal Topic Settings
#############################
# The replication factor for mm2 internal topics "heartbeats",
"B.checkpoints.internal" and
# "mm2-offset-syncs.B.internal"
# For anything other than development testing, a value greater than 1 is
recommended to ensure availability such as 3.
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3
# The replication factor for connect internal topics "mm2-configs.B.internal",
"mm2-offsets.B.internal" and
# "mm2-status.B.internal"
# For anything other than development testing, a value greater than 1 is
recommended to ensure availability such as 3.
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3
# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5
# tasks.max results in parallelism
tasks.max = 16
refresh.topics.enabled = true
sync.topic.configs.enabled = true
# Setting the below too low can result in performance issues
refresh.topics.interval.seconds = 300
refresh.groups.interval.seconds = 300
readahead.queue.capacity = 100
emit.checkpoints.enabled = true
emit.checkpoints.interval.seconds = 5
> Mirror Maker 2 not replicating new topics until restart
> -------------------------------------------------------
>
> Key: KAFKA-10582
> URL: https://issues.apache.org/jira/browse/KAFKA-10582
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.5.1
> Environment: RHEL 7 Linux.
> Reporter: Robert Martin
> Priority: Minor
>
> We are using Mirror Maker 2 from the 2.5.1 release for replication on some
> clusters. Replication is working as expected for existing topics. When we
> create a new topic, however, Mirror Maker 2 creates the replicated topic as
> expected but never starts replicating it. If we restart Mirror Maker 2
> within 2-3 minutes the topic starts replicating as expected. From
> documentation we haveve seen it appears this should start replicating without
> a restart based on the settings we have.
> *Example:*
> Create topic "mytesttopic" on source cluster
> MirrorMaker 2 creates "source.mytesttopioc" on target cluster with no issue
> MirrorMaker 2 does not replicate "mytesttopic" -> "source.mytesttopic"
> Restart MirrorMaker 2 and now replication works for "mytesttopic" ->
> "source.mytesttopic"
> *Example config:*
> name = source->target
> group.id = source-to-target
> # specify any number of cluster aliases
> clusters = source, target
> # connection information for each cluster
> # This is a comma separated host:port pairs for each cluster
> # for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
> source.bootstrap.servers = sourcehosts:9092
> target.bootstrap.servers = targethosts:9092
> # enable and configure individual replication flows
> source->target.enabled = true
> # regex which defines which topics gets replicated. For eg "foo-.*"
> source->target.topics = .*
> target->source = false
> target->source.topics = .*
> # Setting replication factor of newly created remote topics
> replication.factor=3
> # The replication factor for mm2 internal topics "heartbeats",
> "B.checkpoints.internal" and
> # "mm2-offset-syncs.B.internal"
> # For anything other than development testing, a value greater than 1 is
> recommended to ensure availability such as 3.
> checkpoints.topic.replication.factor=3
> heartbeats.topic.replication.factor=3
> offset-syncs.topic.replication.factor=3
> # The replication factor for connect internal topics
> "mm2-configs.B.internal", "mm2-offsets.B.internal" and
> # "mm2-status.B.internal"
> # For anything other than development testing, a value greater than 1 is
> recommended to ensure availability such as 3.
> offset.storage.replication.factor=3
> status.storage.replication.factor=3
> config.storage.replication.factor=3
> # customize as needed
> # replication.policy.separator = _
> # sync.topic.acls.enabled = false
> # emit.heartbeats.interval.seconds = 5
> # tasks.max results in parallelism
> tasks.max = 16
> refresh.topics.enabled = true
> sync.topic.configs.enabled = true
> # Setting the below too low can result in performance issues
> refresh.topics.interval.seconds = 300
> refresh.groups.interval.seconds = 300
> readahead.queue.capacity = 100
> emit.checkpoints.enabled = true
> emit.checkpoints.interval.seconds = 5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
