[
https://issues.apache.org/jira/browse/KAFKA-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-14610:
-------------------------------------
Assignee: Chris Egerton
> Publish Mirror Maker 2 offset syncs in task commit method
> ---------------------------------------------------------
>
> Key: KAFKA-14610
> URL: https://issues.apache.org/jira/browse/KAFKA-14610
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> Mirror Maker 2 periodically publishes offset sync messages to a Kafka topic
> that contains the corresponding upstream and downstream offsets for a
> replicated topic partition.
>
> Currently, this publishing takes place inside the [commitRecord
> method|https://github.com/apache/kafka/blob/e38526e375389868664c8977c7a2125e5da2388c/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L192],
> which is invoked by the Kafka Connect framework after a source record has
> been successfully sent by its producer (i.e., ack'd by the requested number
> of brokers).
>
> Mirror Maker 2 also has logic to limit the number of in-flight offset sync
> messages. Once ten messages have been dispatched to the producer used for
> offset syncs (which is a separate producer from the one that the Kafka
> Connect framework uses for sending records received from the [poll
> method|https://github.com/apache/kafka/blob/e38526e375389868664c8977c7a2125e5da2388c/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L134])
> that have not yet been ack'd by the requested number of brokers, Mirror
> Maker 2 begins to skip sending offset sync messages, and will only resume
> sending messages once the number of in-flight offset syncs goes below 10, and
> new calls to the {{commitRecord}} method take place.
>
> When bursts of throughput occur in replicated topic partitions, this can
> cause offset syncs to be dropped for long periods of time if an offset sync
> is skipped for some topic partition due to a high number of in-flight
> messages and then no further messages are read from that same topic partition
> for a while.
>
> Instead, the task should cache offset syncs in its {{{}commitRecord
> method{}}}, and only actually send offset sync messages in its [commit
> method|https://github.com/apache/kafka/blob/e38526e375389868664c8977c7a2125e5da2388c/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L108],
> which is invoked periodically by the Kafka Connect framework. Any offset
> syncs that are skipped due to too many in-flight messages will then be
> automatically retried later when {{commit}} is re-invoked, regardless of
> whether any more records are read from the corresponding topic partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
