[
https://issues.apache.org/jira/browse/KAFKA-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739243#comment-15739243
]
ASF GitHub Bot commented on KAFKA-4521:
---------------------------------------
GitHub user lindong28 opened a pull request:
https://github.com/apache/kafka/pull/2241
KAFKA-4521; MirrorMaker should flush all messages before releasing
partition ownership during rebalance
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lindong28/kafka KAFKA-4521
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2241.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2241
----
commit cdbd79189f1cdb53f93d014c9b9758a5941b8168
Author: Dong Lin <do...@linkedin.com>
Date: 2016-12-11T07:03:46Z
KAFKA-4521; MirrorMaker should flush all messages before releasing
partition ownership during rebalance
----
> MirrorMaker should flush all messages before releasing partition ownership
> during rebalance
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-4521
> URL: https://issues.apache.org/jira/browse/KAFKA-4521
> Project: Kafka
> Issue Type: Bug
> Reporter: Dong Lin
> Assignee: Dong Lin
>
> In order to ensure that messages from a given partition in the source cluster
> are mirrored to the same partition in the destination cluster in the *same*
> order, MirrorMaker needs to produce and flush all messages that its consumer
> has received from source cluster before giving up partition in the cluster.
> However, as of current implementation of Apache Kafka, this is not guaranteed
> and will cause out-of-order message delivery in the following scenario.
> - mirror maker process 1 fetches messages 1, 2, 3, from source cluster in one
> FetchRequest
> - mirror maker process 2 starts up and triggers rebalance.
> - `InternalRebalanceListenerForOldConsumer` is executed by zookeeper listener
> thread, which does producer.flush() and commit offset of this consumer.
> However, at this moment messages 1, 2, 3 haven't even been produced.
> - consumer of mirror maker process 1 releases ownership of this partition
> - consumer of mirror maker process 2 gets ownership of this partition.
> - mirror maker process 2 fetches messages 4, 5, 6 from source cluster.
> - messages 4, 5, 6 can be produced before messages 1, 2, 3.
> To fix this problem, the rebalance listener callback function should signal
> MirrorMakerThread to get all messages from consumer, produce these messages
> to destination cluster, flush producer, and commit offset. Rebalance listener
> callback function should wait for MirrorMakerThread to finish these steps
> before it allows ownership of this partition to be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
