[
https://issues.apache.org/jira/browse/KAFKA-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603743#comment-17603743
]
Guozhang Wang commented on KAFKA-14224:
---------------------------------------
cc [~pnee] I think we can address this by always pairing the commit with the
revocation whenever the latter happened when auto-commit is enabled. More
specifically, anytime the consumer need to revoke a partition, it would:
1) "mute" the partition from returning any more data to the poll call, so that
its position would not advance any more.
2) set the flag so that the next poll call from the polling thread would then
trigger the {{onPartitionRevoked}} listener.
3) after 2), we know that the position of those partitions would not advance
any more, and we can then send out the commit offset req.
4) After the resp is returned, remove those partitions, so that the next HB
could let the coordinator know that it has removed those partitions.
> Consumer should auto-commit prior to cooperative partition revocation
> ---------------------------------------------------------------------
>
> Key: KAFKA-14224
> URL: https://issues.apache.org/jira/browse/KAFKA-14224
> Project: Kafka
> Issue Type: Bug
> Reporter: Jason Gustafson
> Priority: Major
> Labels: new-consumer-threading-should-fix
>
> With the old "eager" reassignment logic, we always revoked all partitions
> prior to each rebalance. When auto-commit is enabled, a part of this process
> is committing current position. Under the new "cooperative" logic, we defer
> revocation until after the rebalance, which means we can continue fetching
> while the rebalance is in progress. However, when reviewing KAFKA-14196, we
> noticed that there is no similar logic to commit offsets prior to this
> deferred revocation. This means that cooperative consumption is more likely
> to lead to have duplicate consumption even when there is no failure involved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
