[
https://issues.apache.org/jira/browse/KAFKA-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788096#comment-16788096
]
ASF GitHub Bot commented on KAFKA-7703:
---------------------------------------
viktorsomogyi commented on pull request #6407: KAFKA-7703: position() may
return a wrong offset after seekToEnd
URL: https://github.com/apache/kafka/pull/6407
When poll is called which resets the offsets to the beginning, followed by a
seekToEnd and a position, it could happen that the "reset to earliest" call in
poll overrides the "reset to latest" initiated by seekToEnd in a very delicate
way:
1. both request has been issued and returned to the client side
(listOffsetResponse has happened)
2. in Fetcher.resetOffsetIfNeeded(TopicPartition, Long, OffsetData) the
thread scheduler could prefer the heartbeat thread with the "reset to earliest"
call, overriding the offset to the earliest and setting the SubscriptionState
with that position.
3. The thread scheduler continues execution of the thread (application
thread) with the "reset to latest" call and discards it as the "reset to
earliest" already set the position - the wrong one.
4. The blocking position call returns with the earliest offset instead of
the latest, despite it wasn't expected.
The fix makes the TopicPartitionState in SubscriptionState synchronized and
starts to track the requested reset timestamp. With this we can precisely
decide if the incoming offset reset is really what we want. Therefore the
latest initiated offset reset will happen only. Synchronization furthermore
ensures that this is done in an atomic manner to avoid further similar bugs.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> KafkaConsumer.position may return a wrong offset after "seekToEnd" is called
> ----------------------------------------------------------------------------
>
> Key: KAFKA-7703
> URL: https://issues.apache.org/jira/browse/KAFKA-7703
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0
> Reporter: Shixiong Zhu
> Assignee: Viktor Somogyi-Vass
> Priority: Major
>
> After "seekToEnd" is called, "KafkaConsumer.position" may return a wrong
> offset set by another reset request.
> Here is a reproducer:
> https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246
> In this reproducer, "poll(0)" will send an "earliest" request in background.
> However, after "seekToEnd" is called, due to a race condition in
> "Fetcher.resetOffsetIfNeeded" (It's not atomic, "seekToEnd" could happen
> between the check
> https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R585
> and the seek
> https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R605),
> "KafkaConsumer.position" may return an "earliest" offset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
