[ 
https://issues.apache.org/jira/browse/KAFKA-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788096#comment-16788096
 ] 

ASF GitHub Bot commented on KAFKA-7703:
---------------------------------------

viktorsomogyi commented on pull request #6407: KAFKA-7703: position() may 
return a wrong offset after seekToEnd
URL: https://github.com/apache/kafka/pull/6407
 
 
   When poll is called which resets the offsets to the beginning, followed by a 
seekToEnd and a position, it could happen that the "reset to earliest" call in 
poll overrides the "reset to latest" initiated by seekToEnd in a very delicate 
way: 
   1. both request has been issued and returned to the client side 
(listOffsetResponse has happened)
   2. in Fetcher.resetOffsetIfNeeded(TopicPartition, Long, OffsetData) the 
thread scheduler could prefer the heartbeat thread with the "reset to earliest" 
call, overriding the offset to the earliest and setting the SubscriptionState 
with that position.
   3. The thread scheduler continues execution of the thread (application 
thread) with the "reset to latest" call and discards it as the "reset to 
earliest" already set the position - the wrong one.
   4. The blocking position call returns with the earliest offset instead of 
the latest, despite it wasn't expected.
   
   The fix makes the TopicPartitionState in SubscriptionState synchronized and 
starts to track the requested reset timestamp. With this we can precisely 
decide if the incoming offset reset is really what we want. Therefore the 
latest initiated offset reset will happen only. Synchronization furthermore 
ensures that this is done in an atomic manner to avoid further similar bugs.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> KafkaConsumer.position may return a wrong offset after "seekToEnd" is called
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-7703
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7703
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0
>            Reporter: Shixiong Zhu
>            Assignee: Viktor Somogyi-Vass
>            Priority: Major
>
> After "seekToEnd" is called, "KafkaConsumer.position" may return a wrong 
> offset set by another reset request.
> Here is a reproducer: 
> https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246
> In this reproducer, "poll(0)" will send an "earliest" request in background. 
> However, after "seekToEnd" is called, due to a race condition in 
> "Fetcher.resetOffsetIfNeeded" (It's not atomic, "seekToEnd" could happen 
> between the check 
> https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R585
>  and the seek 
> https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R605),
>  "KafkaConsumer.position" may return an "earliest" offset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to