[
https://issues.apache.org/jira/browse/KAFKA-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642299#comment-17642299
]
Chetan commented on KAFKA-14366:
--------------------------------
Hi [~pnee] ,
The reason behind having it as the earliest is we don't want to lose any of the
messages. and when the consumer restarts/rebalances, the consumer can start
reading from the last committed message.
Are you suspecting/relating this with manual commit over in our code?
Regards,
Chetan N
> Kafka consumer rebalance issue, offsets points back to very old committed
> offset
> --------------------------------------------------------------------------------
>
> Key: KAFKA-14366
> URL: https://issues.apache.org/jira/browse/KAFKA-14366
> Project: Kafka
> Issue Type: Bug
> Components: consumer, offset manager
> Affects Versions: 2.8.1
> Environment: Production
> Reporter: Chetan
> Priority: Major
> Attachments: image-2022-11-30-17-58-15-046.png,
> image-2022-11-30-18-10-37-180.png, image-2022-11-30-18-29-36-445.png,
> rebalance issue.docx
>
>
> Hi All,
> We are facing an issue while the client consumer restart (again not all
> restarts are ending up with this issue) and during the re-balancing scenario,
> sometimes one of the partition offsets goes back a long way from the
> committed offset.
> Scenario :
> Assume we have 4 instances of consumer and restarts of consumer one after the
> other.
> # At the time of starting restarts assume the offset on partition 10 of a
> topic being consumed is pointing to 50000. (last offset of the topic and 0
> lag)
> # When restarts start (rebalancing) suddenly the offsets start pointing to
> 20000.
> # While all the restarts are going on the consumer who is attached starts
> reading from 20000 and goes on.
> # Once all rebalance is completed, and all messages from 20000 to 50000
> offset has been read (where it had stopped initially)
> We end up having around 30K duplicates.
> (The numbers here are just an example, in production, we are facing huge
> duplicates and every two rebalance during restarts of consumer out of 10
> restart exercise activity ends up in such duplicates and not all partitions
> and only one or two partitions behave this way and randomly)
> This seems to be a bug. I am attaching all screenshots for reference as well.
> Can someone kindly help out here?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
