philipnee opened a new pull request, #12580:
URL: https://github.com/apache/kafka/pull/12580
We are observing flakiness in the OffsetValidationTest, and I believe
(evidently) it is due to not handling the uncommitted offset during the
rebalancing. Here the fix is very simple, commit the offset when the
onPartitionRevoke is invoked.
Here are the findings of this observation:
**Error:**
AssertionError: Total consumed records 3813 did not match consumed position
3809
Of the 3 consumers it spawned, only 2 were actually consuming data (maybe it
is just this specific run), and the largest offset was actually 3809.
```
{"timestamp":1661999681665,"name":"records_consumed","count":1,"partitions":[{"topic":"test_topic","partition":0,"count":1,"minOffset":3808,"maxOffset":3808}]}
{"timestamp":1661999681680,"name":"partitions_revoked","partitions":[{"topic":"test_topic","partition":0}]}
{"timestamp":1661999681693,"name":"shutdown_complete"}
```
Also, during the rebalancing, turned out, one of the consumer re-consume
some of the already consumed reacord
```
819
{"timestamp":1661999681453,"name":"records_consumed","count":1,"partitions":[{"topic":"test_topic","partition":0,"count":1,"minOffset":3703,"maxOffset":3703}]}
820
{"timestamp":1661999681454,"name":"partitions_revoked","partitions":[{"topic":"test_topic","partition":0}]}
821
{"timestamp":1661999681465,"name":"partitions_assigned","partitions":[{"topic":"test_topic","partition":0}]}
822
{"timestamp":1661999681471,"name":"records_consumed","count":12,"partitions":[{"topic":"test_topic","partition":0,"count":12,"minOffset":3700,"maxOffset":3711}]}
```
(sorry for the vim line number)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
