[ https://issues.apache.org/jira/browse/KAFKA-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
TengYao Chi reassigned KAFKA-18641: ----------------------------------- Assignee: TengYao Chi > AsyncKafkaConsumer could lose records with auto offset commit > ------------------------------------------------------------- > > Key: KAFKA-18641 > URL: https://issues.apache.org/jira/browse/KAFKA-18641 > Project: Kafka > Issue Type: Bug > Components: clients, consumer > Affects Versions: 4.0.0 > Reporter: Jun Rao > Assignee: TengYao Chi > Priority: Blocker > > In the new AsyncKafkaConsumer, the application thread will keep updating the > auto commit timer through PollEvent. In the consumer network thread, once the > timer has expired, it generates an offset commit request with the current > offset position in subscriptions. However, at this point, the records before > that offset could just be polled from FetchBuffer, but not actually consumed > by the application. If the application dies immediately, those records may > never be consumed by the application since the offset could have been > committed. > The ClassicKafkaConsumer doesn't seem to have this problem. In each poll() > call, before fetching new records, it first calls ConsumerCoordinator.poll(), > which generates an OffsetCommitRequest with the current offset position in > subscriptions. Since this is done in the same application thread, it > guarantees that all records returned in the previous poll() have been > processed. The problem exists in AsyncKafkaConsumer because the polling of > new records and committing offsets are done in separate threads. > > This problem exists for async offset commit and auto offset commit during > rebalance in AsyncKafkaConsumer too. -- This message was sent by Atlassian Jira (v8.20.10#820010)