Matthias J. Sax created KAFKA-19236: ---------------------------------------
Summary: Auto.offset.reset "by-duration" should reset a single time Key: KAFKA-19236 URL: https://issues.apache.org/jira/browse/KAFKA-19236 Project: Kafka Issue Type: Improvement Components: clients, consumer, streams Affects Versions: 4.0.0 Reporter: Matthias J. Sax KIP-1106 introduced a new option "by-duration" for config `auto.offset.reset` ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-1106%3A+Add+duration+based+offset+reset+option+for+consumer+clients)] If a consumer tries to reset to a "future" time, the observed behavior is somewhat odd, and we should change it: Assume there is a topic for which no new data was written for the last hour. A new consumer starts up at 1pm, and tries to reset by 10 minutes. There is no data for 12:50pm, so the consumer won't complete the reset, but will keep retrying (every 30 seconds by default) until it can resolve offsets – the issue is, that the "seek ts" (ie 12:50pm) is recomputed on every retry and thus move while the consumer wait. Because the consumer could not resolve offset, but still has `offsets=null`, it keep re-executing the reset logic. Hence, if there is not data for another 30 minutes, the consumer would now retry to find offset at 1:20pm. This is rather unexpected, as if one starts the consumer at 1:00pm and resets by 10 minutes, it's reasonable to assume that data would start flowing when the topic reaches 12:50pm, even if the consumer was idling for 30 minutes. Thus, instead of executing the reset logic every 30 seconds, the reset logic should be executed once, to compute 12:50pm as "seek ts", and if the request returns `null` (ie, no offset found), the same request should be resent every 30 second, w/o re-triggering the rest logic itself, to keep the "seek ts" at 12:50pm. Kafka Streams, which re-implement the by-duration reset logic by itself, has the same behavior as the consumer, and should be updated, as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)