Hi Kafka people, We have a Kafka Streams application that replicates a database, and transforms it to a different data model. Some tables/topics move fast, with many changes a second, some might be dormant for months. For those slow moving topics, we have some trouble with the 'offsets.retention.minutes'. After that timeout it's offset is effectively invalid. As long as it keeps running, everything is ok, but when we restart the application it will fall back to 'earliest' and redo the entire topic, which is obviously wrong.
How can we fix this or work around this? Our first idea was to enable a very slow auto commit, so the offset.retention.minutes never times out, but Kafka Streams seems to explicitly forbid that. The StreamConfig javadoc says: ""Furthermore, it is not allowed to enable "enable.auto.commit" that is disabled by Kafka Streams by default." So how do we keep those commits alive? Why doesn't Kafka Streams allow a slow auto commit to prevent this? Any other ideas on how to circumvent this problem? regards, Frank We're running the trunk version of last week.
