Hi Frank, There is a similar discussion here https://www.mail-archive.com/users@kafka.apache.org/msg25089.html <https://www.mail-archive.com/users@kafka.apache.org/msg25089.html>, with a JIRA.
Have you tried to increase the retention minutes to something large? Thanks Eno > On 21 Mar 2017, at 19:13, Frank Lyaruu <flya...@gmail.com> wrote: > > Hi Kafka people, > > We have a Kafka Streams application that replicates a database, and > transforms it to a different data model. Some tables/topics move fast, with > many changes a second, some might be dormant for months. For those slow > moving topics, we have some trouble with the 'offsets.retention.minutes'. > After that timeout it's offset is effectively invalid. As long as it keeps > running, everything is ok, but when we restart the application it will fall > back to 'earliest' and redo the entire topic, which is obviously wrong. > > How can we fix this or work around this? Our first idea was to enable a > very slow auto commit, so the offset.retention.minutes never times out, but > Kafka Streams seems to explicitly forbid that. The StreamConfig javadoc > says: ""Furthermore, it is not allowed to enable "enable.auto.commit" that > is disabled by Kafka Streams by default." > > So how do we keep those commits alive? Why doesn't Kafka Streams allow a > slow auto commit to prevent this? Any other ideas on how to circumvent this > problem? > > regards, Frank > > We're running the trunk version of last week.
