Re: Commits of slow moving topics in Kafka Streams time out

2017-03-22 Thread Frank Lyaruu
Ok, yes increasing the retention minutes to something silly is indeed what we intend to do, and for now, that should work fine as we don't use many short lived consumers. For the long term, the change in semantics in the link you provide looks good to us. regards, Frank On Tue, Mar 21, 2017 at 8

Re: Commits of slow moving topics in Kafka Streams time out

2017-03-21 Thread Eno Thereska
Hi Frank, There is a similar discussion here https://www.mail-archive.com/users@kafka.apache.org/msg25089.html , with a JIRA. Have you tried to increase the retention minutes to something large? Thanks Eno > On 21 Mar 2017, at

Commits of slow moving topics in Kafka Streams time out

2017-03-21 Thread Frank Lyaruu
Hi Kafka people, We have a Kafka Streams application that replicates a database, and transforms it to a different data model. Some tables/topics move fast, with many changes a second, some might be dormant for months. For those slow moving topics, we have some trouble with the 'offsets.retention.m