Re: Kafka Streams and retention limits

2021-04-26 Thread Bruno Cadonna
Additionally, with KIP-698 (https://cwiki.apache.org/confluence/x/7CnZCQ), we will add verifications of changelogs to find misconfigurations and report them to the users. Best, Bruno On 24.04.21 22:58, Matthias J. Sax wrote: The topics used by Kafka Streams to backup state stores, are calle

Re: Kafka Streams and retention limits

2021-04-24 Thread Matthias J. Sax
The topics used by Kafka Streams to backup state stores, are called changelog topic, and they are configures with log compaction instead of retention, and thus, there is no size limit. Cf https://kafka.apache.org/documentation/#compaction -Matthias On 4/24/21 1:32 PM, Dan Bradley wrote: > I'm t

Kafka Streams and retention limits

2021-04-24 Thread Dan Bradley
I'm trying to understand the interactions between stateful processing in Kafka Streams and topic retention size restrictions. I'm fairly naive about both Kafka and Kafka Streams so I'm sure I have something wrong here, but my understanding is that, under stateful processing, local state in the