Pointers to documentation that would explain this would also be helpful. I've tried to answer this question for myself but with no luck so far. Thank you very much.
I'm trying to understand the interactions between stateful processing in
Kafka Streams and topic retention size restrictions. I'm fairly naive
about both Kafka and Kafka Streams so I'm sure I have something wrong
here, but my understanding is that, under stateful processing, local
state in the application is made fault tolerant by the topic's commit
log on the broker. That data, I assume, will count towards the topic's
retention.bytes limit. If so, then what happens when I hit that limit?
Let's say that the topic has a retention limit of 10mb and my stateful
processor needs more than that to do its work. Does the processor fail?
And what impact is there on retaining messages? Does the amount of
messages stored in the topic trend toward 0 as the allocated space is
used up by the processor?
- Kafka Streams and retention limits Dan Bradley
- Re: Kafka Streams and retention limits Matthias J. Sax
- Re: Kafka Streams and retention limits Bruno Cadonna