I'm trying to understand the interactions between stateful processing in Kafka Streams and topic retention size restrictions. I'm fairly naive about both Kafka and Kafka Streams so I'm sure I have something wrong here, but my understanding is that, under stateful processing, local state in the application is made fault tolerant by the topic's commit log on the broker. That data, I assume, will count towards the topic's retention.bytes limit. If so, then what happens when I hit that limit? Let's say that the topic has a retention limit of 10mb and my stateful processor needs more than that to do its work. Does the processor fail? And what impact is there on retaining messages? Does the amount of messages stored in the topic trend toward 0 as the allocated space is used up by the processor?

Pointers to documentation that would explain this would also be helpful. I've tried to answer this question for myself but with no luck so far. Thank you very much.

Reply via email to