Hi all, While working on the bytes-based version of KIP-1325 with Murali, I looked at where the restore buffer actually grows.
In StoreChangelogReader, records that can be applied are applied right after they are polled. Whether the buffer grows depends on the task type and on which topic the store uses as its changelog: - Active task, own changelog topic: it restores up to the end offset and applies every record it polls. The buffer does not grow. - Active task, input topic as changelog: it restores up to the committed offset and then finishes restoring. The buffer does not grow. - Standby task, own changelog topic: there is no limit, so it applies every record it polls. The buffer does not grow. - Standby task, input topic as changelog: it may only apply records up to the offset committed by the active task, but it keeps fetching beyond it. The buffer grows with every poll. This is the only case that can lead to an OutOfMemoryError, for example during a backfill. A store uses the input topic as its changelog when it comes from builder.table() and topology.optimization is set to "all" or "reuse.ktable.source.topics". Before 2.5.0, Streams paused such a standby partition. This was lost in the refactoring of KAFKA-9113. I opened KAFKA-21276 and a PR to restore that behavior, with no new config: https://github.com/apache/kafka/pull/23815 If this is the only case where the restore buffer grows, the limit in KIP-1325 would never be reached after the fix. So I'd like to ask: do we still need a configurable limit for the restore buffer, for example as a safety net together with KIP-770, or is the fix enough? Karsten has since shared his configs on KAFKA-19992. He builds the topology with topology.optimization=all and num.standby.replicas=1, and a heap dump from his case showed StoreChangelogReader#changelogs holding more than 80% of the heap. So this looks like his case. I also reproduced it with a Debezium setup similar to his (600k records of 2 KB, -Xmx1g): with topology.optimization=all the standby ran out of memory in all runs on trunk, and in none with the fix. Thanks, Eric
