Hi Eric I’m totally convinced by your sleuthing, and I’ve left some comments on your PR.
We could revisit KIP-1325 with your finding once the PR gets merged Best, Chia-Ping On 2026-10-16 02:30 UTC Eric Chang wrote: > Hi all, > > While working on the bytes-based version of KIP-1325 with Murali, I looked > at where the restore buffer actually grows. > > In StoreChangelogReader, records that can be applied are applied right after > they are polled. Whether the buffer grows depends on the task type and on > which topic the store uses as its changelog: > > - Active task, own changelog topic: it restores up to the end offset and > applies every record it polls. The buffer does not grow. > - Active task, input topic as changelog: it restores up to the committed > offset and then finishes restoring. The buffer does not grow. > - Standby task, own changelog topic: there is no limit, so it applies every > record it polls. The buffer does not grow. > - Standby task, input topic as changelog: it may only apply records up to > the offset committed by the active task, but it keeps fetching beyond it. > The buffer grows with every poll. This is the only case that can lead to > an OutOfMemoryError, for example during a backfill. > > A store uses the input topic as its changelog when it comes from > builder.table() and topology.optimization is set to "all" or > "reuse.ktable.source.topics". > > Before 2.5.0, Streams paused such a standby partition. This was lost in the > refactoring of KAFKA-9113. I opened KAFKA-21276 and a PR to restore that > behavior, with no new config: > https://github.com/apache/kafka/pull/23815 > > If this is the only case where the restore buffer grows, the limit in > KIP-1325 would never be reached after the fix. So I'd like to ask: do we > still need a configurable limit for the restore buffer, for example as a > safety net together with KIP-770, or is the fix enough? > > Karsten has since shared his configs on KAFKA-19992. He builds the topology > with topology.optimization=all and num.standby.replicas=1, and a heap dump > from his case showed StoreChangelogReader#changelogs holding more than 80% > of the heap. So this looks like his case. I also reproduced it with a > Debezium setup similar to his (600k records of 2 KB, -Xmx1g): with > topology.optimization=all the standby ran out of memory in all runs on > trunk, and in none with the fix. > > Thanks, > Eric >
