spoorthibasu commented on PR #4418: URL: https://github.com/apache/flink-cdc/pull/4418#issuecomment-4963479929
> So I wonder whether the reported OOM happened while the snapshot metadata was still growing, or after the job had already entered the binlog phase. In the latter case, I agree that keeping and repeatedly serializing such a large state is still wasteful and can keep memory pressure high, but it would be helpful to clarify why it still leads to OOM after the state size has become stable, e.g. checkpoint overlap, delayed GC, direct memory/container RSS pressure, etc. It's the second case you described, after the job has entered the binlog phase rather than while the snapshot state is still growing. The finished-split count is stable by then, so later checkpoints carry a roughly constant state size. It's the direct memory / container RSS pressure you listed. Even with the state size stable, every post-snapshot checkpoint still re-serializes and transfers the full metadata, around 300K finished splits for a 2.5B-row table at the default chunk size. That can create several transient copies in a short window, the serialized state bytes plus the off-heap buffers used during transfer. Since Flink doesn't set a direct-memory limit for the JobManager by default (`jobmanager.memory.enable-jvm-direct-memory-limit=false`), that off-heap pressure isn't bounded early by a JVM-level direct-memory error and can instead build up in process RSS until the container is OOMKilled, even while the heap looks fine. This was reported on CDC 3.5.0 / Flink 1.20.x with a Paimon sink. So it's exactly the repeated serialization of the large stable state you flagged as wasteful. Happy to add a note on this to the PR description if that would help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
