Spoorthi Basu created FLINK-40732:
-------------------------------------
Summary: Release finished snapshot split metadata in the MongoDB
CDC connector
Key: FLINK-40732
URL: https://issues.apache.org/jira/browse/FLINK-40732
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: Spoorthi Basu
The MongoDB CDC connector keeps the snapshot split metadata (assigned splits,
finished offsets and table schemas) in the source coordinator for the whole
lifetime of the job, and rebuilds it on every checkpoint. On a job with a very
large number of snapshot splits this keeps JobManager memory high and inflates
checkpoint size.
That metadata is released by the incremental source framework
({{flink-cdc-base}}) once the source enters the stream phase, behind
{{scan.incremental.snapshot.metadata.release.enabled}} (FLINK-40697). The four
JDBC dialects inherit the option from {{BaseSourceConfig}} and its validation
from {{JdbcSourceConfig}}. MongoDB implements {{SourceConfig}} directly, so it
inherits neither and MongoDB jobs cannot enable the option.
This proposes adding it to the MongoDB connector: the config plumbing, the
fail-fast against {{scan.newly-added-table.enabled}}, and the option itself via
both the DataStream builder and Flink SQL.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
