spoorthibasu opened a new pull request, #4553: URL: https://github.com/apache/flink-cdc/pull/4553
## What is the purpose of this pull request? The MongoDB CDC connector keeps the snapshot split metadata (assigned splits, finished offsets, table schemas) in the source coordinator for the whole lifetime of the job and re-serializes it on every checkpoint. On a job with a very large number of snapshot splits this keeps JobManager memory high and inflates checkpoint size. FLINK-40697 added the release to the incremental source framework (flink-cdc-base) behind `scan.incremental.snapshot.metadata.release.enabled`. The four JDBC dialects inherit the option from `BaseSourceConfig` and its validation from `JdbcSourceConfig`. MongoDB is built on the same framework but implements `SourceConfig` directly, so it inherits neither and the option cannot be enabled there. This PR adds it, which makes MongoDB the last connector built on flink-cdc-base to get the option. The option stays opt-in and defaults to `false`, so existing jobs are unaffected. Unlike the JDBC dialects, MongoDB defaults `scan.incremental.snapshot.enabled` to `false`, so the release applies only when incremental snapshot is also enabled. ## Brief change log - Add the `releaseSnapshotMetadataEnabled` field, getter and factory setter to `MongoDBSourceConfig` and `MongoDBSourceConfigFactory` - Fail fast when the option is enabled together with `scan.newly-added-table.enabled`, which MongoDB supports; the check is implemented here because MongoDB does not extend `JdbcSourceConfig` - Expose the option through `MongoDBSourceBuilder`, and through Flink SQL in `MongoDBTableSourceFactory` and `MongoDBTableSource` - Document the option (EN and ZH), including the downgrade restriction --- ## Verifying this change This change added tests and can be verified as follows: - Unit tests in `MongoDBSourceConfigFactoryTest` for the default, for the setter reaching the config, and for the fail-fast on both options - A unit test in `MongoDBTableFactoryTest` that sets the option in the DDL and asserts it reaches `MongoDBTableSource` - Integration tests on a real MongoDB container in `MongoDBParallelSourceITCase` covering the group-fetch release path plus JobManager and TaskManager failover in the stream phase, verifying no snapshot or stream data is lost through the release and restore ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs The Chinese doc entry mirrors the English and is a best-effort translation. Please correct the wording where needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
