suntectec opened a new pull request, #4533: URL: https://github.com/apache/flink-cdc/pull/4533
## What is the purpose of this pull request? Add a DB2 pipeline connector so the YAML Pipeline API supports IBM DB2 as a source, in the same way the existing MySQL / Postgres / Oracle / SQL Server pipeline connectors do. Before this change DB2 was only reachable from the DataStream / SQL APIs via `flink-connector-db2-cdc`. JIRA: [FLINK-40508](https://issues.apache.org/jira/browse/FLINK-40508) ## Brief change log - New module `flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-db2`, registered in the pipeline-connectors `pom.xml`. - `Db2DataSourceFactory` (identifier `db2`) plus `Db2DataSource`, `Db2PipelineSource`, `Db2EventDeserializer`, `Db2MetadataAccessor`, `Db2PipelineRecordEmitter`, `Db2SchemaDataTypeInference`, `Db2SchemaUtils` and `Db2TypeUtils`. - `Db2DataSourceOptions` exposes the options the DB2 source connector already supports: `hostname`, `port`, `username`, `password`, `database`, `tables`, `tables.exclude`, `server-time-zone`, `scan.startup.mode` (`initial` / `latest-offset`), `schema-change.enabled`, `metadata.list`, `scan.incremental.snapshot.chunk.key-column`, `scan.incremental.snapshot.chunk.size`, `chunk-meta.group.size`, `chunk-key.even-distribution.factor.upper-bound`, `chunk-key.even-distribution.factor.lower-bound`, `scan.snapshot.fetch.size`, `connect.timeout`, `connection.pool.size`, `connect.max-retries`, `scan.incremental.snapshot.backfill.skip`, `scan.incremental.close-idle-reader.enabled` and `scan.incremental.snapshot.unbounded-chunk-first.enabled`. - The source connector itself is reused as-is through the incremental snapshot framework; no change to `flink-connector-db2-cdc`. - DB2 specific type mapping, notably `DECFLOAT(16)` → `DOUBLE`, `DECFLOAT(34)` → `DECIMAL(34, 0)`, `XML` / `SQLXML` / `CLOB` → `STRING`, `BLOB` / `BINARY` / `VARBINARY` → `BYTES`. - Captured table names are resolved as `schemaName.tableName`, and `latest-offset` maps to `StartupOptions.latest()` with `initial` as the default. - `.github/workflows/modules.py`: the new module is added to `MODULES_DB2`, so CI compiles and tests it on both Flink 1.20 and Flink 2.x. - `.github/labeler.yml`: adds a `db2-pipeline-connector` rule. - Documentation in both `docs/content` and `docs/content.zh`, including a row in each pipeline-connectors `overview.md`. One thing worth pointing out: `src/test/resources/db2_server/` is a byte-identical copy of the source connector's test resources. That is not accidental — `Db2TestBase#getFilePath` resolves the Docker build context with `Paths.get(url.toURI())`, which cannot read a resource out of the `test-jar`, so the files have to exist on the module's own test classpath as real files. The added `.gitattributes` keeps these container scripts at LF endings, which matters for contributors on Windows. ## Verifying this change This change added tests and can be verified as follows: - Unit tests in `Db2TypeUtilsTest` cover every supported DB2 column type plus the unsupported-type failure path. - Integration tests in `Db2DataSourceFactoryITCase` and `Db2PipelineITCase` run against a real DB2 instance through testcontainers, reusing `Db2TestBase` from the source connector's test-jar. - Verified locally with `mvn test` on Java 11 / Flink 1.20, and on CI in the `Source Unit Tests / test (…, db2, mongodb)` and `Source Unit Tests 2.x / test (…, 2.2.0, db2, mongodb)` jobs, which run `mvn verify` and therefore execute the ITCases against both Flink 1.20.3 and Flink 2.2.0. ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs — see `docs/content/docs/connectors/pipeline-connectors/db2.md` and its Chinese counterpart. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Qoder) <!-- Generated-by: Qoder --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
