Hi Taher, Since we’re creating a DataStream-based pipeline job with SQL Server CDC, schema change events must be handled manually. A possible approach would be:
1) Enable schema change events with `.includeSchemaChanges(true)` option, so DDL events will be parsed and encoded in `SourceRecord`s. 2) Write a customized `DebeziumDeserializationSchema` class and parse schema change events. `MySqlEventDeserializer#deserializeSchemaChangeRecord` could be used as a reference [1]. 3) Evolve sink schema of Paimon tables with `PaimonCatalog` manually. `PaimonMetadataApplier` [2] is an existing schema evolving implementation supporting a few frequently used schema change events. Also, CDC Pipeline framework [3] has provided a fully-automatic schema sensing and evolving solution, but unfortunately Microsoft SQL Server source is not supported yet until we close #3445 [4] or #3507 [5]. [1] https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/MySqlEventDeserializer.java [2] https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java [3] https://nightlies.apache.org/flink/flink-cdc-docs-release-3.3/docs/core-concept/data-pipeline/ [4] https://github.com/apache/flink-cdc/pull/3445 [5] https://github.com/apache/flink-cdc/pull/3507 Best Regards, Xiqian Taher Koitawala <taher...@gmail.com> 於 2025年2月11日 15:59 寫道： Hi Devs, As a POC we are trying to create a steaming pipeline from MSSQL cdc to Paimon: To do this we are doing 1. msSql server cdc operator 2. Transform operator 3. paimon sink We have written the cdc connector with is a JsonDebeziumDeserialisedSchema String I wish to write this paimon in a table format with same columns as source. As far as i know paimon automatically handles schema updates like new field additions. Please can someone point me on how to write this stream efficiently to paimon table with schema updates? For now i have SouceFunction<String> Which is the record mentioned above! Regards, Taher Koitawala
