In case this helps, here are the versions I was working with when I was experimenting with this last year:
https://github.com/ottomata/flink-cdc-spike/blob/main/dockerfiles/build_env_1.17.env https://github.com/ottomata/flink-cdc-spike/blob/main/dockerfiles/download_dependencies.sh I was trying many things though (flink-cdc, paimon action, etc.) so YMMV. Good luck! On Tue, Mar 11, 2025 at 9:14 AM Taher Koitawala <taher...@gmail.com> wrote: > Hi All, > I am facing issues finding a class > > Org.apache.flink.cdc.connectors.shaded.org.apache.kafka.connect.json.JsonConverter > > > I have added > 1. Flink-connector-base 1.18.1 > Flink-connector-debezium > 3.1.0 > Flink-cdc-pipeline-connectors-values > 3.1.0 > Flink-cdc-base > 3.1.0 > Flink-cdc-pipeline-connector-kafka 3.1.0 > > > What am i missing? > > > On Fri, 7 Mar 2025 at 8:13 AM, Taher Koitawala <taher...@gmail.com> wrote: > >> Hi Leonard, >> Yes i did see Xianqian’s reply however i thought my email did not go >> through as the community is often very active but I did not receive a >> response until Xianqian’s reply. >> >> Thank you Xianqian we are currently trying out the suggestions for >> altering schema via the catalog. >> >> Thank you Andrew Otto for the docs as i will I will take a look at this. >> >> >> >> >> On Wed, 5 Mar 2025 at 5:47 AM, Leonard Xu <xbjt...@gmail.com> wrote: >> >>> Hey Taher, >>> >>> Xianqian has replied your email, did you subscribe Flink dev >>> mailing-list? >>> >>> Best, >>> Leonard >>> >>> > 2025年2月11日 16:52，Xiqian YU <kono....@outlook.com> 写道： >>> > >>> > Hi Taher, >>> > >>> > Since we’re creating a DataStream-based pipeline job with SQL Server >>> CDC, schema change events must be handled manually. A possible approach >>> would be: >>> > >>> > 1) Enable schema change events with `.includeSchemaChanges(true)` >>> option, so DDL events will be parsed and encoded in `SourceRecord`s. >>> > >>> > 2) Write a customized `DebeziumDeserializationSchema` class and parse >>> schema change events. >>> `MySqlEventDeserializer#deserializeSchemaChangeRecord` could be used as a >>> reference [1]. >>> > >>> > 3) Evolve sink schema of Paimon tables with `PaimonCatalog` manually. >>> `PaimonMetadataApplier` [2] is an existing schema evolving implementation >>> supporting a few frequently used schema change events. >>> > >>> > Also, CDC Pipeline framework [3] has provided a fully-automatic schema >>> sensing and evolving solution, but unfortunately Microsoft SQL Server >>> source is not supported yet until we close #3445 [4] or #3507 [5]. >>> > >>> > [1] >>> https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/MySqlEventDeserializer.java >>> > [2] >>> https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java >>> > [3] >>> https://nightlies.apache.org/flink/flink-cdc-docs-release-3.3/docs/core-concept/data-pipeline/ >>> > [4] https://github.com/apache/flink-cdc/pull/3445 >>> > [5] https://github.com/apache/flink-cdc/pull/3507 >>> > >>> > Best Regards, >>> > Xiqian >>> > >>> > Taher Koitawala <taher...@gmail.com> 於 2025年2月11日 15:59 寫道： >>> > >>> > Hi Devs, >>> > As a POC we are trying to create a steaming pipeline from MSSQL >>> cdc >>> > to Paimon: >>> > >>> > To do this we are doing >>> > 1. msSql server cdc operator >>> > 2. Transform operator >>> > 3. paimon sink >>> > >>> > We have written the cdc connector with is a >>> JsonDebeziumDeserialisedSchema >>> > String >>> > >>> > I wish to write this paimon in a table format with same columns as >>> source. >>> > >>> > As far as i know paimon automatically handles schema updates like new >>> field >>> > additions. >>> > >>> > Please can someone point me on how to write this stream efficiently to >>> > paimon table with schema updates? >>> > >>> > For now i have SouceFunction<String> >>> > >>> > Which is the record mentioned above! >>> > >>> > Regards, >>> > Taher Koitawala >>> > >>> >>>
