Hi! This isn't exactly what you are looking for, but we have been looking into some of these things at WMF too. See John Mwangi's experiment project and writeup: https://github.com/john-mwangi/mariadb-iceberg-pipeline/blob/main/WRITEUP.md
And the attached task with details in comments. T373144 [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally <https://phabricator.wikimedia.org/T373144> Maybe there will be some insights you can use there? Good luck! On Tue, Mar 4, 2025 at 9:44 AM Taher Koitawala <taher...@gmail.com> wrote: > Hi users, please help me here > ---------- Forwarded message --------- > From: Taher Koitawala <taher...@gmail.com> > Date: Tue, 11 Feb 2025 at 11:59 AM > Subject: Flink CDC to Paimon > To: Selvam Ragav <selvam.ra...@gmail.com>, <d...@flink.apache.org> > > > Hi Devs, > As a POC we are trying to create a steaming pipeline from MSSQL cdc > to Paimon: > > To do this we are doing > 1. msSql server cdc operator > 2. Transform operator > 3. paimon sink > > We have written the cdc connector with is a JsonDebeziumDeserialisedSchema > String > > I wish to write this paimon in a table format with same columns as source. > > As far as i know paimon automatically handles schema updates like new > field additions. > > Please can someone point me on how to write this stream efficiently to > paimon table with schema updates? > > For now i have SouceFunction<String> > > Which is the record mentioned above! > > Regards, > Taher Koitawala > > > > >
