Hi Taher, Could you explain a bit more your use case and what do you expect Flink SQL to support? That could help us to better understand and plan the future roadmap.
Best, Jark On Wed, 5 May 2021 at 19:42, Taher Koitawala <taher...@gmail.com> wrote: > Thank you for the reply Jack Wu, however that does not satisfy my > requirements, my use case is to have something that supports a schema drift > over avro format. Column addition and column datatype change both types of > variations is what I am trying to solve for. Either way thanks for > the help, much appreciated. > > Regards, > Taher Koitawala > > On Wed, May 5, 2021 at 3:53 PM Jark Wu <imj...@gmail.com> wrote: > > > Hi Taher, > > > > Currently, Flink (SQL) CDC doesn't support automatic schema change > > and doesn't support to consume schema change events in source. > > But you can upgrade schema manually, for example, if you have a table > > with columns [a, b, c], you can define a flink table t1 with these 3 > > columns. > > When you add new column in source RDBMS, the Flink SQL job on t1 > > should work fine if you are using format 'debezium-json' or > > 'debezium-avro-confluent', > > because they supports schema compatibility. > > When you are notified there is a schema change in the source RDBMS, > > then you can upgrade your Flink SQL DDL and job to include the added > > column, > > and consume from the previous savepoint [1]. > > > > Best, > > Jark > > > > [1]: > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sqlclient/#start-a-sql-job-from-a-savepoint > > > > > > > > On Wed, 5 May 2021 at 13:34, Taher Koitawala <taher...@gmail.com> wrote: > > > > > Hi All, > > > I have a CDC use case where I want to capture and process > > debezium > > > logs that are streamed to Kafka via Debezium. As per all the flink > > examples > > > we have to pre create the schema of the tables where I want to perform > a > > > write. > > > > > > However my question is what if there is an alter table modify column > data > > > type query that hits the source RDBMS, how does flink handle that > schema > > > change and what changes are supported. If someone can give a full > example > > > it will be very very helpful. > > > > > > > > > Regards, > > > Taher Koitawala > > > > > >
