Hi, Rex. Part of what enabled CDC support in Flink 1.11 was the refactoring of the table source interfaces (FLIP-95 [1]), and the new ScanTableSource [2], which allows to emit bounded/unbounded streams with insert, update and delete rows.
In theory, you could consume data generated with Debezium as regular JSON-encoded events before Flink 1.11 — there just wasn't a convenient way to really treat it as "changelog". As a workaround, what you can do in Flink 1.10 is process these messages as JSON and extract the "after" field from the payload, and then apply de-duplication [3] to keep only the last row. The DDL for your source table would look something like: CREATE TABLE tablename ( *... * after ROW(`field1` DATATYPE, `field2` DATATYPE, ...) ) WITH ( 'connector' = 'kafka', 'format' = 'json', ... ); Hope this helps! Marta [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces [2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/table/connector/source/ScanTableSource.html [3] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html#deduplication On Fri, Aug 21, 2020 at 10:28 AM Chesnay Schepler <ches...@apache.org> wrote: > @Jark Would it be possible to use the 1.11 debezium support in 1.10? > > On 20/08/2020 19:59, Rex Fenley wrote: > > Hi, > > I'm trying to set up Flink with Debezium CDC Connector on AWS EMR, > however, EMR only supports Flink 1.10.0, whereas Debezium Connector arrived > in Flink 1.11.0, from looking at the documentation. > > https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html > > https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/formats/debezium.html > > I'm wondering what alternative solutions are available for connecting > Debezium to Flink? Is there an open source Debezium connector that works > with Flink 1.10.0? Could I potentially pull the code out for the 1.11.0 > Debezium connector and compile it in my project using Flink 1.10.0 api? > > For context, I plan on doing some fairly complicated long lived stateful > joins / materialization using the Table API over data ingested from > Postgres and possibly MySQL. > > Appreciate any help, thanks! > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | > FOLLOW US <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> > > >
