Hi everyone, Since this FLIP was proposed, the community has discussed a lot about the first approach: introducing new TableSource and TableSink interfaces to support changelog. And yes, that is FLIP-95 which has been accepted last week. So most of the work has been merged into FLIP-95.
In order to support the goal of FLIP-105, there is still a little things to discuss: how to connect external CDC formats. We propose to introduce 2 new formats: Debezium format and Canal format. They are the most popular CDC tools according to the survey in user [1] and user-zh [2] mailing list. I have updated the FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL Welcome feedbacks! Best, Jark [1]: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SURVEY-What-Change-Data-Capture-tools-are-you-using-td33569.html [2]: http://apache-flink.147419.n8.nabble.com/SURVEY-CDC-td1910.html On Fri, 14 Feb 2020 at 22:08, Jark Wu <imj...@gmail.com> wrote: > Hi everyone, > > I would like to start discussion about how to support interpreting > external changelog into Flink SQL, and how to emit changelog from Flink SQL. > > This topic has already been mentioned several times in the past. CDC > (Change Data Capture) data has been a very important streaming data in the > world. Connect to CDC is a significant feature for Flink, it fills the > missing piece for Flink's streaming processing. > > In FLIP-105, we propose 2 approaches to achieve. > One is introducing new TableSource interface (higher priority), > the other is introducing new SQL syntax to interpret and emit changelog. > > FLIP-105: > https://docs.google.com/document/d/1onyIUUdWAHfr_Yd5nZOE7SOExBc6TiW5C4LiL5FrjtQ/edit# > > Thanks for any feedback! > > Best, > Jark >
