Re: [DISCUSS] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

2020-04-10 Thread Jark Wu
Hi, After a short offline discussion with Kurt, It seems that I misunderstood Kurt's meaning. Kurt meant: is `format=debezium` is enough, or split into two options `format=debezium` and `format.encoding=json`. Debezium not only support JSON encoding, but also Avro. Canal supports JSON and Protobu

Re: [DISCUSS] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

2020-04-08 Thread Jark Wu
Hi Kurt, The JSON encoding of Debezium can be configured to include or exclude the message schema using the `value.converter.schemas.enable` properties. That's why we propose to have a `format.schema-include` property key to config how to parse the json. Besides, the encoding format of debezium i

Re: [DISCUSS] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

2020-04-07 Thread Kurt Young
One minor comment, is there any other encoding or format in debezium? I'm asking because the format name is debezium-json, i'm wondering whether debezium is enough. This also applies to canal. Best, Kurt On Tue, Apr 7, 2020 at 11:49 AM Jark Wu wrote: > Hi everyone, > > Since this FLIP was prop

Re: [DISCUSS] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

2020-04-06 Thread Jark Wu
Hi everyone, Since this FLIP was proposed, the community has discussed a lot about the first approach: introducing new TableSource and TableSink interfaces to support changelog. And yes, that is FLIP-95 which has been accepted last week. So most of the work has been merged into FLIP-95. In order

[DISCUSS] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

2020-02-14 Thread Jark Wu
Hi everyone, I would like to start discussion about how to support interpreting external changelog into Flink SQL, and how to emit changelog from Flink SQL. This topic has already been mentioned several times in the past. CDC (Change Data Capture) data has been a very important streaming data in