Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-21 Thread OpenInx
> Could you list some needs from your point of view ? I know the typical requirements about extending the native flink sql for iceberg tables are: 1. There are many table management operations in apache iceberg which can be friendly if we can express in SQL for end users. such as: expire iceber

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-21 Thread zhou chao
Hi, Openlnx, FG and Jingsong, thanks for your attention about this issue. I am sorry for the late reply, I’ m working on the further job about the implement of merge statement, and now we support this in our internal for the basic SQL syntax. About: Implement our flink sql plugin/extensions whi

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-17 Thread Jingsong Li
Hi zoucao, Thanks for your proposal. I believe this discussion will take us one step further for iceberg/hudi integration. ## `MERGE` for streaming I feel that `MERGE` is very good for stream computing. And, this is where our Flink can have an advantage over other computation systems. MERGE INT

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-16 Thread Francesco Guardiani
> In the theory aspect, incremental data should be carefully considered for streaming data. In this situation, the data flow from target_table to target_table will be a loop, and the incremental data with one key will keep going through the loop. It looks very strange. This is the same concern I

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-13 Thread OpenInx
I'm currently maintaining the iceberg flink modules from apache iceberg community. Currently, the spark has a great integration experience with iceberg format v2 in batch mode. In this document [1], The merge into syntax from spark sql extensions does really help a lot when people want to change

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-11 Thread zhou chao
Hi, Martijn Visser, thanks for your reply. Firstly, I am sorry for posting the discussion twice. I sent the message to the dev mail group from an unsub- scribed account, but the message was not shown for a while, and I guessed that the dev mail group would not post an email coming from an unsubscr

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-10 Thread Martijn Visser
Hi zoucao, I see that this message was posted twice, so I choose to only reply to the latest one (this one). Thanks for bringing this up for discussion. I agree that support for a merge statement would be a welcome addition to Flink SQL for those that are using it for bounded jobs. How would you

[DISCUSS]Support the merge statement in FlinkSQL

2022-02-08 Thread zhou chao
Hi, devs! Jingfeng and I would like to start a discussion about the MERGE statement, and the discussion consists of two parts. In the first part, we want to explore and collect the cases and motivations of the MERGE statement users. In the second part, we want to find out the possibility for Fli

[DISCUSS]Support the merge statement in FlinkSQL

2022-02-08 Thread cao zou
Hi, devs! Jingfeng and I would like to start a discussion about the MERGE statement, and the discussion consists of two parts. In the first part, we want to explore and collect the cases and motivations of the MERGE statement users. In the second part, we want to find out the possibility for Fli