Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-21 Thread OpenInx
> Could you list some needs from your point of view ? I know the typical requirements about extending the native flink sql for iceberg tables are: 1. There are many table management operations in apache iceberg which can be friendly if we can express in SQL for end users. such as: expire iceber

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-21 Thread zhou chao
Hi, Openlnx, FG and Jingsong, thanks for your attention about this issue. I am sorry for the late reply, I’ m working on the further job about the implement of merge statement, and now we support this in our internal for the basic SQL syntax. About: Implement our flink sql plugin/extensions whi

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-17 Thread Jingsong Li
Hi zoucao, Thanks for your proposal. I believe this discussion will take us one step further for iceberg/hudi integration. ## `MERGE` for streaming I feel that `MERGE` is very good for stream computing. And, this is where our Flink can have an advantage over other computation systems. MERGE INT

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-16 Thread Francesco Guardiani
> In the theory aspect, incremental data should be carefully considered for streaming data. In this situation, the data flow from target_table to target_table will be a loop, and the incremental data with one key will keep going through the loop. It looks very strange. This is the same concern I

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-13 Thread OpenInx
I'm currently maintaining the iceberg flink modules from apache iceberg community. Currently, the spark has a great integration experience with iceberg format v2 in batch mode. In this document [1], The merge into syntax from spark sql extensions does really help a lot when people want to change

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-11 Thread zhou chao
Hi, Martijn Visser, thanks for your reply. Firstly, I am sorry for posting the discussion twice. I sent the message to the dev mail group from an unsub- scribed account, but the message was not shown for a while, and I guessed that the dev mail group would not post an email coming from an unsubscr

Re: [DISCUSS]Support the merge statement in FlinkSQL

2022-02-10 Thread Martijn Visser
Hi zoucao, I see that this message was posted twice, so I choose to only reply to the latest one (this one). Thanks for bringing this up for discussion. I agree that support for a merge statement would be a welcome addition to Flink SQL for those that are using it for bounded jobs. How would you