Hi everyone,

I started a vote for this FLIP[1], please vote there[2] or ask additional
questions here[3].

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
[2] https://lists.apache.org/thread/089glskx8ydphnyx8q1v996s8y83xkn6
[3] https://lists.apache.org/thread/bk8x0nqg4oc62jqryj9ntzzlpj062wd9

Best,
Lincoln Lee


Lincoln Lee <lincoln.8...@gmail.com> 于2023年3月10日周五 16:44写道:

> Hi everyone,
>
> If there are no other questions or concerns for the FLIP[1], I'd like to
> start the vote next Monday(3.13).
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
>
> Best,
> Lincoln Lee
>
>
> Jing Ge <j...@ververica.com.invalid> 于2023年3月10日周五 05:59写道:
>
>> Hi Lincoln,
>>
>> sounds good to me. Thanks!
>>
>> Best,
>> Jing
>>
>> On Wed, Mar 8, 2023 at 2:01 PM Lincoln Lee <lincoln.8...@gmail.com>
>> wrote:
>>
>> > Hi Jing,
>> > Agree with you that using formal terms can be easier to users, I've
>> updated
>> > the FLIP[1], since this is only one of the application scenarios for
>> > partial insert, our java doc for the corresponding interface will
>> describe
>> > the partial insert message itself from a generic point of view, WDTY?
>> >
>> > @Jacky thanks for your feedback!
>> > here are my thoughts for the two questions:
>> > for this scenario, I don't think the planner should report an error. We
>> > cannot assume that such usage will necessarily result in errors or that
>> > users are unaware of potential risks (just like in a database, similar
>> > operations are not prompted with errors). In the streaming scenario,
>> > regarding the risks associated with the multi-insert operation with
>> > overlapping fields, we may consider expanding the plan advice (FLIP-280
>> has
>> > just added possibilities to support this) to prompt users instead of
>> > rejecting the operation with an error.
>> > > 1. if the two insert into with same columns, the result is not
>> > nondeterminism. will it check in planner and throw exception
>> >
>> > yes, not all connectors support partial insert. Therefore, the
>> introduction
>> > of this interface is only intended as additional information for the
>> > connectors that need it. The new `targetColumns` only provide the column
>> > list information corresponding to the statement according to the SQL
>> > standard, and existing connectors do not need to make any passive
>> changes
>> > by default.
>> > > 2. some sink connectors can not supports it like queue such as kafka
>> > compacted topic. will also it check in planner  and throw exception
>> >
>> > welcome your feedback!
>> >
>> >
>> > [1]
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
>> >
>> >
>> > Best,
>> > Lincoln Lee
>> >
>> >
>> > Jacky Lau <liuyong...@gmail.com> 于2023年3月8日周三 20:11写道:
>> >
>> > > Thanks for bringing this up. this is a good feature. but i have two
>> > > questions:
>> > > 1. if the two insert into with same columns, the result is
>> > > not  nondeterminism. will it check in planner and throw exception
>> > > 2. some sink connectors can not supports it like queue such as kafka
>> > > compacted topic. will also it check in planner  and throw exception
>> > >
>> > > Lincoln Lee <lincoln.8...@gmail.com> 于2023年3月7日周二 14:53写道:
>> > >
>> > > > Hi Aitozi,
>> > > >
>> > > > Thanks for your feedback!  Yes, including HBase and JDBC connector,
>> > they
>> > > > can be considered for support in the next step (JDBC as as a
>> standard
>> > > > protocol supported not only in traditional databases, but also in
>> more
>> > > and
>> > > > more new types of storage). Considering the ongoing externalizing of
>> > > > connectors and the release cycles of the connectors are decoupled
>> with
>> > > the
>> > > > release cycle of Flink, we can initiate corresponding support issues
>> > for
>> > > > specific connectors to follow up on support after finalizing the API
>> > > > changes, WDYT?
>> > > >
>> > > > Best,
>> > > > Lincoln Lee
>> > > >
>> > > >
>> > > > Hang Ruan <ruanhang1...@gmail.com> 于2023年3月7日周二 12:14写道:
>> > > >
>> > > > > Hi, Lincoln,
>> > > > >
>> > > > > Thanks for bringing this up. It looks good to me. I also agree
>> with
>> > > > > Jingsong's suggestion.
>> > > > >
>> > > > > Best,
>> > > > > Hang
>> > > > >
>> > > > > Jingsong Li <jingsongl...@gmail.com> 于2023年3月7日周二 11:15写道:
>> > > > >
>> > > > > > Wow, we have 300 FLIPs...
>> > > > > >
>> > > > > > Thanks Lincoln,
>> > > > > >
>> > > > > > Have you considered returning an Optional<int[][]>?
>> > > > > >
>> > > > > > Empty array looks a little weird to me.
>> > > > > >
>> > > > > > Best,
>> > > > > > Jingsong
>> > > > > >
>> > > > > > On Tue, Mar 7, 2023 at 10:32 AM Aitozi <gjying1...@gmail.com>
>> > wrote:
>> > > > > > >
>> > > > > > > Hi Lincoln,
>> > > > > > >     Thank you for sharing this FLIP. Overall, it looks good to
>> > me.
>> > > I
>> > > > > have
>> > > > > > > one question: with the introduction of this interface,
>> > > > > > > will any existing Flink connectors need to be updated in
>> order to
>> > > > take
>> > > > > > > advantage of its capabilities? For example, HBase.
>> > > > > > >
>> > > > > > > yuxia <luoyu...@alumni.sjtu.edu.cn> 于2023年3月7日周二 10:01写道:
>> > > > > > >
>> > > > > > > > Thanks. It makes sense to me.
>> > > > > > > >
>> > > > > > > > Best regards,
>> > > > > > > > Yuxia
>> > > > > > > >
>> > > > > > > > ----- 原始邮件 -----
>> > > > > > > > 发件人: "Lincoln Lee" <lincoln.8...@gmail.com>
>> > > > > > > > 收件人: "dev" <dev@flink.apache.org>
>> > > > > > > > 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
>> > > > > > > > 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to
>> > > > > > DynamicTableSink#Context
>> > > > > > > > to solve the null overwrite problem of partial-insert
>> > > > > > > >
>> > > > > > > > hi yuxia,
>> > > > > > > >
>> > > > > > > > Thanks for your feedback and tracking the issue of update
>> > > > statement!
>> > > > > > I've
>> > > > > > > > updated the FLIP[1] and also the poc[2].
>> > > > > > > > Since the bug and flip are orthogonal, we can focus on
>> > finalizing
>> > > > the
>> > > > > > api
>> > > > > > > > changes first, and then work on the flip implementation and
>> > > bugfix
>> > > > > > > > separately, WDYT?
>> > > > > > > >
>> > > > > > > > [1]
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
>> > > > > > > > [2] https://github.com/apache/flink/pull/22041
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Lincoln Lee
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > yuxia <luoyu...@alumni.sjtu.edu.cn> 于2023年3月6日周一 21:21写道:
>> > > > > > > >
>> > > > > > > > > Hi, Lincoln.
>> > > > > > > > > Thanks for bringing this up. +1 for this FLIP, it's
>> helpful
>> > for
>> > > > > > external
>> > > > > > > > > storage system to implement partial update.
>> > > > > > > > > The FLIP looks good to me. I only want to add one comment,
>> > > update
>> > > > > > > > > statement also doesn't support updating nested column, I
>> have
>> > > > > created
>> > > > > > > > > FLINK-31344[1] to track it.
>> > > > > > > > > Maybe we also need to explain it in this FLIP.
>> > > > > > > > >
>> > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-31344
>> > > > > > > > >
>> > > > > > > > > Best regards,
>> > > > > > > > > Yuxia
>> > > > > > > > >
>> > > > > > > > > ----- 原始邮件 -----
>> > > > > > > > > 发件人: "Lincoln Lee" <lincoln.8...@gmail.com>
>> > > > > > > > > 收件人: "dev" <dev@flink.apache.org>
>> > > > > > > > > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
>> > > > > > > > > 主题: [DISCUSS] FLIP-300: Add targetColumns to
>> > > > > > DynamicTableSink#Context to
>> > > > > > > > > solve the null overwrite problem of partial-insert
>> > > > > > > > >
>> > > > > > > > > Hi everyone,
>> > > > > > > > >
>> > > > > > > > > This FLIP[1] aims to support connectors in avoiding
>> > overwriting
>> > > > > > > > non-target
>> > > > > > > > > columns with null values when processing partial column
>> > > updates,
>> > > > we
>> > > > > > > > propose
>> > > > > > > > > adding information on the target column list to
>> > > > > > DynamicTableSink#Context.
>> > > > > > > > >
>> > > > > > > > > FLINK-18726[2] supports inserting statements with
>> specified
>> > > > column
>> > > > > > list,
>> > > > > > > > it
>> > > > > > > > > fills null values (or potentially declared default values
>> in
>> > > the
>> > > > > > future)
>> > > > > > > > > for columns not appearing in the column list of insert
>> > > statement
>> > > > to
>> > > > > > the
>> > > > > > > > > target table.
>> > > > > > > > > But this behavior does not satisfy some partial column
>> update
>> > > > > > > > requirements
>> > > > > > > > > of some storage systems which allow storing null values.
>> The
>> > > > > problem
>> > > > > > is
>> > > > > > > > > that connectors cannot distinguish whether the null value
>> of
>> > a
>> > > > > > column is
>> > > > > > > > > really from the user's data or whether it is a null value
>> > > > populated
>> > > > > > > > because
>> > > > > > > > > of partial insert behavior.
>> > > > > > > > >
>> > > > > > > > > Looking forward to your comments or feedback.
>> > > > > > > > >
>> > > > > > > > > [1]
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
>> > > > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-18726
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Lincoln Lee
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Reply via email to