gt;
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>>
> >> >>>> Best regards,
> >> >>>> Mang Zhang
> >&
>>>> Hi, Mang
>> >>>>>
>> >>>>> I have a question about the implementation details. For the
>> atomicity case,
>> >>>>> since the target table is not created before the JobGraph is
>> generated, but
>> >
get table is required to exist when optimizing plan to
> generate
> >>>>> the JobGraph. So how do you solve this problem?
> >>>>>
> >>>>> Best,
> >>>>> Ron
> >>>>>
> >>>>>
t;
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Mang Zhang
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> At 2023-04-15 04:20:40, "Jing Ge&
;> The main or important reason is that the TwoPhaseCatalogTable enables
> >> >> external connectors to implement theirs own logic for commit / abort.
> >> >> In FLIP-218, for atomic CTAS, the Catalog will then just drop the table
> >> >>
it's too generic to work well.
>> >> For example, some connectors will need to clean some temporary files in
>> >> abort method. And the actual connector can know the specific logic for
>> >> aborting.
>> >>
>> >> Best regards,
>> >&
the table
>> >> when the job fail. It's not ideal for it's too generic to work well.
>> >> For example, some connectors will need to clean some temporary files in
>> >> abort method. And the actual connector can know the specific logic for
>> >
re quite different -
> >> >atomic vs. non-atomic. Another question is, before we call
> >> >Catalog#twoPhaseCreateTable(...), we have to know the value of
> >> >isStreamingMode. In case only non-atomic is supported for streaming mode,
> >> >we could just fol
t;zhangmang1"
>> 收件人: "dev" , "Jing Ge"
>> 抄送: "ron9 liu" , "lincoln 86xy" <
>> lincoln.8...@gmail.com>, luoyu...@alumni.sjtu.edu.cn
>> 发送时间: 星期三, 2023年 4 月 19日 下午 3:13:36
>> 主题: Re:Re: [DISCUSS] FLIP-305: Support ato
mes there will be no difference
>> >> >by
>> >> >using true/false isStreamingMode, sometimes they are quite different -
>> >> >atomic vs. non-atomic. Another question is, before we call
>> >> >Catalog#twoPhaseCreateTable(...), we have to
non-atomic is supported for streaming mode,
> >> >we could just follow FLIP-218 instead of (twistedly) calling
> >> >Catalog#twoPhaseCreateTable(...) with a false isStreamingMode. Did I miss
> >> >anything here?
> >> >
> >
t;> >
>> >On Fri, Apr 14, 2023 at 1:55 PM yuxia wrote:
>> >
>> >> Hi, Mang.
>> >> +1 for completing the support for atomicity of CTAS, this is very useful
>> >> in batch scenarios and integrate with the data lake which support
>>
t;> I just have one question, IIUC, the DynamiacTableSink will need to know
> >> it's for normal case or the atomicity with CTAS as well as neccessary
> >> context.
> >> Take jdbc catalog as an example, if it's CTAS with atomicity supports, the
> >
c for
> aborting.
>
> Best regards,
> Yuxia
>
>
> 发件人: "zhangmang1"
> 收件人: "dev" , "Jing Ge"
> 抄送: "ron9 liu" , "lincoln 86xy" <
> lincoln.8...@gmail.com>, luoyu...@alumni.sjtu.edu.cn
> 发送时间: 星期三, 2023年 4
g1"
收件人: "dev" , "Jing Ge"
抄送: "ron9 liu" , "lincoln 86xy" ,
luoyu...@alumni.sjtu.edu.cn
发送时间: 星期三, 2023年 4 月 19日 下午 3:13:36
主题: Re:Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS)
statement
hi, Jing
Thank you for your r
ble defined in the
>> TwoPhaseCatalogTable which is different from normal case.
>>
>> How can the DynamiacTableSink can get it? Could you give some explanation
>> or example in this FLIP?
>>
>>
>> Best regards,
>> Yuxia
>>
>> - 原始邮件
t;example in this FLIP?
>
>
>Best regards,
>Yuxia
>
>- 原始邮件 -
>发件人: "zhangmang1"
>收件人: "dev" , "ron9 liu" , "lincoln
>86xy"
>发送时间: 星期五, 2023年 4 月 14日 下午 2:50:40
>主题: Re:Re: [DISCUSS] FLIP-305: Support atomic for CRE
or example in this FLIP?
>
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "zhangmang1"
> 收件人: "dev" , "ron9 liu" ,
> "lincoln 86xy"
> 发送时间: 星期五, 2023年 4 月 14日 下午 2:50:40
> 主题: Re:Re: [DISCUSS] FLIP-305: Support atomic for
anation or
example in this FLIP?
Best regards,
Yuxia
- 原始邮件 -
发件人: "zhangmang1"
收件人: "dev" , "ron9 liu" , "lincoln
86xy"
发送时间: 星期五, 2023年 4 月 14日 下午 2:50:40
主题: Re:Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS)
statement
Hi, Lincoln and Ron
Thank you for your reply.
On the naming wise I think OK, the future expansion of new features more
uniform. I have updated the FLIP.
About Hive support atomicity CTAS, Hive is rich in usage scenarios and can be
divided into three scenarios: 1. writing Hive tables 2. writin
Hi, Mang
+1 for completing the support for atomicity of CTAS, this is very useful in
batch scenarios.
I have two questions:
1. naming wise:
a) can we rename the `Catalog#getTwoPhaseCommitCreateTable` to
`Catalog#twoPhaseCreateTable` (and we may add
twoPhaseReplaceTable/twoPhaseCreateOrReplaceTa
Hi, Mang
Atomicity is very important for CTAS, especially for batch jobs. This FLIP
is a continuation of FLIP-218, which is valuable for CTAS.
I just have one question, in the Motivation part of FLIP-218, we mentioned
three levels of atomicity semantics, can this current design do the same as
Spark
Hi, everyone
I'd like to start a discussion about FLIP-305: Support atomic for CREATE TABLE
AS SELECT(CTAS) statement [1].
CREATE TABLE AS SELECT(CTAS) statement has been support, but it's not atomic.
It will create the table first before job running. If the job execution fails,
or is ca
23 matches
Mail list logo
