Hi all,
Thanks for all the valuable feedback. If there's no further question or
concern, I will start voting for FLIP-294 [1] later.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener
Best,
Shammon FY
On Fri, Jun 9, 2023 at 9:07 A
Thanks Jing, it makes sense to me and I have updated the FLIP
Best,
Shammon FY
On Thu, Jun 8, 2023 at 11:15 PM Jing Ge wrote:
> Hi Shammon,
>
> If we take a look at the JDK Event design as a reference, we can even add
> an Object into the event [1]. Back to the CatalogModificationEvent,
> eve
Hi Shammon,
If we take a look at the JDK Event design as a reference, we can even add
an Object into the event [1]. Back to the CatalogModificationEvent,
everything related to the event could be defined in the Event. If we want
to group some information into the Context, we could also consider add
Hi,
To @Jing Ge
> Thanks for the clarification. Just out of curiosity, if the context is
not part of the event, why should it be the input parameter of each onEvent
call?
I think it's quite strange to put some information in an Event, such as a
factory identifier for catalog, but they will be use
Hi Shammon,
FLIP generally looks good and I'm excited to see this feature.
1. I'm also curious about default implementations. Would atlas/datahub be
supported by default?
2. The FLIP title is confusing to me, especially in distinguishing it from
FLIP-314. Would a better FLIP title be "Support Cat
Hi Shammon,
Thanks for the clarification. Just out of curiosity, if the context is not
part of the event, why should it be the input parameter of each onEvent
call?
Best regards,
Jing
On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu wrote:
> Thanks Shammon for the timely update, the updated FLIP loo
Thanks Shammon for the timely update, the updated FLIP looks good to me.
Hope to see the vote thread and following FLIP-314 discussion thread.
Best,
Leonard
> On Jun 6, 2023, at 5:04 PM, Shammon FY wrote:
>
> Hi,
>
> Thanks for all the feedback.
>
> For @Jing Ge,
> I forget to update the dem
Hi,
Thanks for all the feedback.
For @Jing Ge,
I forget to update the demo code in the FLIP, the method is
`onEvent(CatalogModificationEvent, CatalogModificationContext)` and there
is no `onEvent(CatalogModificationEvent)`. I have updated the code. Context
contains some additional information tha
Hi,
Thanks for the FLIP! Data lineage is an important problem to tackle.
Can you please expand on how this is planned to be wired into the
JobManager? As I understand, the listeners will be configured globally (per
cluster), so this won't introduce a new code path for running per-job /
per-sessio
Thanks Shammon for driving this FLIP forward, I’ve several comments about the
updated FLIP.
1. CatalogModificationContext is introduced as a class instead of an interface,
is it a typo?
2. The FLIP defined multiple Map config(); methods in some
Context classes, Could we use Configuration ge
see the follow-up discussion on FLIP-314.
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> On Thu, Jun 1, 2023 at 5:45 AM yuxia
> wrote:
> >>
> >> > Thanks for explanation. Make sense to me.
> >> >
> >
gt;> Best regards,
> >>
> >> Martijn
> >>
> >> On Thu, Jun 1, 2023 at 5:45 AM yuxia
> wrote:
> >>
> >> > Thanks for explanation. Make sense to me.
> >> >
> >> > Best regards,
> >> > Yux
; >
>> > Best regards,
>> > Yuxia
>> >
>> > ----- 原始邮件 -
>> > 发件人: "Shammon FY"
>> > 收件人: "dev"
>> > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12
>> > 主题: Re: [DISCUSS] FLIP-294: Support Customize
r can know what has been
> modified
> > > for the database. Or the listener don't need to care about the actual
> > > modification.
> > > Also, I'm wondering whether AlterDatabaseEven need to include
> > > ignoreIfNotExists method since alter databas
- 原始邮件 -
> 发件人: "Shammon FY"
> 收件人: "dev"
> 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12
> 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
>
> Thanks yuxia, you're right and I'll add the new database to
> AlterDatabaseEvent.
&
Thanks for explanation. Make sense to me.
Best regards,
Yuxia
- 原始邮件 -
发件人: "Shammon FY"
收件人: "dev"
发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12
主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks yuxia, you're right and I'll add the new
se if exists xxx'.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Shammon FY"
> 收件人: "dev"
> 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26
> 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
>
> Hi yuxia
>
> Than
to include ignoreIfNotExists
method since alter database operation don't have such syntax like 'alter
database if exists xxx'.
Best regards,
Yuxia
- 原始邮件 -
发件人: "Shammon FY"
收件人: "dev"
发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26
主题: Re: [DISCUSS] FLIP-294:
t;
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "ron9 liu"
> 收件人: "dev"
> 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04
> 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
>
> Hi, Shammon
>
> Thanks for driving this FLIP, It will
Hi ron,
Thanks for your feedback.
> 1. Regarding CatalogModificationListenerFactory#createListener method, I
think it would be better to pass Context as its parameter instead of two
specific Object. In this way, we can easily extend it in the future and
there will be no compatibility problems.
I
rigin CatalogDatabase so that listener
have ways to know what changes?
Best regards,
Yuxia
- 原始邮件 -
发件人: "ron9 liu"
收件人: "dev"
发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04
主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, Shammon
Thanks for d
Hi, Shammon
Thanks for driving this FLIP, It will enforce the Flink metadata capability
from the platform produce perspective. The overall design looks good to me,
I just have some small question:
1. Regarding CatalogModificationListenerFactory#createListener method, I
think it would be better to
Thanks Feng, the catalog modification listener is only used to report
read-only ddl information to other components or systems.
> 1. Will an exception thrown by the listener affect the normal execution
process?
Users need to handle the exception in the listener themselves. Many DDLs
such as drop
Hi, Shammon
Thanks for driving this Flip, [Support Customized Job Meta Data Listener]
will make it easier for Flink to collect lineage information.
I fully agree with the overall solution and have a small question:
1. Will an exception thrown by the listener affect the normal execution
process?
Hi devs,
We would like to bring up a discussion about FLIP-294: Support Customized
Job Meta Data Listener[1]. We have had several discussions with Jark Wu,
Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and
interfaces, and thanks for their valuable advice.
The overall job and
25 matches
Mail list logo
