> >>> -> process - > asset – see for example
>> > >>>
>> https://egeria-project.org/features/lineage-management/overview/#the-lineage-graph
>> > >>>
>> > >>> Maybe I am missing it, but it seems to be that it wo
> store the process in the lineage graph.
> > >>>
> > >>> It is useful to have the top level lineage as source -> Flink job ->
> > >>> sink. Where the Flink job is the process, but also to have this
> asset ->
> > >>> process -> a
verview/#the-lineage-graph
> >>>
> >>> Maybe I am missing it, but it seems to be that it would be useful to
> >>> store the process in the lineage graph.
> >>>
> >>> It is useful to have the top level lineage as source -> Flink job ->
it would be useful to
>>> store the process in the lineage graph.
>>>
>>> It is useful to have the top level lineage as source -> Flink job ->
>>> sink. Where the Flink job is the process, but also to have this asset ->
>>> process -> asset pattern f
link job is the process, but also to have this asset ->
>> process -> asset pattern for each of the steps in the job. If this is
>> present, please could you point me to it,
>>
>> Kind regards, David.
>>
>>
>>
>>
>>
>> From: David R
gt;
>
>
>
> From: David Radley
> Date: Tuesday, 19 September 2023 at 16:11
> To: dev@flink.apache.org
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-314: Support Customized Job Lineage
> Listener
> Hi,
> I notice that there is an experimental lineage integration for Flink w
m: XTransfer
Date: Tuesday, 19 September 2023 at 15:47
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-314: Support Customized Job Lineage
Listener
Thanks Shammon for this proposal.
That’s helpful for collecting the lineage of Flink tasks.
Looking forward to its implementat
2023 at 15:47
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-314: Support Customized Job Lineage
Listener
Thanks Shammon for this proposal.
That’s helpful for collecting the lineage of Flink tasks.
Looking forward to its implementation.
Best,
Jiabao
> 2023年9月18日 20:56，Leonard
Thanks Shammon for this proposal.
That’s helpful for collecting the lineage of Flink tasks.
Looking forward to its implementation.
Best,
Jiabao
> 2023年9月18日 20:56，Leonard Xu 写道：
>
> Thanks Shammon for the informations, the comment makes the lifecycle clearer.
> +1
>
>
> Best,
> Leonard
>
Thanks Shammon for the informations, the comment makes the lifecycle clearer.
+1
Best,
Leonard
> On Sep 18, 2023, at 7:54 PM, Shammon FY wrote:
>
> Hi devs,
>
> After discussing with @Qingsheng, I fixed a minor issue of the lineage
> lifecycle in `StreamExecutionEnvironment`. I have added
Hi devs,
After discussing with @Qingsheng, I fixed a minor issue of the lineage
lifecycle in `StreamExecutionEnvironment`. I have added the comment to
explain that the lineage information in `StreamExecutionEnvironment` will
be consistent with that of transformations. When users clear the existing
Hi devs,
Thanks for all the valuable feedback. If there are no more concerns about
FLIP-314[1], I would like to start voting later, thanks.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
Best,
Shammon FY
On Wed, Jul 12, 2023 at 11:18
Thanks for the valuable feedback, Leonard.
I have discussed with Leonard off-line. We have reached some conclusions
about these issues and I have updated the FLIP as follows:
1. Simplify the `LineageEdge` interface by creating an edge from one source
vertex to sink vertex.
2. Remove the `TableCol
acef8b11e82c9187c
>>>
>>> Best,
>>> Shammon FY
>>>
>>>
>>> On Sun, Jun 25, 2023 at 4:17 PM Yun Tang wrote:
>>>
>>>> Hi Shammon,
>>>>
>>>> I like the idea in general and it will help to analy
t; > > interface.
> > > > >
> > > > > > By the way, if you want to focus on job lineage instead of data
> > > column
> > > > > lineage in this FLIP, why we must introduce so many column-lineage
> > > > related
>
in SQL jobs includes table lineage and column
> > > > lineage. Although SQL jobs currently do not support column lineage,
> we
> > > > would like to support this in the next step. So we have
> comprehensively
> > > > considered the table lineage and column lineage interfaces here, and
8acef8b11e82c9187c
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Sun, Jun 25, 2023 at 4:17 PM Yun Tang wrote:
> > >
> > > > Hi Shammon,
> > > >
> > > > I like the idea in general and it will help to
; > defined these two interfaces together clearly
>> >
>> >
>> > [1]
>> >
>> >
>> https://github.com/FangYongs/flink/commit/d4bfe57e7a5315b790e79b8acef8b11e82c9187c
>> >
>> > Best,
>> > Shammon FY
>> >
>
t; > RuntimeExecutionMode, as the latter one is not easy to understand for
> > users.
> > >
> > > I have one more question on the lookup-join dim tables, it seems this
> > FLIP
> > > does not touch them, and will them become part of the
> List
> &g
for
> users.
> >
> > I have one more question on the lookup-join dim tables, it seems this
> FLIP
> > does not touch them, and will them become part of the List
> > sources() or adding another interface?
> >
> > By the way, if you want to focus on job line
tter FlinkSQL or Flink jar jobs in production environments.
> > >
> > > For Qingsheng's concern, I'd like the name of JobType more than
> > > RuntimeExecutionMode, as the latter one is not easy to understand for
> > users.
> > >
> > >
the lookup-join dim tables, it seems this
> FLIP
> > does not touch them, and will them become part of the List
> > sources() or adding another interface?
> >
> > By the way, if you want to focus on job lineage instead of data column
> > lineage in this FLIP,
ge instead of data column
> lineage in this FLIP, why we must introduce so many column-lineage related
> interface here?
>
>
> Best
> Yun Tang
> ____________________
> From: Shammon FY
> Sent: Sunday, June 25, 2023 16:13
> To: dev@flink.apache.org
> Subjec
ce so many column-lineage related interface
here?
Best
Yun Tang
From: Shammon FY
Sent: Sunday, June 25, 2023 16:13
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
Hi Qingsheng,
Thanks for your valuable feedback
Hi Qingsheng,
Thanks for your valuable feedback.
> 1. Is there any specific use case to expose the batch / streaming info to
listeners or meta services?
I agree with you that Flink is evolving towards batch-streaming
unification, but the lifecycle of them is different. If a job processes a
bound
s FLIP to make us know
better about the interfaces.
Best regards,
Yuxia
- 原始邮件 -
发件人: "Qingsheng Ren"
收件人: "dev"
抄送: "Shammon FY"
发送时间: 星期二, 2023年 6 月 20日 下午 6:19:10
主题: Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
Hi Shammon,
Thanks for
Hi Shammon,
Thanks for starting this FLIP! Data lineage is a very important topic,
which has been missing for a long time in Flink. I have some questions
about the FLIP.
About events and listeners:
1. I’m not sure if it is necessary to expose JobType to in JobCreatedEvent.
This is an internal cl
Hi devs,
Is there any comment or feedback for this FLIP? Hope to hear from you,
thanks
Best,
Shammon FY
On Tue, Jun 6, 2023 at 8:22 PM Shammon FY wrote:
> Hi devs,
>
> I would like to start a discussion on FLIP-314: Support Customized Job
> Lineage Listener[1] which is the next stage of FLIP-
28 matches
Mail list logo
