Hi devs, I would like to start a discussion on FLIP-314: Support Customized Job Lineage Listener[1] which is the next stage of FLIP-294 [2]. Flink streaming and batch jobs create lineage dependency between source and sink, users can manage their data and jobs according to this lineage information. For example, when there is a delay in Flink ETL or data, users can easily trace the problematic jobs and affected data. On the other hand, when users need to correct data or debug, they can perform operations based on lineage too.
In FLIP-314 we want to introduce lineage related interfaces for Flink and users can create customized job status listeners. When job status changes, users can get job status and information to add, update or delete lineage. Looking forward to your feedback, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener Best, Shammon FY
