Hi, Shammon Thanks for driving this Flip, [Support Customized Job Meta Data Listener] will make it easier for Flink to collect lineage information. I fully agree with the overall solution and have a small question:
1. Will an exception thrown by the listener affect the normal execution process? 2. What is the order of execution? Is the listener executed first or are specific operations executed first? If I want to perform DDL permission verification(such as integrating with Ranger based on the listener) , is that possible? Best, Feng On Fri, May 26, 2023 at 4:09 PM Shammon FY <zjur...@gmail.com> wrote: > Hi devs, > > We would like to bring up a discussion about FLIP-294: Support Customized > Job Meta Data Listener[1]. We have had several discussions with Jark Wu, > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and > interfaces, and thanks for their valuable advice. > The overall job and connector information is divided into metadata and > lineage, this FLIP focuses on metadata and lineage will be discussed in > another FLIP in the future. In this FLIP we want to add a customized > listener in Flink to report catalog modifications to external metadata > systems such as datahub[2] or atlas[3]. Users can view the specific > information of connectors such as source and sink for Flink jobs in these > systems, including fields, watermarks, partitions, etc. > > Looking forward to hearing from you, thanks. > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > [2] https://datahub.io/ > [3] https://atlas.apache.org/#/ >
