Hi devs, We would like to bring up a discussion about FLIP-294: Support Customized Job Meta Data Listener[1]. We have had several discussions with Jark Wu, Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and interfaces, and thanks for their valuable advice. The overall job and connector information is divided into metadata and lineage, this FLIP focuses on metadata and lineage will be discussed in another FLIP in the future. In this FLIP we want to add a customized listener in Flink to report catalog modifications to external metadata systems such as datahub[2] or atlas[3]. Users can view the specific information of connectors such as source and sink for Flink jobs in these systems, including fields, watermarks, partitions, etc.
Looking forward to hearing from you, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener [2] https://datahub.io/ [3] https://atlas.apache.org/#/
