Hi, Thanks for the FLIP! Data lineage is an important problem to tackle.
Can you please expand on how this is planned to be wired into the JobManager? As I understand, the listeners will be configured globally (per cluster), so this won't introduce a new code path for running per-job / per-session user code. Is that correct? Best, D On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu <xbjt...@gmail.com> wrote: > Thanks Shammon for driving this FLIP forward, I’ve several comments about > the updated FLIP. > > 1. CatalogModificationContext is introduced as a class instead of an > interface, is it a typo? > > 2. The FLIP defined multiple Map<String, String> config(); methods in > some Context classes, Could we use Configuration getConfiguration()；Class > org.apache.flink.configuration.Configuration is recommend as it’s public > API and offers more useful methods as well. > > 3. The Context of CatalogModificationListenerFactory should be an > interface too, and getUserClassLoder() > would be more aligned with flink’s naming style. > > > Best, > Leonard > > > On May 26, 2023, at 4:08 PM, Shammon FY <zjur...@gmail.com> wrote: > > > > Hi devs, > > > > We would like to bring up a discussion about FLIP-294: Support Customized > > Job Meta Data Listener[1]. We have had several discussions with Jark Wu, > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and > > interfaces, and thanks for their valuable advice. > > The overall job and connector information is divided into metadata and > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > another FLIP in the future. In this FLIP we want to add a customized > > listener in Flink to report catalog modifications to external metadata > > systems such as datahub[2] or atlas[3]. Users can view the specific > > information of connectors such as source and sink for Flink jobs in these > > systems, including fields, watermarks, partitions, etc. > > > > Looking forward to hearing from you, thanks. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > [2] https://datahub.io/ > > [3] https://atlas.apache.org/#/ > >
