Hi, Thanks for all the feedback.
For @Jing Ge, I forget to update the demo code in the FLIP, the method is `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there is no `onEvent(CatalogModificationEvent)`. I have updated the code. Context contains some additional information that is not part of an Event, but needs to be used in the listener, so we separate it from the event. For @Panagiotis, I think `ioExecutor` make sense to me and I have added it in `ContextModificationContext`, thanks For @Leonard, Thanks for your input. 1. I have updated `CatalogModificationContext` as an interface, as well as Context in CatalogModificationListenerFactory 2. Configuration sounds good to me, I have updated the method name and getConfiguration in Context For @David, Yes, you're right. The listener will only be used on the client side and won't introduce a new code path for running per-job/per-session jobs. The listener will be created in `TableEnvironment` and `SqlGateway` which can a `CatalogManager` with the listener. Best, Shammon FY On Tue, Jun 6, 2023 at 3:33 PM David Morávek <david.mora...@gmail.com> wrote: > Hi, > > Thanks for the FLIP! Data lineage is an important problem to tackle. > > Can you please expand on how this is planned to be wired into the > JobManager? As I understand, the listeners will be configured globally (per > cluster), so this won't introduce a new code path for running per-job / > per-session user code. Is that correct? > > Best, > D > > On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu <xbjt...@gmail.com> wrote: > > > Thanks Shammon for driving this FLIP forward, I’ve several comments about > > the updated FLIP. > > > > 1. CatalogModificationContext is introduced as a class instead of an > > interface, is it a typo? > > > > 2. The FLIP defined multiple Map<String, String> config(); methods in > > some Context classes, Could we use Configuration > getConfiguration()；Class > > org.apache.flink.configuration.Configuration is recommend as it’s public > > API and offers more useful methods as well. > > > > 3. The Context of CatalogModificationListenerFactory should be an > > interface too, and getUserClassLoder() > > would be more aligned with flink’s naming style. > > > > > > Best, > > Leonard > > > > > On May 26, 2023, at 4:08 PM, Shammon FY <zjur...@gmail.com> wrote: > > > > > > Hi devs, > > > > > > We would like to bring up a discussion about FLIP-294: Support > Customized > > > Job Meta Data Listener[1]. We have had several discussions with Jark > Wu, > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > and > > > interfaces, and thanks for their valuable advice. > > > The overall job and connector information is divided into metadata and > > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > > another FLIP in the future. In this FLIP we want to add a customized > > > listener in Flink to report catalog modifications to external metadata > > > systems such as datahub[2] or atlas[3]. Users can view the specific > > > information of connectors such as source and sink for Flink jobs in > these > > > systems, including fields, watermarks, partitions, etc. > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > > [2] https://datahub.io/ > > > [3] https://atlas.apache.org/#/ > > > > >
