Thanks Shammon for the timely update, the updated FLIP looks good to me. Hope to see the vote thread and following FLIP-314 discussion thread.
Best, Leonard > On Jun 6, 2023, at 5:04 PM, Shammon FY <zjur...@gmail.com> wrote: > > Hi, > > Thanks for all the feedback. > > For @Jing Ge, > I forget to update the demo code in the FLIP, the method is > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there > is no `onEvent(CatalogModificationEvent)`. I have updated the code. Context > contains some additional information that is not part of an Event, but > needs to be used in the listener, so we separate it from the event. > > For @Panagiotis, > I think `ioExecutor` make sense to me and I have added it in > `ContextModificationContext`, thanks > > For @Leonard, > Thanks for your input. > 1. I have updated `CatalogModificationContext` as an interface, as well as > Context in CatalogModificationListenerFactory > 2. Configuration sounds good to me, I have updated the method name and > getConfiguration in Context > > For @David, > Yes, you're right. The listener will only be used on the client side and > won't introduce a new code path for running per-job/per-session jobs. The > listener will be created in `TableEnvironment` and `SqlGateway` which can a > `CatalogManager` with the listener. > > > Best, > Shammon FY > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek <david.mora...@gmail.com> > wrote: > >> Hi, >> >> Thanks for the FLIP! Data lineage is an important problem to tackle. >> >> Can you please expand on how this is planned to be wired into the >> JobManager? As I understand, the listeners will be configured globally (per >> cluster), so this won't introduce a new code path for running per-job / >> per-session user code. Is that correct? >> >> Best, >> D >> >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu <xbjt...@gmail.com> wrote: >> >>> Thanks Shammon for driving this FLIP forward, I’ve several comments about >>> the updated FLIP. >>> >>> 1. CatalogModificationContext is introduced as a class instead of an >>> interface, is it a typo? >>> >>> 2. The FLIP defined multiple Map<String, String> config(); methods in >>> some Context classes, Could we use Configuration >> getConfiguration()；Class >>> org.apache.flink.configuration.Configuration is recommend as it’s public >>> API and offers more useful methods as well. >>> >>> 3. The Context of CatalogModificationListenerFactory should be an >>> interface too, and getUserClassLoder() >>> would be more aligned with flink’s naming style. >>> >>> >>> Best, >>> Leonard >>> >>>> On May 26, 2023, at 4:08 PM, Shammon FY <zjur...@gmail.com> wrote: >>>> >>>> Hi devs, >>>> >>>> We would like to bring up a discussion about FLIP-294: Support >> Customized >>>> Job Meta Data Listener[1]. We have had several discussions with Jark >> Wu, >>>> Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions >> and >>>> interfaces, and thanks for their valuable advice. >>>> The overall job and connector information is divided into metadata and >>>> lineage, this FLIP focuses on metadata and lineage will be discussed in >>>> another FLIP in the future. In this FLIP we want to add a customized >>>> listener in Flink to report catalog modifications to external metadata >>>> systems such as datahub[2] or atlas[3]. Users can view the specific >>>> information of connectors such as source and sink for Flink jobs in >> these >>>> systems, including fields, watermarks, partitions, etc. >>>> >>>> Looking forward to hearing from you, thanks. >>>> >>>> >>>> [1] >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener >>>> [2] https://datahub.io/ >>>> [3] https://atlas.apache.org/#/ >>> >>> >>
