Hi ron, Thanks for your feedback.
> 1. Regarding CatalogModificationListenerFactory#createListener method, I think it would be better to pass Context as its parameter instead of two specific Object. In this way, we can easily extend it in the future and there will be no compatibility problems. It sounds good to me I will add Context in CatalogModificationListenerFactory, thanks > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same physical table, so does the Listener report this physical table repeatedly? Yes, the listeners for different jobs may receive the same physical table, users should check and update the table information based on the identifier. For example, users may create tables on the same kafka topic in different jobs, which will notify listeners for the same kafka topic. > 3. When registering a Listener object, will it connect to an external system such as Datahub? If the Listener object registration times out due to permission issues, it will affect the execution of all subsequent SQL, what should we do in this case? Users should establish connections to external systems when creating a listener as needed, and they should handle the exceptions too. If users fail to create a listener and throw an exception, Flink will throw the exception too. Best, Shammon FY On Wed, May 31, 2023 at 11:36 AM liu ron <ron9....@gmail.com> wrote: > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from the platform produce perspective. The overall design looks good to me, > I just have some small question: > 1. Regarding CatalogModificationListenerFactory#createListener method, I > think it would be better to pass Context as its parameter instead of two > specific Object. In this way, we can easily extend it in the future and > there will be no compatibility problems. Refer to > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same > physical table, so does the Listener report this physical table repeatedly? > 3. When registering a Listener object, will it connect to an external > system such as Datahub? If the Listener object registration times out due > to permission issues, it will affect the execution of all subsequent SQL, > what should we do in this case? > > Best, > Ron > > Shammon FY <zjur...@gmail.com> 于2023年5月31日周三 08:53写道: > > > Thanks Feng, the catalog modification listener is only used to report > > read-only ddl information to other components or systems. > > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > Users need to handle the exception in the listener themselves. Many DDLs > > such as drop tables and alter tables cannot be rolled back, Flink cannot > > handle these exceptions for the listener. It will cause the operation to > > exit if an exception is thrown, but the executed DDL will be successful. > > > > > 2. What is the order of execution? Is the listener executed first or > are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > The listener will be notified to report catalog modification after DDLs > are > > successful, so you can not do permission verification for DDL in the > > listener. As mentioned above, Flink will not roll back the DDL even when > > the listener throws an exception. I think permission verification is > > another issue and can be discussed separately. > > > > > > Best, > > Shammon FY > > > > On Tue, May 30, 2023 at 1:07 AM Feng Jin <jinfeng1...@gmail.com> wrote: > > > > > Hi, Shammon > > > > > > Thanks for driving this Flip, [Support Customized Job Meta Data > Listener] > > > will make it easier for Flink to collect lineage information. > > > I fully agree with the overall solution and have a small question: > > > > > > 1. Will an exception thrown by the listener affect the normal execution > > > process? > > > > > > 2. What is the order of execution? Is the listener executed first or > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > > > > Best, > > > Feng > > > > > > On Fri, May 26, 2023 at 4:09 PM Shammon FY <zjur...@gmail.com> wrote: > > > > > > > Hi devs, > > > > > > > > We would like to bring up a discussion about FLIP-294: Support > > Customized > > > > Job Meta Data Listener[1]. We have had several discussions with Jark > > Wu, > > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > > and > > > > interfaces, and thanks for their valuable advice. > > > > The overall job and connector information is divided into metadata > and > > > > lineage, this FLIP focuses on metadata and lineage will be discussed > in > > > > another FLIP in the future. In this FLIP we want to add a customized > > > > listener in Flink to report catalog modifications to external > metadata > > > > systems such as datahub[2] or atlas[3]. Users can view the specific > > > > information of connectors such as source and sink for Flink jobs in > > these > > > > systems, including fields, watermarks, partitions, etc. > > > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > > > [2] https://datahub.io/ > > > > [3] https://atlas.apache.org/#/ > > > > > > > > > >