Hi yuxia Thanks for your input. The `AlterDatabaseEvent` extends `DatabaseModificationEvent` which has the original database.
Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > Thanks Shammon for driving it. > The FLIP generally looks good to me. I only have one question. > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and > the new CatalogDatabase after modified. Is it enough only pass the origin > database name? Will it be better to contain the origin CatalogDatabase so > that listener have ways to know what changes? > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "ron9 liu" <ron9....@gmail.com> > 收件人: "dev" <dev@flink.apache.org> > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from the platform produce perspective. The overall design looks good to me, > I just have some small question: > 1. Regarding CatalogModificationListenerFactory#createListener method, I > think it would be better to pass Context as its parameter instead of two > specific Object. In this way, we can easily extend it in the future and > there will be no compatibility problems. Refer to > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same > physical table, so does the Listener report this physical table repeatedly？ > 3. When registering a Listener object, will it connect to an external > system such as Datahub? If the Listener object registration times out due > to permission issues, it will affect the execution of all subsequent SQL, > what should we do in this case? > > Best, > Ron > > Shammon FY <zjur...@gmail.com> 于2023年5月31日周三 08:53写道： > > > Thanks Feng, the catalog modification listener is only used to report > > read-only ddl information to other components or systems. > > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > Users need to handle the exception in the listener themselves. Many DDLs > > such as drop tables and alter tables cannot be rolled back, Flink cannot > > handle these exceptions for the listener. It will cause the operation to > > exit if an exception is thrown, but the executed DDL will be successful. > > > > > 2. What is the order of execution? Is the listener executed first or > are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > The listener will be notified to report catalog modification after DDLs > are > > successful, so you can not do permission verification for DDL in the > > listener. As mentioned above, Flink will not roll back the DDL even when > > the listener throws an exception. I think permission verification is > > another issue and can be discussed separately. > > > > > > Best, > > Shammon FY > > > > On Tue, May 30, 2023 at 1:07 AM Feng Jin <jinfeng1...@gmail.com> wrote: > > > > > Hi, Shammon > > > > > > Thanks for driving this Flip, [Support Customized Job Meta Data > Listener] > > > will make it easier for Flink to collect lineage information. > > > I fully agree with the overall solution and have a small question: > > > > > > 1. Will an exception thrown by the listener affect the normal execution > > > process? > > > > > > 2. What is the order of execution? Is the listener executed first or > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > > > > Best, > > > Feng > > > > > > On Fri, May 26, 2023 at 4:09 PM Shammon FY <zjur...@gmail.com> wrote: > > > > > > > Hi devs, > > > > > > > > We would like to bring up a discussion about FLIP-294: Support > > Customized > > > > Job Meta Data Listener[1]. We have had several discussions with Jark > > Wu, > > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > > and > > > > interfaces, and thanks for their valuable advice. > > > > The overall job and connector information is divided into metadata > and > > > > lineage, this FLIP focuses on metadata and lineage will be discussed > in > > > > another FLIP in the future. In this FLIP we want to add a customized > > > > listener in Flink to report catalog modifications to external > metadata > > > > systems such as datahub[2] or atlas[3]. Users can view the specific > > > > information of connectors such as source and sink for Flink jobs in > > these > > > > systems, including fields, watermarks, partitions, etc. > > > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > > > [2] https://datahub.io/ > > > > [3] https://atlas.apache.org/#/ > > > > > > > > > >
