Hi Dongoo, You are right that plan advisor doesn't have ability to block user request. Actually, we add a new REST API in the gateway side to validate the users request is legal or not in our inner branch. Therefore, user can validate their statements before execution or execute their statement directly. I feel that separating execution from validation follows the design principle of single responsibility, and also returns the power of choice to the user, allowing the user to explicitly include the risks in the requested action. WDYT?
Furthermore, I feel that in this example, I am more inclined towards the Kafka connector itself providing this type of validation, rather than perceiving the internal implementation details of the connector from the outside. This approach is not intuitive and is not conducive to maintaining this information. In my view, the gateway is just a tool for submitting and managing jobs, with responsibilities including permission management, cluster management, and not encompassing awareness of the specific connector implementation details. Best, Shengkai Dongwoo Kim <dongwoo7....@gmail.com> 于2024年7月22日周一 22:34写道： > Hi Shengkai, > > Thank you for introducing an interesting concept of plan advice. > After reviewing the FLIP, I thought that plan advice cannot fully replace > the original intention of the OperationValidator. > Similar to the pluggable ResourceValidator[1] in flink k8s operator, the > OperationValidator aims to block certain operations and throw exceptions, > acting as a safeguard rather than an interactive advisor. > > For example, it can target CREATE TABLE queries that set configurations > which can lead to issues due to human errors, such as group.id and > scan.startup.timestamp-millis in kafka tables. > Additionally, it can block INSERT operations if the operator wants to > ensure the sql gateway remains read-only. > > I might have misunderstood the extensibility of plan advice, so please feel > free to correct me. > Any feedback is appreciated. > Thanks. > > Best, > Dongwoo > > [1] > > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/plugins/#custom-flink-resource-validators > > > > 2024년 7월 19일 (금) 오후 3:50, Shengkai Fang <fskm...@gmail.com>님이 작성: > > > Hi Dongwoo, > > > > Thanks for your proposal! > > > > I think the idea about OperationValidator is similar to the plan > advice[1], > > which can give advice if the statement has problems. I wonder if it would > > be more beneficial to develop the concept within the existing > > infrastructure, rather than introducing a new concept on the gateway > side. > > Additionally, I find it challenging for OperationValidator to validate > the > > query behavior as a plan advisor, as it lacks the necessary tools for > > obtaining the upsert key or statistics. > > > > Best, > > Shengkai > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-280%3A+Introduce+EXPLAIN+PLAN_ADVICE+to+provide+SQL+advice > > > > Dongwoo Kim <dongwoo7....@gmail.com> 于2024年7月18日周四 21:14写道： > > > > > Hi Martijn, > > > > > > Thanks for the guidance on making a FLIP. > > > > > > Before proceeding, I’d like to gather some initial thoughts from the > > > community to see if we agree on the usefulness of this feature. > > > Could anyone share their opinions on the custom query operation > validator > > > for Flink SQL Gateway? > > > Thanks in advance! > > > > > > Best, > > > Dongwoo > > > > > > 2024년 7월 18일 (목) 오후 9:52, Martijn Visser <martijnvis...@apache.org>님이 > > 작성: > > > > > > > Hi Dongwoo, > > > > > > > > I think this should first have been a FLIP [1], given that it > proposes > > to > > > > make something pluggable with custom interfaces. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > > > On Thu, Jul 18, 2024 at 2:50 PM Dongwoo Kim <dongwoo7....@gmail.com> > > > > wrote: > > > > > > > > > Hi devs, > > > > > > > > > > A month ago, I created a jira issue and pr for a custom query > > operation > > > > > validator in Flink SQL Gateway to block unsafe queries. > > > > > Since there has been no feedback yet, I would appreciate it if > > someone > > > > > could review it or provide some feedback. > > > > > I'm curious if this feature can be added and if others have felt > the > > > same > > > > > need. > > > > > > > > > > Thanks in advance! > > > > > > > > > > jira: https://issues.apache.org/jira/browse/FLINK-35560 > > > > > pr: https://github.com/apache/flink/pull/25092 > > > > > > > > > > Best, > > > > > Dongwoo > > > > > > > > > > > > > > >
