Hi ShengKai, After I read FLIP-91[1], I want to add an init-file option. Its functionality is the same as option '-i' of Flink SQL Client.
When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by this option after SQL Gateway starts every time. Shall we name this option `sql-gateway.session.init-file` and write it into the FLIP-91? Best regards, LuNing Wang [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91 Martijn Visser <mart...@ververica.com> 于2022年4月26日周二 20:32写道： > Hi Shengkai, > > Thanks for opening this discussion. I did a first brief pass over the FLIP > and I have a couple of questions/remarks: > > * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway > which would be used by Flink's SQL Client (or other applications)? > > * From a user perspective, I would have expected that we start with the > REST endpoint before explaining how we would integrate this into Flink. Now > it's quite hard to first understand what we want to offer to users and if > that will be sufficient for a first version. > > * With Flink 1.15, we're introducing an OpenAPI specification [1]. Can we > also do this straight away for the REST Gateway? > > * Should we introduce the REST Gateway as part of Flink's main repository? > Wouldn't we be better off to maintain this in a separate repository under > ASF? Ideally you would like to be able to support multiple Flink versions > with one version of the REST Gateway I think? > > * There's no mention of Batch or Streaming in this concept. If I recall > correctly, the current Flink SQL Gateway can only support Batch. How will > we support Streaming? Will there be any requirement with JDBC, as there > currently is? > > * I can imagine that if a user wants to use a REST Gateway, there's also a > strong need to combine this with a Catalog. Do you think this should be > part of this FLIP? > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager > > On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi. Jiang. > > > > Thanks for your feedback！ > > > > > Do the public interfaces of GatewayService refer to any service? > > > > We will only expose one GatewayService implementation. We will put the > > interface into the common package and the developer who wants to > implement > > a new endpoint can just rely on the interface package rather than the > > implementation. > > > > > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does > > the SQL Client Gateway support application or session mode on Yarn? > > > > I think we can support SQL Client Gateway to submit the jobs in > > application/sesison mode. > > > > > Is there any event trigger in the operation state machine? > > > > Yes. I have already updated the content and add more details about the > > state machine. During the revise, I found that I mix up the two concepts: > > job submission and job execution. In fact, we only control the submission > > mode at the gateway layer. Therefore, we don't need to mapping the > > JobStatus here. If the user expects that the synchronization behavior is > to > > wait for the completion of the job execution before allowing the next > > statement to be executed, then the Operation lifecycle should also > contains > > the job's execution, which means users should set `table.dml-sync`. > > > > > What's the return schema for the public interfaces of GatewayService? > > Like getTable interface, what's the return value schema? > > > > The API of the GatewayService return the java objects and the endpoint > can > > organize the objects with expected schema. The return results is also > list > > the section ComponetAPI#GatewayService#API. The return type of the > > GatewayService#getTable is `ContextResolvedTable`. > > > > > How does the user get the operation log? > > > > The OperationManager will register the LogAppender before the Operation > > execution. The Log Appender will hijack the logger and also write the log > > that related to the Operation to another files. When users wants to fetch > > the Operation log, the GatewayService will read the content in the file > and > > return. > > > > Best, > > Shengkai > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:21写道： > > > > > Hi Shengkai. > > > > > > Thanks for driving the proposal of SQL Client Gateway. I have some > > > knowledge of Kyuubi and have some questions about the design: > > > > > > 1.Do the public interfaces of GatewayService refer to any service? If > > > referring to HiveService, does GatewayService need interfaces like > > > getQueryId etc. > > > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? > Does > > > the SQL Client Gateway support application or session mode on Yarn? > > > > > > 3.Is there any event trigger in the operation state machine? > > > > > > 4.What's the return schema for the public interfaces of GatewayService? > > > Like getTable interface, what's the return value schema? > > > > > > 5.How does the user get the operation log? > > > > > > Thanks, > > > Nicholas Jiang > > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote: > > > > Hi, Flink developers. > > > > > > > > I want to start a discussion about the FLIP-91: Support Flink SQL > > > > Gateway[1]. Flink SQL Gateway is a service that allows users to > submit > > > and > > > > manage their jobs in the online environment with the pluggable > > endpoints. > > > > The reason why we introduce the Gateway with pluggable endpoints is > > that > > > > many users have their preferences. For example, the HiveServer2 users > > > > prefer to use the gateway with HiveServer2-style API, which has > > numerous > > > > tools. However, some filnk-native users may prefer to use the REST > API. > > > > Therefore, we propose the SQL Gateway with pluggable endpoint. > > > > > > > > In the FLIP, we also propose the REST endpoint, which has the similar > > > > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. > At > > > the > > > > last, we discuss how to use the SQL Client to submit the statement to > > the > > > > Gateway with the REST API. > > > > > > > > I am glad that you can give some feedback about FLIP-91. > > > > > > > > Best, > > > > Shengkai > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway > > > > [2] https://github.com/ververica/flink-sql-gateway > > > > > > > > > >
