> * Should we introduce the REST Gateway as part of Flink's main repository? Wouldn't we be better off to maintain this in a separate repository under ASF? Ideally you would like to be able to support multiple Flink versions with one version of the REST Gateway I think?
We would be better off maintaining this in a separate repository. It is important to support multiple Flink versions. Users can upgrade a large number of Flink jobs versions gradually in a Gateway service. LuNing Wang <wang4lun...@gmail.com> 于2022年4月27日周三 17:54写道： > Hi ShengKai, > > After I read FLIP-91[1], I want to add an init-file option. Its > functionality is the same as option '-i' of Flink SQL Client. > > When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by > this option after SQL Gateway starts every time. > > Shall we name this option `sql-gateway.session.init-file` and write it > into the FLIP-91? > > Best regards, > > LuNing Wang > > [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91 > > Martijn Visser <mart...@ververica.com> 于2022年4月26日周二 20:32写道： > >> Hi Shengkai, >> >> Thanks for opening this discussion. I did a first brief pass over the FLIP >> and I have a couple of questions/remarks: >> >> * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway >> which would be used by Flink's SQL Client (or other applications)? >> >> * From a user perspective, I would have expected that we start with the >> REST endpoint before explaining how we would integrate this into Flink. >> Now >> it's quite hard to first understand what we want to offer to users and if >> that will be sufficient for a first version. >> >> * With Flink 1.15, we're introducing an OpenAPI specification [1]. Can we >> also do this straight away for the REST Gateway? >> >> * Should we introduce the REST Gateway as part of Flink's main repository? >> Wouldn't we be better off to maintain this in a separate repository under >> ASF? Ideally you would like to be able to support multiple Flink versions >> with one version of the REST Gateway I think? >> >> * There's no mention of Batch or Streaming in this concept. If I recall >> correctly, the current Flink SQL Gateway can only support Batch. How will >> we support Streaming? Will there be any requirement with JDBC, as there >> currently is? >> >> * I can imagine that if a user wants to use a REST Gateway, there's also a >> strong need to combine this with a Catalog. Do you think this should be >> part of this FLIP? >> >> Best regards, >> >> Martijn Visser >> https://twitter.com/MartijnVisser82 >> https://github.com/MartijnVisser >> >> [1] >> >> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager >> >> On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fskm...@gmail.com> wrote: >> >> > Hi. Jiang. >> > >> > Thanks for your feedback！ >> > >> > > Do the public interfaces of GatewayService refer to any service? >> > >> > We will only expose one GatewayService implementation. We will put the >> > interface into the common package and the developer who wants to >> implement >> > a new endpoint can just rely on the interface package rather than the >> > implementation. >> > >> > > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does >> > the SQL Client Gateway support application or session mode on Yarn? >> > >> > I think we can support SQL Client Gateway to submit the jobs in >> > application/sesison mode. >> > >> > > Is there any event trigger in the operation state machine? >> > >> > Yes. I have already updated the content and add more details about the >> > state machine. During the revise, I found that I mix up the two >> concepts: >> > job submission and job execution. In fact, we only control the >> submission >> > mode at the gateway layer. Therefore, we don't need to mapping the >> > JobStatus here. If the user expects that the synchronization behavior >> is to >> > wait for the completion of the job execution before allowing the next >> > statement to be executed, then the Operation lifecycle should also >> contains >> > the job's execution, which means users should set `table.dml-sync`. >> > >> > > What's the return schema for the public interfaces of GatewayService? >> > Like getTable interface, what's the return value schema? >> > >> > The API of the GatewayService return the java objects and the endpoint >> can >> > organize the objects with expected schema. The return results is also >> list >> > the section ComponetAPI#GatewayService#API. The return type of the >> > GatewayService#getTable is `ContextResolvedTable`. >> > >> > > How does the user get the operation log? >> > >> > The OperationManager will register the LogAppender before the Operation >> > execution. The Log Appender will hijack the logger and also write the >> log >> > that related to the Operation to another files. When users wants to >> fetch >> > the Operation log, the GatewayService will read the content in the file >> and >> > return. >> > >> > Best, >> > Shengkai >> > >> > >> > >> > >> > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:21写道： >> > >> > > Hi Shengkai. >> > > >> > > Thanks for driving the proposal of SQL Client Gateway. I have some >> > > knowledge of Kyuubi and have some questions about the design: >> > > >> > > 1.Do the public interfaces of GatewayService refer to any service? If >> > > referring to HiveService, does GatewayService need interfaces like >> > > getQueryId etc. >> > > >> > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? >> Does >> > > the SQL Client Gateway support application or session mode on Yarn? >> > > >> > > 3.Is there any event trigger in the operation state machine? >> > > >> > > 4.What's the return schema for the public interfaces of >> GatewayService? >> > > Like getTable interface, what's the return value schema? >> > > >> > > 5.How does the user get the operation log? >> > > >> > > Thanks, >> > > Nicholas Jiang >> > > >> > > On 2022/04/21 06:42:30 Shengkai Fang wrote: >> > > > Hi, Flink developers. >> > > > >> > > > I want to start a discussion about the FLIP-91: Support Flink SQL >> > > > Gateway[1]. Flink SQL Gateway is a service that allows users to >> submit >> > > and >> > > > manage their jobs in the online environment with the pluggable >> > endpoints. >> > > > The reason why we introduce the Gateway with pluggable endpoints is >> > that >> > > > many users have their preferences. For example, the HiveServer2 >> users >> > > > prefer to use the gateway with HiveServer2-style API, which has >> > numerous >> > > > tools. However, some filnk-native users may prefer to use the REST >> API. >> > > > Therefore, we propose the SQL Gateway with pluggable endpoint. >> > > > >> > > > In the FLIP, we also propose the REST endpoint, which has the >> similar >> > > > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. >> At >> > > the >> > > > last, we discuss how to use the SQL Client to submit the statement >> to >> > the >> > > > Gateway with the REST API. >> > > > >> > > > I am glad that you can give some feedback about FLIP-91. >> > > > >> > > > Best, >> > > > Shengkai >> > > > >> > > > [1] >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway >> > > > [2] https://github.com/ververica/flink-sql-gateway >> > > > >> > > >> > >> >
