Thanks Shengkai for driving. And all for your discussion. > The reason why we introduce the gateway with pluggable endpoints is that many > users has their preferences. For example, the HiveServer2 users prefer to use > the gateway with HiveServer2-style API, which has numerous tools. However, > some filnk-native users may prefer to use the REST API. Therefore, we hope to > learn from the Kyuubi's design that expose multiple endpoints with different > API that allow the user to use.
My understanding is that we need multiple endpoints, But I don't quite understand why we need both the rest api and the SQLGatewayService API, maybe I'm missing something, what's the difference between them? Is it possible to use one set of rest api to solve all the problems? > Gateway to support multiple Flink versions I think this is a good question to consider. - First of all, I think it is absolutely impossible for gateway to support multiple versions of Flink under the current architecture, because gateway relies on Flink SQL and a lot of SQL compiled and optimized code is bound to the Flink version. - The other way is that gateway does not rely on Flink SQL, and each time a different version of Flink Jar is loaded to compile the job at once, and frankly speaking, stream jobs actually prefer this model. The benefit of gateway support for multiple versions is that it's really more user-friendly. I've seen cases where users must have multiple versions existing in a cluster, and if each version needs to run a gateway, the O&M burden will be heavy. > I don't think that the Gateway is a 'core' function of Flink which should be > included with Flink. First, I think the Gateway is a 'core' function in Flink. Why? I think our architecture should be consistent, which means that Flink sql-client should use the implementation of gateway, which means that sql-client depends on gateway. And sql-client is the basic tool of flink sql, it must exist in flink repository, otherwise flink sql has no most important entrance. So, the gateway itself should be our core functionality as well. Best, Jingsong On Thu, May 5, 2022 at 10:06 PM Jark Wu <imj...@gmail.com> wrote: > > Hi Martijn, > > Regarding maintaining Gateway inside or outside Flink code base, > I would like to share my thoughts: > > > I would like to understand why it's complicated to make the upgrades > problematic. Is it because of relying on internal interfaces? If so, should > we not consider making them public? > > It's not about internal interfaces. Flink itself doesn't provide backward > compatibility for public APIs. > > > > a) it will not be possible to have separate releases of the Gateway, > they will be tied to individual Flink releases > I don't think it's a problem. On the contrary, maintaining a separate repo > for Gateway will take a lot of > extra community efforts, e.g., individual CICD, docs, releases. > > > > b) if you want the Gateway to support multiple Flink versions > Sorry, I don't see any users requesting this feature for such a long time > for SQL Gateway. > Users can build services on Gateway to easily support multi Flink versions > (a Gateway for a Flink version). > It's difficult for Gateway to support multi-version because Flink doesn't > provide an API that supports backward and forward compatibility. > If Gateway wants to support multi-version, it has to invent an > inner-gateway for each version, and Gateway act as a proxy to communicate > with inner-gateway. > So you have to have a gateway to couple with the Flink version. > > In fact, Gateway is the layer to support multi Flink versions for > higher-level applications because its API (REST, gRpc) provides backward > and forward compatibility. > The gateway itself doesn't need to support multi Flink versions. Besides, > Trino/Presto also provides servers[1] for each version. > > > > I don't think that the Gateway is a 'core' function of Flink which should > be included with Flink. > Sorry, I can't agree with this. If I remember correctly, Flink SQL has been > promoted to first-class citizen for a long time. > The community also aims to make Flink a truly batch-stream unified > computing platform, and Gateway would be the entry and center of the > platform. > From my point of view, Gateway is a very "core" function and must be > included in Flink to have better cooperation with SQL and provide an > out-of-box experience. > > Best, > Jark > > [1]: https://trino.io/download.html > > On Thu, 5 May 2022 at 19:57, godfrey he <godfre...@gmail.com> wrote: > > > Hi Shengkai. > > > > Thanks for driving the proposal, it's been silent too long. > > > > I have a few questions: > > about the Architecture > > > The architecture of the Gateway is in the following graph. > > Is the TableEnvironment shared for all sessions ? > > > > about the REST Endpoint > > > /v1/sessions > > Are both local file and remote file supported for `libs` and `jars`? > > Does sql gateway support upload file? > > > > >/v1/sessions/:session_handle/configure_session > > Can this api be replaced with `/v1/sessions/:session_handle/statements` ? > > > > >/v1/sessions/:session_id/operations/:operation_handle/status > > `:session_id` is a typo, it should be `:session_handdle` > > > > >/v1/sessions/:session_handle/statements > > >The statement must be a single command > > Does this api support `begin statement set ... end` or `statement set > > begin ... end` > > DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars? > > > > >/v1/sessions/:session_handle/operations/:operation_handle/result/:token > > >"type": # string value of LogicalType > > Some LogicalTypes can not be serialized, such as: CharType(0) > > > > about Options > > > endpoint.protocol > > I think REST is not a kind of protocol[1], but is an architectural style. > > The value should be `HTTP`. > > > > about SQLGatewayService API > > > Catalog API > > > ... > > I think we should avoid providing such api, because once catalog api > > is changed or added, > > This class should also be changed. SQL statement is a more general > > interface. > > > > > Options > > > sql-gateway.session.idle.timeout > > >sql-gateway.session.check.interval > > >sql-gateway.worker.keepalive.time > > It's better we can keep the option style as Flink, the level should > > not be too deep. > > sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout > > sql-gateway.session.check.interval -> sql-gateway.session.check-interval > > sql-gateway.worker.keepalive.time -> sql-gateway.worker.keepalive->time > > > > [1] https://restfulapi.net/ > > > > Best, > > Godfrey > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年5月5日周四 14:58写道： > > > > > > Hi Shengkai, > > > > > > I have another concern about the submission of batch job. Does the Flink > > SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is > > used to submit batch job. What about the Flink SQL gateway? > > > > > > Best regards, > > > Nicholas Jiang > > > > > > On 2022/04/24 03:28:36 Shengkai Fang wrote: > > > > Hi. Jiang. > > > > > > > > Thanks for your feedback！ > > > > > > > > > Do the public interfaces of GatewayService refer to any service? > > > > > > > > We will only expose one GatewayService implementation. We will put the > > > > interface into the common package and the developer who wants to > > implement > > > > a new endpoint can just rely on the interface package rather than the > > > > implementation. > > > > > > > > > What's the behavior of SQL Client Gateway working on Yarn or K8S? > > Does > > > > the SQL Client Gateway support application or session mode on Yarn? > > > > > > > > I think we can support SQL Client Gateway to submit the jobs in > > > > application/sesison mode. > > > > > > > > > Is there any event trigger in the operation state machine? > > > > > > > > Yes. I have already updated the content and add more details about the > > > > state machine. During the revise, I found that I mix up the two > > concepts: > > > > job submission and job execution. In fact, we only control the > > submission > > > > mode at the gateway layer. Therefore, we don't need to mapping the > > > > JobStatus here. If the user expects that the synchronization behavior > > is to > > > > wait for the completion of the job execution before allowing the next > > > > statement to be executed, then the Operation lifecycle should also > > contains > > > > the job's execution, which means users should set `table.dml-sync`. > > > > > > > > > What's the return schema for the public interfaces of GatewayService? > > > > Like getTable interface, what's the return value schema? > > > > > > > > The API of the GatewayService return the java objects and the endpoint > > can > > > > organize the objects with expected schema. The return results is also > > list > > > > the section ComponetAPI#GatewayService#API. The return type of the > > > > GatewayService#getTable is `ContextResolvedTable`. > > > > > > > > > How does the user get the operation log? > > > > > > > > The OperationManager will register the LogAppender before the Operation > > > > execution. The Log Appender will hijack the logger and also write the > > log > > > > that related to the Operation to another files. When users wants to > > fetch > > > > the Operation log, the GatewayService will read the content in the > > file and > > > > return. > > > > > > > > Best, > > > > Shengkai > > > > > > > > > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:21写道： > > > > > > > > > Hi Shengkai. > > > > > > > > > > Thanks for driving the proposal of SQL Client Gateway. I have some > > > > > knowledge of Kyuubi and have some questions about the design: > > > > > > > > > > 1.Do the public interfaces of GatewayService refer to any service? If > > > > > referring to HiveService, does GatewayService need interfaces like > > > > > getQueryId etc. > > > > > > > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? > > Does > > > > > the SQL Client Gateway support application or session mode on Yarn? > > > > > > > > > > 3.Is there any event trigger in the operation state machine? > > > > > > > > > > 4.What's the return schema for the public interfaces of > > GatewayService? > > > > > Like getTable interface, what's the return value schema? > > > > > > > > > > 5.How does the user get the operation log? > > > > > > > > > > Thanks, > > > > > Nicholas Jiang > > > > > > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote: > > > > > > Hi, Flink developers. > > > > > > > > > > > > I want to start a discussion about the FLIP-91: Support Flink SQL > > > > > > Gateway[1]. Flink SQL Gateway is a service that allows users to > > submit > > > > > and > > > > > > manage their jobs in the online environment with the pluggable > > endpoints. > > > > > > The reason why we introduce the Gateway with pluggable endpoints > > is that > > > > > > many users have their preferences. For example, the HiveServer2 > > users > > > > > > prefer to use the gateway with HiveServer2-style API, which has > > numerous > > > > > > tools. However, some filnk-native users may prefer to use the REST > > API. > > > > > > Therefore, we propose the SQL Gateway with pluggable endpoint. > > > > > > > > > > > > In the FLIP, we also propose the REST endpoint, which has the > > similar > > > > > > APIs compared to the gateway in the > > ververica/flink-sql-gateway[2]. At > > > > > the > > > > > > last, we discuss how to use the SQL Client to submit the statement > > to the > > > > > > Gateway with the REST API. > > > > > > > > > > > > I am glad that you can give some feedback about FLIP-91. > > > > > > > > > > > > Best, > > > > > > Shengkai > > > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway > > > > > > [2] https://github.com/ververica/flink-sql-gateway > > > > > > > > > > > > > > > > >
