Thanks @Gyula, I would like to share our use of sql-gateway with the Flink session cluster and I hope that it could help you to have a clearer understanding of our needs :)
As @Yangze mentioned, currently we use flink as an olap platform by the following steps 1. Setup a flink session cluster by flink k8s session with k8s or zk highavailable. 2. Write a Helm chart for Sql-Gateway image and launch multiple gateway instances to submit jobs to the same flink session cluster. As we mentioned in docs[1], we hope that users can easily launch sql-gateway instances in k8s. Does it only need to add a Helm chart for sql-gateway, or should we need to add this feature to the flink operator? Can you help give the conclusion? Thank you very much @Gyula [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/olap_quickstart/ Best, Shammon FY On Sun, Sep 17, 2023 at 2:02 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > Hi! > It sounds pretty easy to deploy the gateway automatically with session > cluster deployments from the operator , but there is a major limitation > currently. The SQL gateway itself doesn't really support any operator > integration so jobs submitted through the SQL gateway would not be > manageable by the operator (they won't show up as session jobs). > > Without that, this is a very strange feature. We would make something much > easier for users that is not well supported by the operator in the first > place. The operator is designed to manage clusters and jobs > (FlinkDeployment / FlinkSessionJob). It would be good to understand if we > could make the SQL Gateway create a FlinkSessionJob / Deployment (that > would require application cluster support) and basically submit the job > through the operator. > > Cheers, > Gyula > > On Sun, Sep 17, 2023 at 1:26 AM Yangze Guo <karma...@gmail.com> wrote: > > > > There would be many different ways of doing this. One gateway per > > session cluster, one gateway shared across different clusters... > > > > Currently, sql gateway cannot be shared across multiple clusters. > > > > > understand the tradeoff and the simplest way of accomplishing this. > > > > I'm not familiar with the Flink operator codebase, it would be > > appreciated if you could elaborate more on the cost of adding this > > feature. I agree that deploying a gateway using the native Kubernetes > > Deployment can be a simple way and straightforward for users. However, > > integrating it into an operator can provide additional benefits and be > > more user-friendly, especially for users who are less familiar with > > Kubernetes. By using an operator, users can benefit from consistent > > version management with the session cluster and upgrade capabilities. > > > > > > Best, > > Yangze Guo > > > > On Fri, Sep 15, 2023 at 5:38 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > > > > There would be many different ways of doing this. One gateway per > session > > > cluster, one gateway shared across different clusters... > > > I would not rush to add anything anywhere until we understand the > > tradeoff > > > and the simplest way of accomplishing this. > > > > > > The operator already supports ingresses for session clusters so we > could > > > have a gateway sitting somewhere else simply using it. > > > > > > Gyula > > > > > > On Fri, Sep 15, 2023 at 10:18 AM Yangze Guo <karma...@gmail.com> > wrote: > > > > > > > Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a key > > > > component for OLAP scenarios. > > > > > > > > @Gyula > > > > How about add sql gateway as an optional component to Session Cluster > > > > Deployments. User can specify the resource / instance number and > ports > > > > of the sql gateway. I think that would help a lot for OLAP and batch > > > > user. > > > > > > > > > > > > Best, > > > > Yangze Guo > > > > > > > > On Fri, Sep 15, 2023 at 3:19 PM ConradJam <jam.gz...@gmail.com> > wrote: > > > > > > > > > > If we start from the crd direction, I think this mode is more like > a > > > > > sidecar of the session cluster, which is submitted to the session > > cluster > > > > > by sending sql commands to the sql gateway. I don't know if my > > statement > > > > is > > > > > accurate. > > > > > > > > > > Xiaolong Wang <xiaolong.w...@smartnews.com.invalid> 于2023年9月15日周五 > > > > 13:27写道： > > > > > > > > > > > Hi, Dongwoo, > > > > > > > > > > > > Since Flink SQL gateway should run upon a Flink session cluster, > I > > > > think > > > > > > it'd be easier to add more fields to the CRD of > `FlinkSessionJob`. > > > > > > > > > > > > e.g. > > > > > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > > > kind: FlinkSessionJob > > > > > > metadata: > > > > > > name: sql-gateway > > > > > > spec: > > > > > > sqlGateway: > > > > > > endpoint: "hiveserver2" > > > > > > mode: "streaming" > > > > > > hiveConf: > > > > > > configMap: > > > > > > name: hive-config > > > > > > items: > > > > > > - key: hive-site.xml > > > > > > path: hive-site.xml > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim < > > dongwoo7....@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > *@Gyula* > > > > > > > Thanks for the consideration Gyula. My initial idea for the CR > > was > > > > > > roughly > > > > > > > like below. > > > > > > > I focused on simplifying the setup in k8s environment, but I > > agree > > > > with > > > > > > > your opinion that for the sql gateway > > > > > > > we don't need custom operator logic to handle and most of the > > > > > > requirements > > > > > > > can be met by existing k8s resources. > > > > > > > So maybe helm chart that bundles all resources needed should be > > > > enough. > > > > > > > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > > > > kind: FlinkSqlGateway > > > > > > > metadata: > > > > > > > name: flink-sql-gateway-example > > > > > > > namespace: default > > > > > > > spec: > > > > > > > clusterName: flink-session-cluster-example > > > > > > > exposeServiceType: LoadBalancer > > > > > > > flinkSqlGatewayConfiguration: > > > > > > > sql-gateway.endpoint.type: "hiveserver2" > > > > > > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive" > > > > > > > hiveConf: > > > > > > > configMap: > > > > > > > name: hive-config > > > > > > > items: > > > > > > > - key: hive-site.xml > > > > > > > path: hive-site.xml > > > > > > > > > > > > > > > > > > > > > *@xiaolong, @Shammon* > > > > > > > Hi xiaolong and Shammon. > > > > > > > Thanks for taking the time to share. > > > > > > > I'd also like to add my experience with setting up flink sql > > gateway > > > > on > > > > > > > k8s. > > > > > > > Without building a new Docker image, I've added a separate > > container > > > > to > > > > > > the > > > > > > > existing JobManager pod and started the sql gateway using the > > > > > > > "sql-gateway.sh start-foreground" command. > > > > > > > I haven't explored deploying the sql gateway as an independent > > > > deployment > > > > > > > yet, but that's something I'm considering after modifying JM's > > > > address to > > > > > > > desired session cluster. > > > > > > > > > > > > > > Thanks all > > > > > > > > > > > > > > Best > > > > > > > Dongwoo > > > > > > > > > > > > > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang > > > > > > > <xiaolong.w...@smartnews.com.invalid>님이 작성: > > > > > > > > > > > > > > > Hi, Shammon, > > > > > > > > > > > > > > > > Yes, I want to create a Flink SQL-gateway in a job-manager. > > > > > > > > > > > > > > > > Currently, the above script is generally a work-around and > > allows > > > > me to > > > > > > > > start a Flink session job manager with a SQL gateway running > > upon. > > > > > > > > > > > > > > > > I agree that it'd be more elegant that we create a new job > > type and > > > > > > > write a > > > > > > > > script, which is much easier for the user to use (since they > > do not > > > > > > need > > > > > > > to > > > > > > > > build a separate Flink image any more). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY < > zjur...@gmail.com > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > Currently `sql-gateway` can be started with the script > > > > > > `sql-gateway.sh` > > > > > > > > in > > > > > > > > > an existing node, it is more like a simple "standalone" > > node. I > > > > think > > > > > > > > it's > > > > > > > > > valuable if we can do more work to start it in k8s. > > > > > > > > > > > > > > > > > > For xiaolong: > > > > > > > > > Do you want to start a sql-gateway instance in the > jobmanager > > > > pod? I > > > > > > > > think > > > > > > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to > > start > > > > > > > > > `sql-gateway` pods with a flink image, what do you think? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang > > > > > > > > > <xiaolong.w...@smartnews.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > Hi, I've experiment this feature on K8S recently, here is > > some > > > > of > > > > > > my > > > > > > > > > trial: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Create a new kubernetes-jobmanager.sh script with the > > > > following > > > > > > > > > content > > > > > > > > > > > > > > > > > > > > #!/usr/bin/env bash > > > > > > > > > > $FLINK_HOME/bin/sql-gateway.sh start > > > > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh > > kubernetes-session > > > > > > > > > > > > > > > > > > > > 2. Build your own Flink docker image something like this > > > > > > > > > > FROM flink:1.17.1-scala_2.12-java11 > > > > > > > > > > > > > > > > > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > $FLINK_HOME/bin/ > > > > > > > > > > kubernetes-jobmanager1.sh > > > > > > > > > > COPY ./kubernetes-jobmanager.sh > > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > > > > > > > > > > > > > > > > > RUN chmod +x $FLINK_HOME/bin/*.sh > > > > > > > > > > USER flink > > > > > > > > > > > > > > > > > > > > 3. Create a Flink session job with the operator using the > > above > > > > > > > image. > > > > > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra < > > > > gyula.f...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > > > > > > > I don't completely understand what would be a content > of > > such > > > > > > CRD, > > > > > > > > > could > > > > > > > > > > > you give a minimal example how the Flink SQL Gateway CR > > yaml > > > > > > would > > > > > > > > look > > > > > > > > > > > like? > > > > > > > > > > > > > > > > > > > > > > Adding a CRD would mean you need to add some > > > > operator/controller > > > > > > > > logic > > > > > > > > > as > > > > > > > > > > > well. Why not simply use a Deployment / StatefulSet in > > > > > > Kubernetes? > > > > > > > > > > > > > > > > > > > > > > Or a Helm chart if you want to make it more user > > friendly? > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim < > > > > > > > dongwoo7....@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > I've been working on setting up a flink SQL gateway > in > > a > > > > k8s > > > > > > > > > > environment > > > > > > > > > > > > and it got me thinking — what if we had a CRD for > this? > > > > > > > > > > > > > > > > > > > > > > > > So I have quick questions below. > > > > > > > > > > > > 1. Is there ongoing work to create a CRD for the > Flink > > SQL > > > > > > > Gateway? > > > > > > > > > > > > 2. If not, would the community be open to > considering a > > > > CRD for > > > > > > > > this? > > > > > > > > > > > > > > > > > > > > > > > > I've noticed a growing demand for simplified setup of > > the > > > > flink > > > > > > > sql > > > > > > > > > > > gateway > > > > > > > > > > > > in flink's slack channel. > > > > > > > > > > > > Implementing a CRD could make deployments easier and > > offer > > > > > > better > > > > > > > > > > > > integration with k8s. > > > > > > > > > > > > > > > > > > > > > > > > If this idea is accepted, I'm open to drafting a FLIP > > for > > > > > > further > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your time and looking forward to your > > thoughts! > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Dongwoo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best > > > > > > > > > > ConradJam > > > > > > >
