Hi all,
*@Gyula*
Thanks for the consideration Gyula. My initial idea for the CR was roughly
like below.
I focused on simplifying the setup in k8s environment, but I agree with
your opinion that for the sql gateway
we don't need custom operator logic to handle and most of the requirements
can be met by existing k8s resources.
So maybe helm chart that bundles all resources needed should be enough.
apiVersion: flink.apache.org/v1beta1
kind: FlinkSqlGateway
metadata:
name: flink-sql-gateway-example
namespace: default
spec:
clusterName: flink-session-cluster-example
exposeServiceType: LoadBalancer
flinkSqlGatewayConfiguration:
sql-gateway.endpoint.type: "hiveserver2"
sql-gateway.endpoint.hiveserver2.catalog.name: "hive"
hiveConf:
configMap:
name: hive-config
items:
- key: hive-site.xml
path: hive-site.xml
*@xiaolong, @Shammon*
Hi xiaolong and Shammon.
Thanks for taking the time to share.
I'd also like to add my experience with setting up flink sql gateway on k8s.
Without building a new Docker image, I've added a separate container to the
existing JobManager pod and started the sql gateway using the
"sql-gateway.sh start-foreground" command.
I haven't explored deploying the sql gateway as an independent deployment
yet, but that's something I'm considering after modifying JM's address to
desired session cluster.
Thanks all
Best
Dongwoo
2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang
<xiaolong.w...@smartnews.com.invalid>님이 작성:
> Hi, Shammon,
>
> Yes, I want to create a Flink SQL-gateway in a job-manager.
>
> Currently, the above script is generally a work-around and allows me to
> start a Flink session job manager with a SQL gateway running upon.
>
> I agree that it'd be more elegant that we create a new job type and write a
> script, which is much easier for the user to use (since they do not need to
> build a separate Flink image any more).
>
>
>
> On Fri, Sep 15, 2023 at 10:29 AM Shammon FY <zjur...@gmail.com> wrote:
>
> > Hi,
> >
> > Currently `sql-gateway` can be started with the script `sql-gateway.sh`
> in
> > an existing node, it is more like a simple "standalone" node. I think
> it's
> > valuable if we can do more work to start it in k8s.
> >
> > For xiaolong:
> > Do you want to start a sql-gateway instance in the jobmanager pod? I
> think
> > maybe we need a script like `kubernetes-sql-gatewah.sh` to start
> > `sql-gateway` pods with a flink image, what do you think?
> >
> > Best,
> > Shammon FY
> >
> >
> > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang
> > <xiaolong.w...@smartnews.com.invalid> wrote:
> >
> > > Hi, I've experiment this feature on K8S recently, here is some of my
> > trial:
> > >
> > >
> > > 1. Create a new kubernetes-jobmanager.sh script with the following
> > content
> > >
> > > #!/usr/bin/env bash
> > > $FLINK_HOME/bin/sql-gateway.sh start
> > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session
> > >
> > > 2. Build your own Flink docker image something like this
> > > FROM flink:1.17.1-scala_2.12-java11
> > >
> > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh $FLINK_HOME/bin/
> > > kubernetes-jobmanager1.sh
> > > COPY ./kubernetes-jobmanager.sh
> $FLINK_HOME/bin/kubernetes-jobmanager.sh
> > >
> > > RUN chmod +x $FLINK_HOME/bin/*.sh
> > > USER flink
> > >
> > > 3. Create a Flink session job with the operator using the above image.
> > >
> > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra <gyula.f...@gmail.com>
> wrote:
> > >
> > > > Hi!
> > > >
> > > > I don't completely understand what would be a content of such CRD,
> > could
> > > > you give a minimal example how the Flink SQL Gateway CR yaml would
> look
> > > > like?
> > > >
> > > > Adding a CRD would mean you need to add some operator/controller
> logic
> > as
> > > > well. Why not simply use a Deployment / StatefulSet in Kubernetes?
> > > >
> > > > Or a Helm chart if you want to make it more user friendly?
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim <dongwoo7....@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've been working on setting up a flink SQL gateway in a k8s
> > > environment
> > > > > and it got me thinking — what if we had a CRD for this?
> > > > >
> > > > > So I have quick questions below.
> > > > > 1. Is there ongoing work to create a CRD for the Flink SQL Gateway?
> > > > > 2. If not, would the community be open to considering a CRD for
> this?
> > > > >
> > > > > I've noticed a growing demand for simplified setup of the flink sql
> > > > gateway
> > > > > in flink's slack channel.
> > > > > Implementing a CRD could make deployments easier and offer better
> > > > > integration with k8s.
> > > > >
> > > > > If this idea is accepted, I'm open to drafting a FLIP for further
> > > > > discussion
> > > > >
> > > > > Thanks for your time and looking forward to your thoughts!
> > > > >
> > > > > Best regards,
> > > > > Dongwoo
> > > > >
> > > >
> > >
> >
>
