Hi all,
Currently cloud native architectures has been introduced to many companies
in production. They use kubernetes to run deep learning, web server, etc.
If we could deploy the per-job/session flink cluster on kubernetes to make
it mix-run with other workloads, the cluster resource utilization will be
better. Also many kubernetes users are more easier to have a taste on the
flink.
By now we have three options to run flink jobs on k8s.
[1]. Create jm/tm/service yaml and apply, then you will get a flink
standalone cluster on k8s. Use flink run to submit job to the existed flink
cluster. Some companies may have their own deploy system to manage the
flink cluster.
[2]. Use flink-k8s-operator to manage multiple flink clusters, including
session and perjob. It could manage the complete deployment lifecycle of
the application. I think this option is really easy to use for the k8s
users. They are familiar with k8s-opertor, kubectl and other tools of k8s.
They could debug and run the flink cluster just like other k8s applications.
[3]. Natively integration with k8s, use the flink run or
kubernetes-session.sh to start a flink cluster. It is very similar to
submitting an flink cluster to Yarn. KubernetesClusterDescriptor talks to
k8s api server to start a flink master deployment of 1.
KubernetesResourceManager dynamically allocates resource from k8s to start
task manager as demand. This option is very easy for flink users to get
started. In the simplest case, we just need to update the '-m yarn-cluster'
to -m '-m kubernetes-cluster'.
We have make an internal implementation of option [3] and use it in
production. After fully tested, we hope to contribute it to the community.
Now we want to get some feedbacks about the three options. Any comments are
welcome.
> What do we need to prepare when start a flink cluster on k8s using native
integration?
Download the flink release binary and create the ~/.kube/config file
corresponding to the k8s cluster. It is all what you need.
> Flink Session cluster
* start a session cluster
./bin/kubernetes-session.sh -d -n 2 -tm 512 -s 4 -nm flink-session-example
-i flink:latest -kD kubernetes.service.exposed.type=NODE_PORT
* You will get an address to submit job, specify it through ’-ksa’ option
./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-session-example
-ksa {x.x.x.x:12345} examples/streaming/WindowJoin.jar
> Flink Job Cluster
* running with official flink image
./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-perjob-example-1
-ki flink:latest examples/streaming/WindowJoin.jar
* running with user image
./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-perjob-example-1
-ki flink-user:latest examples/streaming/WindowJoin.jar
[1].
https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html
[2].https://github.com/lyft/flinkk8soperator
[3].
https://docs.google.com/document/d/1Zmhui_29VASPcBOEqyMWnF3L6WEWZ4kedrCqya0WaAk/edit#
