Hi dev and users,
I just want to revive this discussion because we have some meaningful
progress about
kubernetes native integration. I have made a draft implementation to
complete the poc.
Cli and submission are both working as expected. The design doc[1] has been
updated,
including the detailed
Hi Zhenghua,
You are right. For per-job cluster, the taskmanagers will be allocated
dynamically by KubernetesResourceManager. For session cluster, we hope
taskmangers could be pre-allocated even though it does not work now.
Please navigate to the doc[1] for more details.
Hi Thomas,
We have
Till had already summed it up, but I want to emphasize that Flink as
project only needs to provide #1 (reactive mode) and #3 (active mode, which
necessarily is tied to the cluster manager of choice). The latter would be
needed for Flink jobs to be elastic (in the future), although we may want
to di
Thanks Yang for bringing this up. I think option1 is very useful for early
adopters.
People do not know much about k8s and can easily set up on minikube to have
a taste.
For option2 and option3, i prefer option3 because i am familiar yarn and
don't have much concept of k8s.
And there is some doube
Hi till,
Thanks for your reply. I agree with you that both option 1 and 3 need to be
supported.
Option 1 is reactive mode of resource management and flink is not aware of
underlying cluster. If a user has limited resources to run flink jobs, this
option will be very useful. On the other side, o
Hi Yang,
thanks for reviving the discussion about Flink's Kubernetes integration. In
a nutshell, I think that Flink should support option 1) and 3). Concretely,
option 1) would be covered by the reactive mode [1] which is not
necessarily bound to Kubernetes and works in all environments equally we
Hi kaibo,
I am really appreciated that you could share your use case.
As you say, our users in production also could be divided into two groups.
The common users have more knowledge about flink, they could use the
command line to submit job and debug job from logs of job manager and
taskmanager
Hi Jeff,
Thank you for your attention. You are right, the design doc is out of date.
I will try to contact with Till and Sun Jin to get the edit permission. And
then update the design doc.
Currently we get the k8s cluster informations from the kube config file.
You may have multiple contexts in y
Thanks for bringing this up. Obviously, option 2 and 3 are both useful for
fink users on kubernetes. But option 3 is easy for users that not have many
concepts of kubernetes, they can start flink on kubernetes quickly, I think
it should have a higher priority.
I have worked some time to integrate
Thanks Yang. K8s natively integration is very necessary and important for
the adoption of flink IMO.
I notice that the design doc is written in 2018, is there any changes or
update ?
>>> Download the flink release binary and create the ~/.kube/config file
corresponding to the k8s cluster. It is al
10 matches
Mail list logo
