Hi Erik, thanks for reaching out to the Flink community. The Flink community is actually actively working on improving our K8s support because we see a lot of adoption and our users would like to run Flink on K8s. Apart from the existing Flink docker images there are currently two development threads:
1) Active K8s integration [1] The idea of the active K8s integration is to enable Flink to talk to the K8s master in order to actively start or terminate new Pods which run Flink's TaskManager. That way, Flink would be able to adjust the set of available resources dynamically based on the actual needs (coming from a Flink job). I know that the community is actively working on drafting a design document for this mode which will be published very soon. Moreover, I have a dev branch where I prototyped the K8s integration [2]. 2) Reactive container mode [3] The reactive container mode works actually independent of the underlying container framework but K8s will be the natural deployment target for this. The idea is that not Flink but an external system/user is responsible for allocating/releasing resources. E.g. some auto-scaling groups could decide that Flink is currently not keeping up wrt some metric and start some new Pods (running Flink's TaskManager). If this happens, Flink should react to such an event and use the newly available resources. Usually this means that Flink scales up the parallelism of its job to deploy new tasks to the newly started TaskManager. There is actually a design document linked under the respective JIRA issue for more details. I would be happy to join your meeting to give an overview what the Flink community is currently developing, answering questions and gathering feedback for what we intend to do. How would I join this meeting? [1] https://issues.apache.org/jira/browse/FLINK-9953 [2] https://github.com/tillrohrmann/flink/tree/nativeKubernetes [3] https://issues.apache.org/jira/browse/FLINK-10407 Cheers, Till On Mon, Oct 22, 2018 at 6:55 PM Erik Erlandson <eerla...@redhat.com> wrote: > Hi Flink community, > > I serve as chair on the Kubernetes Big Data SIG, and we've had some > interest from the SIG in learning more about what the Flink community is > working on in the kubernetes space. > > I'm putting out feelers to see if anybody from the community would be > interested in giving a brief talk and/or demo of Flink integrations with > k8s at one of our regular meetings. > > The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next > meeting is Oct 31). Feel free to attend our meetings, or visit our slack > channel: > https://kubernetes.slack.com/messages/C0ELB338T/ > > More SIG info is available on our regular meeting minutes doc: > > https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/ > > Thanks! > Erik Erlandson >
