Thanks everyone for the info. I have used the Flink docker containers to launch a Flink cluster, but what I'm looking forward to is more along the lines asked by Eron: submitting a Flink pipeline to a k8s cluster without having to deal with the traditional Flink cluster. I think this would be very valuable, especially if the resources can be adjusted dynamically.
Looking forward to discuss more around Flink Forward! Cheers, Thomas On Wed, Mar 21, 2018 at 8:02 AM, Eron Wright <eronwri...@gmail.com> wrote: > Thanks Stephan for the information, it's exciting. This is an aspect for > which the FLIP-6 design doc is lacking detail; somewhere we should work > through how this will work (e.g. from a job recovery and upgrade > perspective). If there's a doc or an open issue, please share, thanks! > > On Wed, Mar 21, 2018 at 5:41 AM, Stephan Ewen <se...@apache.org> wrote: > > > Hi Christophe! > > > > I could imagine 1.5.1 or 1.6. > > > > It is important to point out that you can use Flink with Kubernetes quite > > well already. That addition is mainly intended to make it more idiomatic > > for Kubernetes users. > > > > Best, > > Stephan > > > > > > On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif <cjo...@gmail.com> > > wrote: > > > > > Hi Stephan, > > > > > > > This feature did not make it for 1.5, but should come very > > > soon after. > > > > > > Are you talking about a 1.5.1 here? What is the envisioned timeline? > > > > > > Thanks! > > > > > > > > > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen <se...@apache.org> > wrote: > > > > > > > @Eron That is definitely the way we want to suggest as the way to use > > k8s > > > > in the future. This feature did not make it for 1.5, but should come > > very > > > > soon after. > > > > > > > > @Thomas An implementation of a ResourceManager for k8s should come in > > the > > > > near future. Would be happy to jump on a joint FLIP, after the 1.5 > > > release > > > > and Flink Forward (in three weeks or so). > > > > > > > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright <eronwri...@gmail.com> > > > wrote: > > > > > > > > > Till, is it possible to package a Flink application as a > > > self-contained > > > > > deployment on Kubernetes? I mean, run a Flink application using > > 'flink > > > > > run' such that it launches its own RM/JM and waits for a > sufficient # > > > of > > > > > TMs to join? > > > > > > > > > > Thanks! > > > > > > > > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann < > trohrm...@apache.org > > > > > > > > wrote: > > > > > > > > > > > I forgot to add Flink's K8 documentation [1] which might also be > > > > helpful > > > > > > with getting started. > > > > > > > > > > > > [1] > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > master/ops/deployment/ > > > > > > kubernetes.html > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann < > > > trohrm...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi Thomas, > > > > > > > > > > > > > > I think the one way to get started would be to adapt the Flink > > > docker > > > > > > > images [1,2] to run with Flink 1.5. Per default, they will use > > the > > > > > Flip-6 > > > > > > > components. > > > > > > > > > > > > > > Flink 1.5 won't come with a dedicated integration with > Kubernetes > > > > which > > > > > > is > > > > > > > able to start new pods. However, it should work that you > manually > > > or > > > > by > > > > > > the > > > > > > > virtues of an external system start new pods which can be used. > > > Once > > > > > the > > > > > > > new pods have been started and the TaskExecutors have > registered > > > one > > > > > > would > > > > > > > have to rescale the job manually to the new parallelism. > > > > > > > > > > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker- > > > > > image.html > > > > > > > [2] https://hub.docker.com/r/_/flink/ > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <t...@apache.org> > > > > wrote: > > > > > > > > > > > > > >> Hi, > > > > > > >> > > > > > > >> What would be a good starting point to try out Flink on > > Kubernetes > > > > > (any > > > > > > >> examples / tutorials)? > > > > > > >> > > > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on > > > > > Kubernetes? > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Thomas > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Christophe > > > > > >
