Hi everyone, Hi Fabian, I am also in favor of option 1.
Besides the playgrounds it is a good opportunity to explore this process for official Docker images as Till suggested. This needs a separate discussion, though. Best, Konstantin On Fri, Aug 9, 2019 at 5:25 AM Yang Wang <danrtsey...@gmail.com> wrote: > Hey Fabian, > > > Sounds great! It will be more easier to build an end-to-end play ground > with docker. > > > I prefer option 1. > > We need to build the official docker images and push to docker hub after > every version is released. It could be used to play with docker compose, > kubernetes, etc. > > > Today we could found that more and more uses are running flink with > container on kubernetes, Yarn, Mesos. So if the official flink image could > be released and maintained, it will help our users a lot. > > They could build their own image with dependencies based on the official > image to run a flink application. Also they could start a flink cluster > with official image, then submit flink jobs to the existed cluster. > > > > > Best, > > Yang > > Stephan Ewen <se...@apache.org> 于2019年8月9日周五 上午1:58写道： > > > I remember that Patrick (who maintained the docker-flink images so far) > > frequently raised the point that its good practice to have the images > > decoupled from the project release cycle. > > Changes to the images can be done frequently and released fast that way. > > > > In addition, one typically supports images for multiple releases, meaning > > the versioning is different than in the main code base. > > > > On Thu, Aug 8, 2019 at 6:35 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > > > I would be in favour of option 1. > > > > > > We could also think about making the flink-playgrounds and the Flink > > docker > > > image release part of the Flink release process [1] if we don't want to > > > have independent release cycles. I think at the moment the official > Flink > > > docker image is too often forgotten. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release > > > > > > Cheers, > > > Till > > > > > > On Thu, Aug 8, 2019 at 6:25 PM Seth Wiesman <sjwies...@gmail.com> > wrote: > > > > > > > Hey Fabian, > > > > > > > > I support option 1. > > > > > > > > As per FLIP-42, playgrounds are going to become core to flinks > getting > > > > started experience and I believe it is worth the effort to get this > > > right. > > > > > > > > - As you mentioned, we may (and in my opinion definitely will) add > more > > > > images in the future. Setting up an integration now will set the > stage > > > for > > > > those future additions. > > > > > > > > - These images will be many users first exposure to Flink and having > a > > > > proper release cycle to ensure they work properly may be worth the > > effort > > > > in and of itself. We already found during the first PR to that repo > > that > > > we > > > > needed to find users with different OSs to test. > > > > > > > > - Similarly to the above point, having the images hosted under an > > > official > > > > Apache account adds a certain amount of credibility and shows the > > > community > > > > that we take on-boarding new users seriously. > > > > > > > > - I am generally opposed having the official flink docs rely on > > something > > > > that is hosted under someone’s personal account. I don’t want bug > fixes > > > or > > > > updates to be blocked by your (or some else’s ) availability. > > > > > > > > Seth > > > > > > > > > On Aug 8, 2019, at 10:36 AM, Fabian Hueske <fhue...@gmail.com> > > wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > As you might know, some of us are currently working on Docker-based > > > > > playgrounds that make it very easy for first-time Flink users to > try > > > out > > > > > and play with Flink [0]. > > > > > > > > > > Our current setup (still work in progress with some parts merged to > > the > > > > > master branch) looks as follows: > > > > > * The playground is a Docker Compose environment [1] consisting of > > > Flink, > > > > > Kafka, and Zookeeper images (ZK for Kafka). The playground is based > > on > > > a > > > > > specific Flink job. > > > > > * We had planned to add the example job of the playground as an > > example > > > > to > > > > > the flink main repository to bundle it with the Flink distribution. > > > > Hence, > > > > > it would have been included in the Docker-hub-official (soon to be > > > > > published) Flink 1.9 Docker image [2]. > > > > > * The main motivation of adding the job to the examples module in > the > > > > flink > > > > > main repo was to avoid the maintenance overhead for a customized > > Docker > > > > > image. > > > > > > > > > > When discussing to backport the playground job (and its data > > generator) > > > > to > > > > > include it in the Flink 1.9 examples, concerns were raised about > > their > > > > > Kafka dependency which will become a problem, if the community > agrees > > > on > > > > > the recently proposed repository split, which would remove > > flink-kafka > > > > from > > > > > the main repository [3]. I think this is a fair concern, that we > did > > > not > > > > > consider when designing the playground (also the repo split was not > > > > > proposed yet). > > > > > > > > > > If we don't add the playground job to the examples, we need to put > it > > > > > somewhere else. The obvious choice would be the flink-playgrounds > [4] > > > > > repository, which was intended for the docker-compose configuration > > > > files. > > > > > However, we would not be able to include it in the > > Docker-hub-official > > > > > Flink image any more and would need to maintain a custom Docker > > image, > > > > what > > > > > we tried to avoid. The custom image would of course be based on the > > > > > Docker-hub-official Flink image. > > > > > > > > > > There are different approaches for this: > > > > > > > > > > 1) Building one (or more) official ASF images > > > > > There is an official Apache Docker Hub user [5] and a bunch of > > projects > > > > > publish Docker images via this user. Apache Infra seems to support > an > > > > > process that automatically builds and publishes Docker images when > a > > > > > release tag is added to a repository. This feature needs to be > > > enabled. I > > > > > haven't found detailed documentation on this but there is a bunch > of > > > > INFRA > > > > > Jira tickets that discuss this mechanism. > > > > > This approach would mean that we need a formal Apache release for > > > > > flink-playgrounds (similar to flink-shaded). The obvious benefits > are > > > > that > > > > > these images would be ASF-official Docker images. In case we can > > > publish > > > > > more than one image per repo, we could also publish images for > other > > > > > playgrounds (like the SQL playground, which could be based on the > SQL > > > > > training that I built [6] which uses an image that is published > under > > > my > > > > > user [7]). > > > > > > > > > > 2) Rely on an external image > > > > > This image could be build by somebody in the community (like me). > > > Problem > > > > > is of course, that the image is not an official image and we would > > rely > > > > on > > > > > a volunteer to build the images. > > > > > OTOH, the overhead would be pretty small. No need to roll run full > > > > > releases, integration with Infra's build process, etc. > > > > > > > > > > IMO, the first approach is clearly the better choice but also > needs a > > > > bunch > > > > > of things to be put into place. > > > > > > > > > > What do others think? > > > > > Does somebody have another idea? > > > > > > > > > > Cheers, > > > > > Fabian > > > > > > > > > > [0] > > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/getting-started/docker-playgrounds/flink_cluster_playground.html > > > > > [1] > > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/getting-started/docker-playgrounds/flink_cluster_playground.html#anatomy-of-this-playground > > > > > [2] https://hub.docker.com/_/flink > > > > > [3] > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/eb841f610ef2c191b8d00b6c07b2eab513da2e4eb2d7da5c5e6846f4@%3Cdev.flink.apache.org%3E > > > > > [4] https://github.com/apache/flink-playgrounds > > > > > [5] https://hub.docker.com/u/apache > > > > > [6] https://github.com/ververica/sql-training/ > > > > > [7] > https://hub.docker.com/r/fhueske/flink-sql-client-training-1.7.2 > > > > > > > > > > -- Konstantin Knauf | Solutions Architect +49 160 91394525 Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2019 -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
