One more thing to add. If we move the code to flink-playgrounds and build custom images, the playgrounds effort won't be tied to the Flink 1.9 release any more. So, we'd be a bit more flexible time-wise but would also need to manually update the playgrounds for every release.
Am Do., 8. Aug. 2019 um 17:36 Uhr schrieb Fabian Hueske <fhue...@gmail.com>: > Hi everyone, > > As you might know, some of us are currently working on Docker-based > playgrounds that make it very easy for first-time Flink users to try out > and play with Flink [0]. > > Our current setup (still work in progress with some parts merged to the > master branch) looks as follows: > * The playground is a Docker Compose environment [1] consisting of Flink, > Kafka, and Zookeeper images (ZK for Kafka). The playground is based on a > specific Flink job. > * We had planned to add the example job of the playground as an example to > the flink main repository to bundle it with the Flink distribution. Hence, > it would have been included in the Docker-hub-official (soon to be > published) Flink 1.9 Docker image [2]. > * The main motivation of adding the job to the examples module in the > flink main repo was to avoid the maintenance overhead for a customized > Docker image. > > When discussing to backport the playground job (and its data generator) to > include it in the Flink 1.9 examples, concerns were raised about their > Kafka dependency which will become a problem, if the community agrees on > the recently proposed repository split, which would remove flink-kafka from > the main repository [3]. I think this is a fair concern, that we did not > consider when designing the playground (also the repo split was not > proposed yet). > > If we don't add the playground job to the examples, we need to put it > somewhere else. The obvious choice would be the flink-playgrounds [4] > repository, which was intended for the docker-compose configuration files. > However, we would not be able to include it in the Docker-hub-official > Flink image any more and would need to maintain a custom Docker image, what > we tried to avoid. The custom image would of course be based on the > Docker-hub-official Flink image. > > There are different approaches for this: > > 1) Building one (or more) official ASF images > There is an official Apache Docker Hub user [5] and a bunch of projects > publish Docker images via this user. Apache Infra seems to support an > process that automatically builds and publishes Docker images when a > release tag is added to a repository. This feature needs to be enabled. I > haven't found detailed documentation on this but there is a bunch of INFRA > Jira tickets that discuss this mechanism. > This approach would mean that we need a formal Apache release for > flink-playgrounds (similar to flink-shaded). The obvious benefits are that > these images would be ASF-official Docker images. In case we can publish > more than one image per repo, we could also publish images for other > playgrounds (like the SQL playground, which could be based on the SQL > training that I built [6] which uses an image that is published under my > user [7]). > > 2) Rely on an external image > This image could be build by somebody in the community (like me). Problem > is of course, that the image is not an official image and we would rely on > a volunteer to build the images. > OTOH, the overhead would be pretty small. No need to roll run full > releases, integration with Infra's build process, etc. > > IMO, the first approach is clearly the better choice but also needs a > bunch of things to be put into place. > > What do others think? > Does somebody have another idea? > > Cheers, > Fabian > > [0] > https://ci.apache.org/projects/flink/flink-docs-master/getting-started/docker-playgrounds/flink_cluster_playground.html > [1] > https://ci.apache.org/projects/flink/flink-docs-master/getting-started/docker-playgrounds/flink_cluster_playground.html#anatomy-of-this-playground > [2] https://hub.docker.com/_/flink > [3] > https://lists.apache.org/thread.html/eb841f610ef2c191b8d00b6c07b2eab513da2e4eb2d7da5c5e6846f4@%3Cdev.flink.apache.org%3E > [4] https://github.com/apache/flink-playgrounds > [5] https://hub.docker.com/u/apache > [6] https://github.com/ververica/sql-training/ > [7] https://hub.docker.com/r/fhueske/flink-sql-client-training-1.7.2 >
