Hi, This has already been discussed a few times notably in August 2022 under the topic Time to start publishing Spark Docker Images?
Having said that, building a docker image is a trivial job not taking a few minutes. Beside most cloud vendors they have their own specific tags. For example see below [image: image.png] In general from a practical point of view the docker image should be able to interact with the backend databases etc that may not support java 17 etc. To me building an official docker image is essentially an academic exercise. HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 8 May 2023 at 09:58, Yikun Jiang <yi...@apache.org> wrote: > This is a call for discussion for how we can unified Apache Spark Docker > image tag fluently. > > As you might know, there is an apache/spark-docker > <https://github.com/apache/spark-docker> repo to store the dockerfiles > and help to publish the docker images, also intended to replace the > original manually publish workflow. > > The scope of new images is to cover previous image cases (K8s / docker > run) and also cover base image, standalone, Docker Official Image. > > - (Previous) apache/spark:v3.4.0, apache/spark-py:v3.4.0, > apache/spark-r:v3.4.0 > > * The image build from apache/spark spark on k8s dockerfiles > <https://github.com/apache/spark/tree/branch-3.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark> > > * Java version: Java 17 (It was Java 11 before v3.4.0, such as > v3.3.0/v3.3.1/v3.3.2), set Java 17 by default in SPARK-40941 > <https://github.com/apache/spark/pull/38417>. > > * Support: K8s / docker run > > * See also: Time to start publishing Spark Docker Images > <https://lists.apache.org/thread/h729bxrf1o803l4wz7g8bngkjd56y6x8> > > * Link: https://hub.docker.com/r/apache/spark-py, > https://hub.docker.com/r/apache/spark-r, > https://hub.docker.com/r/apache/spark > > - (New) apache/spark:3.4.0-python3(3.4.0/latest), apache/spark:3.4.0-r, > apache/spark:3.4.0-scala, and also a all in one image: > apache/spark:3.4.0-scala2.12-java11-python3-r-ubuntu > > * The image build from apache/spark-docker dockerfiles > <https://github.com/apache/spark-docker/tree/master/3.4.0> > > * Java version: Java 11, Java17 is supported by SPARK-40513 > <https://github.com/apache/spark-docker/pull/35> (under review) > > * Support: K8s / docker run / base image / standalone / Docker > Official Image > > * See detail in: Support Docker Official Image for Spark > <https://issues.apache.org/jira/browse/SPARK-40513> > > * About dropping prefix `v`: > https://github.com/docker-library/official-images/issues/14506 > > * Link: https://hub.docker.com/r/apache/spark > > We had some initial discuss on spark-website#458 > <https://github.com/apache/spark-website/pull/458#issuecomment-1522426236>, > the mainly discussion is around version tag and default Java version > behavior changes, so we’d like to hear your idea in here about below > questions: > > *#1.Which Java version should be used by default (latest tag)? Java8 or > Java 11 or Java 17 or Any* > > *#2.Which tag should be used in apache/spark? v3.4.0 (with prefix v) or > 3.4.0 (dropping prefix v) or Both or Any* > > Starts with my prefer: > > 1. Java8 or Java17 are also ok to me (mainly considering the Java > maintenance cycle). BTW, other apache projects: flink (8/11, 11 as default > <https://github.com/docker-library/official-images/blob/93270eb07fb448fe7756b28af5495428242dcd6b/library/flink#L10>), > solr (11 as default > <https://github.com/apache/solr-docker/blob/989825ee6dce2f6bf7b31051f1ba053b6c4426f2/8.11/Dockerfile#L4> > for 8.x, 17 as default > <https://github.com/apache/solr-docker/blob/989825ee6dce2f6bf7b31051f1ba053b6c4426f2/9.2/Dockerfile#L17> > since solr9), zookeeper (11 as default > <https://github.com/31z4/zookeeper-docker/blob/181e5862c85b517e4599d79eb5c2c7339e60a4aa/3.8.1/Dockerfile#L1> > ) > > 2. Only 3.4.0 (dropping prefix v). It will help us transition to the new > tags with less confusion and also consider DOI suggestions > <https://github.com/docker-library/official-images/issues/14506>. > > Please feel free to share your ideas. > >
