Hi, It helps if Spark binaries were added to PATH in the docker images. used to be there in previous versions like 3.1.3 etc
docker run -it apache/spark:3.4.1-scala2.12-java11-r-ubuntu /bin/bash spark@e48cc28ff89e:/opt/spark/work-dir$ which spark-submit spark@e48cc28ff89e:/opt/spark/work-dir$ echo $PATH /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin spark@9440b0e46cee:/opt/spark/work-dir$ PATH=$PATH:/opt/spark/bin spark@9440b0e46cee:/opt/spark/work-dir$ which spark-submit /opt/spark/bin/spark-submit HTH Mich Talebzadeh, Solutions Architect/Engineering Lead Palantir Technologies Limited London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Thu, 20 Jul 2023 at 02:10, Yikun Jiang <yikunk...@gmail.com> wrote: > The spark Docker Official Image is now available: > https://hub.docker.com/_/spark > > $ docker run -it --rm *spark* /opt/spark/bin/spark-shell > $ docker run -it --rm *spark*:python3 /opt/spark/bin/pyspark > $ docker run -it --rm *spark*:r /opt/spark/bin/sparkR > > We had a longer review journey than we expected, if you are also > interested in this journey, you can see more in: > > https://github.com/docker-library/official-images/pull/13089 > > Thanks to everyone who helps in the Docker and Apache Spark community! > > Some background you might want to know: > *- apache/spark*: https://hub.docker.com/r/apache/spark, the Apache Spark > docker image, will be published by *Apache Spark community* when the > Apache Spark is released, no update. > *- spark*: https://hub.docker.com/_/spark, the Docker Official Image, it > will be published by the *Docker community*, keep active rebuilding for > updates and security fixes by the Docker community. > - The source repo of *apache/spark *and *spark: * > https://github.com/apache/spark-docker > > See more in: > [1] [DISCUSS] SPIP: Support Docker Official Image for Spark: > https://lists.apache.org/thread/l1793y5224n8bqkp3s6ltgkykso4htb3 > [2] [VOTE] SPIP: Support Docker Official Image for Spark: > https://lists.apache.org/thread/ro6olodm1jzdffwjx4oc7ol7oh6kshbl > [3] https://github.com/docker-library/official-images/pull/13089 > [4] > https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o/ > [5] https://issues.apache.org/jira/browse/SPARK-40513 > > Regards, > Yikun >
