Hello Holden, Could you please provide more details and plan for docker images support?
So far I see that there are only two tags, I get from them spark version, but there is no information about java, hadoop, scala versions. Also there is no description on docker hub, probably it would be nice to put a link to Docker files in github repository. What directories are expected to be mounted and ports forwarded? How can I mount the krb5.conf file and directory where my kerberos ticket is located? I've pulled the docker image with tag spark 3.2.1 and I see that there is java 11 and hadoop 3.3, but our environment requires us to have other versions. On Tue, 22 Feb 2022 at 16:29, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Well that is just a recommendation. > > The onus is on me the user to download and go through dev and test running > suite of batch jobs to ensure that all work ok, especially on the edge, > sign the release off and roll it in out into production. It won’t be > prudent otherwise. > > HHH > > On Tue, 22 Feb 2022 at 12:12, Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> "Spark 3.1.3 is a maintenance release containing stability fixes. This >> release is based on the branch-3.1 maintenance branch of Spark. We strongly >> recommend all 3.1.3 users to upgrade to this stable release." >> https://spark.apache.org/releases/spark-release-3-1-3.html >> >> Do we have another 3.13 or do we strongly recommend all 3.1.2 users to >> upgrade to this stable release ? >> >> tir. 22. feb. 2022 kl. 09:50 skrev angers zhu <angers....@gmail.com>: >> >>> Hi, seems >>> >>> - [SPARK-35391] <https://issues.apache.org/jira/browse/SPARK-36339>: >>> Memory leak in ExecutorAllocationListener breaks dynamic allocation under >>> high load >>> >>> Links to wrong jira ticket? >>> >>> Mich Talebzadeh <mich.talebza...@gmail.com> 于2022年2月22日周二 15:49写道： >>> >>>> Well, that is pretty easy to do. >>>> >>>> However, a quick fix for now could be to retag the image created. It is >>>> a small volume which can be done manually for now. For example, I just >>>> downloaded v3.1.3 >>>> >>>> >>>> docker image ls >>>> >>>> REPOSITORY TAG >>>> IMAGE ID CREATED SIZE >>>> >>>> apache/spark v3.1.3 >>>> 31ed15daa2bf 12 hours ago 531MB >>>> >>>> Retag it with >>>> >>>> >>>> docker tag 31ed15daa2bf >>>> apache/spark/tags/spark-3.1.3-scala_2.12-8-jre-slim-buster >>>> >>>> docker image ls >>>> >>>> REPOSITORY TAG >>>> IMAGE ID CREATED >>>> SIZE >>>> >>>> apache/spark/tags/spark-3.1.3-scala_2.12-8-jre-slim-buster latest >>>> 31ed15daa2bf 12 hours ago >>>> 531MB >>>> >>>> Then push it with (example) >>>> >>>> docker push apache/spark/tags/spark-3.1.3-scala_2.12-8-jre-slim-buster >>>> >>>> >>>> HTH >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> On Mon, 21 Feb 2022 at 23:51, Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> Yeah I think we should still adopt that naming convention, however no >>>>> one has taken the time submit write a script to do it yet so until we get >>>>> that script merged I think we'll just have one build. I can try and do >>>>> that >>>>> for the next release but it would be a great 2nd issue for someone getting >>>>> more familiar with the release tooling. >>>>> >>>>> On Mon, Feb 21, 2022 at 2:18 PM Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Ok thanks for the correction. >>>>>> >>>>>> The docker pull line shows as follows: >>>>>> >>>>>> docker pull apache/spark:v3.2.1 >>>>>> >>>>>> >>>>>> So this only tells me the version of Spark 3.2.1 >>>>>> >>>>>> >>>>>> I thought we discussed deciding on the docker naming conventions in >>>>>> detail, and broadly agreed on what needs to be in the naming convention. >>>>>> For example, in this thread: >>>>>> >>>>>> >>>>>> Time to start publishing Spark Docker Images? - >>>>>> mich.talebza...@gmail.com - Gmail (google.com) >>>>>> <https://mail.google.com/mail/u/0/?hl=en-GB#search/publishing/FMfcgzGkZQSzbXWQDWfddGDNRDQfPCpg> >>>>>> dated >>>>>> 22nd July 2021 >>>>>> >>>>>> >>>>>> Referring to that, I think the broad agreement was that the docker >>>>>> image name should be of the form: >>>>>> >>>>>> >>>>>> The name of the file provides: >>>>>> >>>>>> - Built for spark or spark-py (PySpark) spark-r >>>>>> - Spark version: 3.1.1, 3.1.2, 3.2.1 etc. >>>>>> - Scala version; 2.1.2 >>>>>> - The OS version based on JAVA: 8-jre-slim-buster, >>>>>> 11-jre-slim-buster meaning JAVA 8 and JAVA 11 respectively >>>>>> >>>>>> I believe it is a good thing and we ought to adopt that convention. >>>>>> For example: >>>>>> >>>>>> >>>>>> spark-py-3.2.1-scala_2.12-11-jre-slim-buster >>>>>> >>>>>> >>>>>> HTH >>>>>> >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, 21 Feb 2022 at 21:58, Holden Karau <hol...@pigscanfly.ca> >>>>>> wrote: >>>>>> >>>>>>> My bad, the correct link is: >>>>>>> >>>>>>> https://hub.docker.com/r/apache/spark/tags >>>>>>> >>>>>>> On Mon, Feb 21, 2022 at 1:17 PM Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> well that docker link is not found! may be permission issue >>>>>>>> >>>>>>>> [image: image.png] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>> >>>>>>>> >>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>>> for any loss, damage or destruction of data or any other property >>>>>>>> which may >>>>>>>> arise from relying on this email's technical content is explicitly >>>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>>> damages >>>>>>>> arising from such loss, damage or destruction. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, 21 Feb 2022 at 21:09, Holden Karau <hol...@pigscanfly.ca> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> We are happy to announce the availability of Spark 3.1.3! >>>>>>>>> >>>>>>>>> Spark 3.1.3 is a maintenance release containing stability fixes. >>>>>>>>> This >>>>>>>>> release is based on the branch-3.1 maintenance branch of Spark. We >>>>>>>>> strongly >>>>>>>>> recommend all 3.1 users to upgrade to this stable release. >>>>>>>>> >>>>>>>>> To download Spark 3.1.3, head over to the download page: >>>>>>>>> https://spark.apache.org/downloads.html >>>>>>>>> >>>>>>>>> To view the release notes: >>>>>>>>> https://spark.apache.org/releases/spark-release-3-1-3.html >>>>>>>>> >>>>>>>>> We would like to acknowledge all community members for >>>>>>>>> contributing to this >>>>>>>>> release. This release would not have been possible without you. >>>>>>>>> >>>>>>>>> *New Dockerhub magic in this release:* >>>>>>>>> >>>>>>>>> We've also started publishing docker containers to the Apache >>>>>>>>> Dockerhub, >>>>>>>>> these contain non-ASF artifacts that are subject to different >>>>>>>>> license terms than the >>>>>>>>> Spark release. The docker containers are built for Linux x86 and >>>>>>>>> ARM64 since that's >>>>>>>>> what I have access to (thanks to NV for the ARM64 machines). >>>>>>>>> >>>>>>>>> You can get them from https://hub.docker.com/apache/spark (and >>>>>>>>> spark-r and spark-py) :) >>>>>>>>> (And version 3.2.1 is also now published on Dockerhub). >>>>>>>>> >>>>>>>>> Holden >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> >>>> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4 >> <https://www.google.com/maps/search/Vestre+Aspehaug+4?entry=gmail&source=g>, >> 6010 Ålesund >> Norge >> >> +47 480 94 297 >> > -- > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > -- //with Best Regards --Denis Bolshakov e-mail: bolshakov.de...@gmail.com
