Hi friends,
Please note that you cannot use any docker image in Google Kubernetes cluster. It needs to be in a given format! Look at this article of mine mentioned below for building docker images. Try to standardize the image with The version of Spark, The Scala version, The Java version, Packages to be deployed for PySpark For GCP it needs to start with the correct format <https://cloud.google.com/container-registry/docs/overview>, example IMAGEGCP="eu.gcr.io/<GCP PROJECT>/ spark-py:3.1.1-scala_2.12-8-jre-slim-buster-addedpackages" and then put it in GCP container registry <https://cloud.google.com/container-registry> and then you can use it as below in spark-submit --conf spark.kubernetes.driver.docker.image=${IMAGEGCP} \ --conf spark.kubernetes.executor.docker.image=${IMAGEGCP} \ --conf spark.kubernetes.container.image=${IMAGEGCP} \ Spark on Kubernetes, A Practitioner’s Guide <https://www.linkedin.com/pulse/spark-kubernetes-practitioners-guide-mich-talebzadeh-ph-d-/> HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
