Re: docker image distribution in Kubernetes cluster

age for that versus the executor image. >>> >>> >>> >>> As a practical example for a ML use case you might want to have the >>> optional Intel MKL or OpenBLAS dependencies which can significantly bloat >>> the size of your container image (by hundre

Re: docker image distribution in Kubernetes cluster

; >> As a practical example for a ML use case you might want to have the >> optional Intel MKL or OpenBLAS dependencies which can significantly bloat >> the size of your container image (by hundreds of megabytes) and would only >> be needed by the executor pods. >>

Re: docker image distribution in Kubernetes cluster

lebzadeh > *Date: *Wednesday, 8 December 2021 at 17:42 > *To: *Khalid Mammadov > *Cc: *"user @spark" , Spark dev list < > dev@spark.apache.org> > *Subject: *Re: docker image distribution in Kubernetes cluster > > > > Thanks Khalid for your notes > >

Re: docker image distribution in Kubernetes cluster

t Subject: Re: docker image distribution in Kubernetes cluster Thanks Khalid for your notes I have not come across a use case where the docker version on the driver and executors need to be different. My thinking is that spark.kubernetes.executor.container.image is the correct reference

Re: docker image distribution in Kubernetes cluster

Thanks Khalid for your notes I have not come across a use case where the docker version on the driver and executors need to be different. My thinking is that spark.kubernetes.executor.container.image is the correct reference as in the Kubernetes where container is the correct terminology and al

Re: docker image distribution in Kubernetes cluster

Hi Mitch IMO, it's done to provide most flexibility. So, some users can have limited/restricted version of the image or with some additional software that they use on the executors that is used during processing. So, in your case you only need to provide the first one since the other two configs

Re: docker image distribution in Kubernetes cluster

Just a correction that in Spark 3.2 documentation it states that Property NameDefaultMeaning spark.kubernetes.container.image (none) Container image to use for the Spark application. This is usually of the form example