One thing to note is you may need to have the S3 credentials in the init-container unless you use a publicly accessible URL. If this is the case, you can either create a Kubernetes secret and use the Spark config option for mounting secrets (secrets will be mounted into the init-container as well as into the main container), or you create a custom init-container with the credentials baked in.
Yinan On Thu, Mar 8, 2018 at 12:05 PM, Anirudh Ramanathan < ramanath...@google.com.invalid> wrote: > You don't need to create the init-container. It's an implementation detail. > If you provide a remote uri, and specify > spark.kubernetes.container.image=<spark-image>, > Spark *internally* will add the init container to the pod spec for you. > *If *for some reason, you want to customize the init container image, you > can choose to do that using the specific options, but I don't think this is > necessary in most scenarios. The init container image, driver and executor > images can be identical by default. > > > On Thu, Mar 8, 2018 at 6:52 AM purna pradeep <purna2prad...@gmail.com> > wrote: > >> Im trying to run spark-submit to kubernetes cluster with spark 2.3 docker >> container image >> >> The challenge im facing is application have a mainapplication.jar and >> other dependency files & jars which are located in Remote location like AWS >> s3 ,but as per spark 2.3 documentation there is something called kubernetes >> init-container to download remote dependencies but in this case im not >> creating any Podspec to include init-containers in kubernetes, as per >> documentation Spark 2.3 spark/kubernetes internally creates Pods >> (driver,executor) So not sure how can i use init-container for spark-submit >> when there are remote dependencies. >> >> https://spark.apache.org/docs/latest/running-on-kubernetes. >> html#using-remote-dependencies >> >> Please suggest >> > > > -- > Anirudh Ramanathan >
