Hi, I think I found it. I should be using OnDemand claim name so it gets replaced to be unique per executor (?)
Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Mon, Mar 15, 2021 at 8:36 PM Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > I've been toying with persistent volumes using Spark 3.1.1 on minikube and > am wondering whether it's a supported platform. I'd not be surprised if not > given all the surprises I've been experiencing lately. > > Can I use spark-shell or any Spark app in client mode with PVCs with the > default 2 executors? Should the following work if I removed --num-executors > 1? > > ./bin/spark-shell \ > --master k8s://$K8S_SERVER \ > --num-executors 1 \ > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH > \ > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$PVC_CLAIM_NAME > \ > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.storageClass=$PVC_STORAGE_CLASS > \ > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.sizeLimit=$PVC_SIZE_LIMIT > \ > --conf spark.kubernetes.container.image=$IMAGE_NAME \ > --conf spark.kubernetes.context=minikube \ > --conf spark.kubernetes.namespace=spark-demo \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ > --verbose > > The whole demo is available at > https://jaceklaskowski.github.io/spark-kubernetes-book/demo/persistentvolumeclaims/ > > Please help. Thank you! > > Pozdrawiam, > Jacek Laskowski > ---- > https://about.me/JacekLaskowski > "The Internals Of" Online Books <https://books.japila.pl/> > Follow me on https://twitter.com/jaceklaskowski > > <https://twitter.com/jaceklaskowski> >
