Hi, I've been toying with persistent volumes using Spark 3.1.1 on minikube and am wondering whether it's a supported platform. I'd not be surprised if not given all the surprises I've been experiencing lately.
Can I use spark-shell or any Spark app in client mode with PVCs with the default 2 executors? Should the following work if I removed --num-executors 1? ./bin/spark-shell \ --master k8s://$K8S_SERVER \ --num-executors 1 \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$PVC_CLAIM_NAME \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.storageClass=$PVC_STORAGE_CLASS \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.sizeLimit=$PVC_SIZE_LIMIT \ --conf spark.kubernetes.container.image=$IMAGE_NAME \ --conf spark.kubernetes.context=minikube \ --conf spark.kubernetes.namespace=spark-demo \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --verbose The whole demo is available at https://jaceklaskowski.github.io/spark-kubernetes-book/demo/persistentvolumeclaims/ Please help. Thank you! Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski>
