Hi all, I am using a Spark standalone cluster (open source) with the root user as the Spark user. I am facing an issue when using the Overwrite mode in Spark write parquet. The jobs are failing because the user is unable to delete the tmp folder within the parquet folder.
I plan to create a Spark service account and use it instead of root. Could you share the steps I need to follow? Specifically: 1. Do I need to assign the service account to the SPARK_USER environment variable? 2. Is updating spark-defaults.conf enough? Thanks, Elango
