Spark-submit --conf spark.hadoop.fs.permissions.umask-mode=007
You may also set sticky bit on staging dir
Sent from my iPhone
> On 26 Feb 2021, at 03:29, Bulldog20630405 wrote:
>
>
>
> we have a spark cluster running on with multiple users...
> when running with the user owning the cluster
we have a spark cluster running on with multiple users...
when running with the user owning the cluster jobs run fine... however,
when trying to run pyspark with a different user it fails because the
.sparkStaging/application_* is written with 700 so the user cannot write to
that directory
how
