we have a spark cluster running on with multiple users... when running with the user owning the cluster jobs run fine... however, when trying to run pyspark with a different user it fails because the .sparkStaging/application_* is written with 700 so the user cannot write to that directory....
how to configure spark/yarn cluster so .sparkStaging is written as 770 instead of 700 so the users with shared group can execute?
