owning the cluster jobs run fine... however, when
> trying to run pyspark with a different user it fails because the
> .sparkStaging/application_* is written with 700 so the user cannot write to
> that directory
>
> how to configure spark/yarn cluster so .sparkStaging is wr
we have a spark cluster running on with multiple users...
when running with the user owning the cluster jobs run fine... however,
when trying to run pyspark with a different user it fails because the
.sparkStaging/application_* is written with 700 so the user cannot write to
that directory
Any feedback please?
Thanks,
Debu
Sent from my iPhone
> On 13-Feb-2020, at 6:36 PM, Debabrata Ghosh wrote:
>
>
> Greetings All !
>
> I have got plenty of application directories lying around sparkStaging , such
> as .sparkStaging/application_1580703507814_0074
>
&g
Greetings All !
I have got plenty of application directories lying around sparkStaging ,
such as .sparkStaging/application_1580703507814_0074
Would you please be able to help advise me which variable I need to set in
spark-env.sh so that the sparkStaging applications aren't preserved afte
in spark 1.0.0 using yarn-client mode i am seeing that the sparkStaging
directories do not get cleaned up.
for example i run:
$ spark-submit --class org.apache.spark.examples.SparkPi
spark-examples-1.0.0-hadoop2.3.0-cdh5.0.2.jar 10
after which i have this directory left behind with one file in
t: 5f48721, github.com/apache/spark/pull/586
From: alee...@hotmail.com
To: user@spark.apache.org
Subject: RE: HDFS folder .sparkStaging not deleted and filled up HDFS in yarn
mode
Date: Wed, 18 Jun 2014 11:24:36 -0700
Forgot to mention that I am using spark-submit to submit jobs, and a verbose
Forgot to mention that I am using spark-submit to submit jobs, and a verbose
mode print out looks like this with the SparkPi examples.The .sparkStaging
won't be deleted. My thoughts is that this should be part of the staging and
should be cleaned up as well when sc gets termi
Hi All,
Have anyone ran into the same problem? By looking at the source code in
official release (rc11),this property settings is set to false by default,
however, I'm seeing the .sparkStaging folder remains on the HDFS and causing it
to fill up the disk pretty fast since SparkContext de
