|
Also, I think the jetty connector will
create a small file or directory in /tmp regardless of the
spark.local.dir
It's very small, about 10KB
Guillaume
I'm not 100% sure but I think it goes
like this :
spark.local.dir can and should be set both on the executors and
on the driver (if the driver broadcast variables, the files will
be stored in this directory)
the SPARK_WORKER_DIR is where the jars and the log output of the
executors is placed (default $SPARK_HOME/work/) and it should be
cleaned regularly
In $SPARK_HOME/logs are found the logs of the workers and master
Guillaume
Hi,
I'm confused about the -Dspark.local.dir and SPARK_WORKER_DIR(--work-dir).
What's the difference?
I have set -Dspark.local.dir for all my worker nodes but I'm still seeing directories being created in /tmp when the job is running.
I have also tried setting -Dspark.local.dir when I run the application.
Thanks!
--
|
Guillaume PITEL, Président
+33(0)6 25 48 86 80
eXenSa
S.A.S.
41, rue Périer -
92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37
05
|
--
|
Guillaume
PITEL, Président
+33(0)6 25 48 86 80
eXenSa
S.A.S.
41, rue Périer -
92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37
05
|
|
