are you setting '-Dspark.local.dir=/mytemp/mytempsubdir' ?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-temp-dir-spark-local-dir-tp2643p5508.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
spark.local.dir can and
should be set both on the executors and on the driver
(if the driver broadcast variables, the files will be
stored in this directory)
>> spark.local.dir can and should be set both on the executors and on the
>> driver (if the driver broadcast variables, the files will be stored in this
>> directory)
Do you mean the worker nodes?
Don’t think they are jetty connectors and the directories are empty:
/tmp/spark-3e330cdc-7540-4313-
Also, I think the jetty connector will
create a small file or directory in /tmp regardless of the
spark.local.dir
It's very small, about 10KB
Guillaume
I'm not 100% sure but I think it goes
like this :
I'm not 100% sure but I think it goes
like this :
spark.local.dir can and should be set both on the executors and on
the driver (if the driver broadcast variables, the files will be
stored in this directory)
the SPARK_WORKER_DIR is where the j
Hi,
I'm confused about the -Dspark.local.dir and SPARK_WORKER_DIR(--work-dir).
What's the difference?
I have set -Dspark.local.dir for all my worker nodes but I'm still seeing
directories being created in /tmp when the job is running.
I have also tried setting -Dspark.local.dir when I run the
