Yes, I have tried that by adding it to the Worker. I can see the "app-20140328124540-000” in the local spark directory of the worker.
But the “spark-local” directories are always written to /tmp since is the default spark.local.dir is taken from java.io.tempdir? On 28 Mar, 2014, at 12:42 pm, Matei Zaharia <matei.zaha...@gmail.com> wrote: > Yes, the problem is that the driver program is overriding it. Have you set it > manually in the driver? Or how did you try setting it in workers? You should > set it by adding > > export SPARK_JAVA_OPTS=“-Dspark.local.dir=whatever” > > to conf/spark-env.sh on those workers. > > Matei > > On Mar 27, 2014, at 9:04 PM, Tsai Li Ming <mailingl...@ltsai.com> wrote: > >> Anyone can help? >> >> How can I configure a different spark.local.dir for each executor? >> >> >> On 23 Mar, 2014, at 12:11 am, Tsai Li Ming <mailingl...@ltsai.com> wrote: >> >>> Hi, >>> >>> Each of my worker node has its own unique spark.local.dir. >>> >>> However, when I run spark-shell, the shuffle writes are always written to >>> /tmp despite being set when the worker node is started. >>> >>> By specifying the spark.local.dir for the driver program, it seems to >>> override the executor? Is there a way to properly define it in the worker >>> node? >>> >>> Thanks! >> >
