Actually I faced it yesterday... I had to put it in spark-env.sh and take it out from spark-defaults.conf on 1.0.1...Note that this settings should be visible on all workers..
After that I validated that SPARK_LOCAL_DIRS was indeed getting used for shuffling... On Thu, Aug 14, 2014 at 10:27 AM, Brad Miller <bmill...@eecs.berkeley.edu> wrote: > Hi All, > > I'm having some trouble setting the disk spill directory for spark. The > following approaches set "spark.local.dir" (according to the "Environment" > tab of the web UI) but produce the indicated warnings: > > *In spark-env.sh:* > export SPARK_JAVA_OPTS=-Dspark.local.dir=/spark/spill > *Associated warning:* > 14/08/14 10:10:39 WARN SparkConf: In Spark 1.0 and later spark.local.dir > will be overridden by the value set by the cluster manager (via > SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN). > 14/08/14 10:10:39 WARN SparkConf: > SPARK_JAVA_OPTS was detected (set to '-Dspark.local.dir=/spark/spill'). > This is deprecated in Spark 1.0+. > Please instead use... > > *In spark-defaults.conf:* > spark.local.dir /spark/spill > *Associated warning:* > 14/08/14 10:09:04 WARN SparkConf: In Spark 1.0 and later spark.local.dir > will be overridden by the value set by the cluster manager (via > SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN). > > The following does not produce any warnings, but also produces no sign of > actually setting "spark.local.dir": > > *In spark-env.sh:* > export SPARK_LOCAL_DIRS=/spark/spill > > Does anybody know whether SPARK_LOCAL_DIRS actually works as advertised, > or if I am perhaps using it incorrectly? > > best, > -Brad >
