Temp directory used by spark-submit

Hello, I notice that when I run spark-submit, a temporary directory containing all the jars and resource files is created under /tmp (for example, /tmp/spark-fd1b77fc-50f4-4b1c-a122-5cf36969407c). Sometimes this directory gets cleanup after the job, but sometimes it doesn't, which fills up my roo

Re: Temp directory used by spark-submit

After setting SPARK_LOCAL_DIRS/SPARK_WORKER_DIR you need to restart your spark instances (stop-all.sh and start-all.sh), You can also try setting java.io.tmpdir while creating the SparkContext. Thanks Best Regards On Wed, Mar 11, 2015 at 1:47 AM, Justin Yip wrote: > Hello, > > I notice that whe