Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. It seems a directory on my local disk. But currently the temp directory is on hdfs, under the path of df.write.parquet(path)
Regard, Junfeng Chen On Wed, Dec 5, 2018 at 6:28 PM Sandip Mehta <[email protected]> wrote: > tryspark.local.dir property. > > > On Wed, Dec 5, 2018 at 1:42 PM JF Chen <[email protected]> wrote: > >> I have two spark apps writing data to one directory. I notice they share >> one temp directory, and the spark fist finish writing will clear the temp >> directory and the slower one may throw "No lease on *** File does not >> exist" error >> So how to specify the temp directory? >> Thanks! >> >> Regard, >> Junfeng Chen >> >
