Hi All,
can I set spark.local.dir to HDFS location instead of /tmp folder ?
I tried setting up temp folder to HDFS but it didn't worked can
spark.local.dir write to HDFS ?
.set("spark.local.dir","hdfs://namednode/spark_tmp/")
16/07/05 15:35:47 ERROR DiskBlockManager: Failed to create local dir in
hdfs://namenode/spark_tmp/. Ignoring this directory.
java.io.IOException: Failed to create a temp directory (under
hdfs://namenode/spark_tmp/) after 10 attempts!
Thanks
Sri
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-local-dir-to-HDFS-tp27291.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
