Hi All, I noticed that Spark doesn't delete local shuffle files of a lost executor in a running system(running in yarn-client mode). For long running system, this might fill up disk space in case of frequent executor failures. Can we delete these files when executor loss reported to driver?
Thanks -Nitin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Does-Spark-delete-shuffle-files-of-lost-executor-in-running-system-on-YARN-tp10755.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
