Thank you for your answer. The think it I actually pointed to a local binary file. And Mesos locally copied the binary file to a specific folder in /var/lib/mesos/... and extract it to every time it launched an Spark executor. With the fetch cache, the copy time is reduced, but the reduction is not much since the file is stored at local any way. The process that takes more time is the extraction. Finally, since Mesos make a new folder for extracting the Spark binary each time a new Spark job runs, the disk usage increases gradually.
Therefore, our expectation is to have Spark running on Mesos without this binary extraction, as well as without storing the same binary every time new Spark job runs. Does that make sense to you? And do you have any idea how to deal with this? Best -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
