Hey, I noticed in MESOS-336 that there was some discussion on how to cache the Mesos executor so it does not need to be repeatedly fetched from HDFS. This parallels a problem faced by users of Aurora which is how to fetch binaries needed for tasks. Twitter mitigated this problem by caching fetched binaries from HDFS on the slave file system and having the first process of each task fetch binaries from the cache if possible. If it is not possible to fetch it from the cache, the process places the binary in the cache for subsequent task starts on the same slave.
The code that does this and a brief explanation on how it works can be found in this gist: https://gist.github.com/zmanji/f41df77510ef9d00265a. I hope it serves as a good example on how this problem can be mitigated. -- Zameer Manji
