On Wed, Jan 10, 2018 at 2:30 PM, Yinan Li <liyinan...@gmail.com> wrote: > 1. Retries of init-containers are automatically supported by k8s through pod > restart policies. For this point, sorry I'm not sure how spark-submit > achieves this.
Great, add that feature to spark-submit, everybody benefits, not just k8s. > 2. The ability to use credentials that are not shared with the main > containers. Not sure what that achieves. > 3. Not only the user code, but Spark internal code like Executor won't be > run if the init-container fails. Not sure what that achieves. Executor will fail if dependency download fails, Spark driver will recover (and start a new executor if needed). > 4. Easier to build tooling around k8s events/status of the init-container in > case of failures as it's doing exactly one thing: downloading dependencies. Again, I don't see what is all this hoopla about fine grained control of dependency downloads. Spark solved this years ago for Spark applications. Don't reinvent the wheel. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
