> With a config value set by the submission code, like what I'm doing to > prevent client mode submission in my p.o.c.?
The contract for what determines the appropriate scheduler backend to
instantiate is then going to be different in Kubernetes versus the other
cluster managers. The cluster manager typically only picks the scheduler
backend implementation based on the master URL format plus the deploy mode.
Perhaps this is an acceptable tradeoff for being able to leverage spark-submit
in the cluster mode deployed driver container. Again though, any flag we expose
in spark-submit is a user-facing option that can be set erroneously, which is a
practice we shouldn’t be encouraging.
Taking a step back though, I think we want to use spark-submit’s internals
without using spark-submit itself. Any flags we add to spark-submit are
user-facing. We ideally would be able to extract the dependency download + run
user main class subroutines from spark-submit, and invoke that in all of the
cluster managers. Perhaps this calls for a refactor in spark-submit itself to
make some parts reusable in other contexts. Just an idea.
On 1/10/18, 1:38 PM, "Marcelo Vanzin" <van...@cloudera.com> wrote:
On Wed, Jan 10, 2018 at 1:33 PM, Matt Cheah <mch...@palantir.com> wrote:
> If we use spark-submit in client mode from the driver container, how do
we handle needing to switch between a cluster-mode scheduler backend and a
client-mode scheduler backend in the future?
With a config value set by the submission code, like what I'm doing to
prevent client mode submission in my p.o.c.?
There are plenty of solutions to that problem if that's what's worrying you.
> Something else re: client mode accessibility – if we make client mode
accessible to users even if it’s behind a flag, that’s a very different
contract from needing to recompile spark-submit to support client mode. The
amount of effort required from the user to get to client mode is very different
between the two cases
Yes. But if we say we don't support client mode, we don't support
client mode regardless of how easy it is for the user to fool Spark
into trying to run in that mode.
--
Marcelo
smime.p7s
Description: S/MIME cryptographic signature
