Hi, I'm trying to get the gist of clientMode input parameter for RpcEnv.create [1]. It is disabled (i.e. false) by default.
I've managed to find out that, in the "general" case, it's enabled for executors and disabled for the driver. (it's also used for Spark Standalone's master and workers but it's infra and I'm not interested in exploring it currently). I've however noticed that in YARN the clientMode parameter means something more, i.e. whether the Spark application runs in client or cluster deploy mode. In YARN my understanding of the parameter is that clientMode is enabled explicitly when Spark on YARN's ApplicationMaster creates the `sparkYarnAM` RPC Environment [2] (when executed for client deploy mode [3]) This is because in client deploy mode the driver runs on some other node and the AM acts simply as a proxy to Spark executors that run in their own YARN containers. This is (also?) because in client deploy mode in Spark on YARN we have separate JVM processes for the driver, the AM and Spark executors. The distinction is Is the last two paragraphs correct? I'd appreciate if you could fix and fill out the gaps where necessary. Thanks a lot to make it so much easier for me and...participants of my Spark workshops ;-) [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala#L42 [2] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L434 [3] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L254 Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
