I am trying to get my head around using Spark on Yarn from a perspective of a cluster. I can start a Spark Shell no issues in Yarn. Works easily. This is done in yarn-client mode and it all works well.
In multiple examples, I see instances where people have setup Spark Clusters in Stand Alone mode, and then in the examples they "connect" to this cluster in Stand Alone mode. This is done often times using the spark:// string for connection. Cool. s But what I don't understand is how do I setup a Yarn instance that I can "connect" to? I.e. I tried running Spark Shell in yarn-cluster mode and it gave me an error, telling me to use yarn-client. I see information on using spark-class or spark-submit. But what I'd really like is a instance I can connect a spark-shell too, and have the instance stay up. I'd like to be able run other things on that instance etc. Is that possible with Yarn? I know there may be long running job challenges with Yarn, but I am just testing, I am just curious if I am looking at something completely bonkers here, or just missing something simple. Thanks!
