spark-submit with Yarn

[Arun Ahuja]

Is there more documentation on using spark-submit with Yarn?  Trying to
launch a simple job does not seem to work.

My run command is as follows:

/opt/cloudera/parcels/CDH/bin/spark-submit \
    --master yarn \
    --deploy-mode client \
    --executor-memory 10g \
    --driver-memory 10g \
    --num-executors 50 \
    --class $MAIN_CLASS \
    --verbose \
    $JAR \
    "$@"

The verbose logging correctly parses the arguments:

System properties:
spark.executor.memory -> 10g
spark.executor.instances -> 50
SPARK_SUBMIT -> true
spark.master -> yarn-client


But when I view the job 4040 page, SparkUI, there is a single executor
(just the driver node) and I see the following in enviroment

spark.master - local[24]

Also, when I run with yarn-cluster, how can I access the SparkUI page?

Thanks,
Arun