Hi
I am working with a Cloudera 5 cluster with 192 nodes and can’t work out how to
get the spark repo to use more than 2 nodes in an interactive session.
So, this works, but is non-interactive (using yarn-client as MASTER)
/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/spark/bin/spark-class \
org.apache.spark.deploy.yarn.Client \
--jar
/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/spark/examples/lib/spark-examples_2.10-0.9.0-cdh5.0.0.jar
\
--class org.apache.spark.examples.SparkPi \
--args yarn-standalone \
--args 10 \
--num-workers 100
There does not appear to be an (obvious?) way to get more than 2 nodes involved
from the repl.
I am running the REPL like this:
#!/bin/sh
. /etc/spark/conf.cloudera.spark/spark-env.sh
export SPARK_JAR=hdfs://nameservice1/user/spark/share/lib/spark-assembly.jar
export SPARK_WORKER_MEMORY=512m
export MASTER=yarn-client
exec $SPARK_HOME/bin/spark-shell
Now if I comment out the line with `export SPARK_JAR=…’ and run this again, I
get an error like this:
14/05/19 08:03:41 ERROR Client: Error: You must set SPARK_JAR environment
variable!
Usage: org.apache.spark.deploy.yarn.Client [options]
Options:
--jar JAR_PATH Path to your application's JAR file (required in
yarn-cluster mode)
--class CLASS_NAME Name of your application's main class (required)
--args ARGS Arguments to be passed to your application's main
class.
Mutliple invocations are possible, each will be
passed in order.
--num-workers NUM Number of workers to start (Default: 2)
[…]
But none of those options are exposed at the `spark-shell’ level.
Thanks in advance for your guidance.
Eric
