Hi Mikhail,
It looks like the documentation is a little out-dated. Neither is true
anymore. In general, we try to shift away from short options ("-em", "-dm"
etc.) in favor of more explicit ones ("--executor-memory",
"--driver-memory"). These options, and "--cores", refer to the arguments
passed in to org.apache.spark.deploy.Client, the now deprecated way of
launching an application in standalone clusters.
SPARK_MASTER_IP/PORT are only used for binding the Master, not to configure
which master the driver connects to. The proper way to specify this is
through "spark.master" in your config or the "--master" parameter to
spark-submit.
We will update the documentation shortly. Thanks for letting us know.
Andrew
2014-07-08 16:29 GMT-07:00 Mikhail Strebkov <streb...@gmail.com>:
> Hi! I've been using Spark compiled from 1.0 branch at some point (~2 month
> ago). The setup is a standalone cluster with 4 worker machines and 1 master
> machine. I used to run spark shell like this:
>
> ./bin/spark-shell -c 30 -em 20g -dm 10g
>
> Today I've finally updated to Spark 1.0 release. Now I can only run spark
> shell like this:
>
> ./bin/spark-shell --master spark://10.2.1.5:7077 --total-executor-cores
> 30
> --executor-memory 20g --driver-memory 10g
>
> The documentation at
> http://spark.apache.org/docs/latest/spark-standalone.html says:
>
> "You can also pass an option --cores <numCores> to control the number of
> cores that spark-shell uses on the cluster."
> This doesn't work, you need to pass "--total-executor-cores <numCores>"
> instead.
>
> "Note that if you are running spark-shell from one of the spark cluster
> machines, the bin/spark-shell script will automatically set MASTER from the
> SPARK_MASTER_IP and SPARK_MASTER_PORT variables in conf/spark-env.sh."
> This is not working for me too. I run the shell from the master machine,
> and
> I do have SPARK_MASTER_IP set up in conf/spark-env.sh like this:
> export SPARK_MASTER_IP='10.2.1.5'
> But if I omit "--master spark://10.2.1.5:7077" then the console starts
> but I
> can't see it in the UI at http://10.2.1.5:8080. But when I go to
> http://10.2.1.5:4040 (the application UI) I see that the app is using only
> master as a worker.
>
> My question is: are those just the bugs in the documentation? That there is
> no --cores option and that SPARK_MASTER_IP is not used anymore when I run
> the Spark shell from the master?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/issues-with-bin-spark-shell-for-standalone-mode-tp9107.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
