Hi! I've been using Spark compiled from 1.0 branch at some point (~2 month ago). The setup is a standalone cluster with 4 worker machines and 1 master machine. I used to run spark shell like this:
./bin/spark-shell -c 30 -em 20g -dm 10g Today I've finally updated to Spark 1.0 release. Now I can only run spark shell like this: ./bin/spark-shell --master spark://10.2.1.5:7077 --total-executor-cores 30 --executor-memory 20g --driver-memory 10g The documentation at http://spark.apache.org/docs/latest/spark-standalone.html says: "You can also pass an option --cores <numCores> to control the number of cores that spark-shell uses on the cluster." This doesn't work, you need to pass "--total-executor-cores <numCores>" instead. "Note that if you are running spark-shell from one of the spark cluster machines, the bin/spark-shell script will automatically set MASTER from the SPARK_MASTER_IP and SPARK_MASTER_PORT variables in conf/spark-env.sh." This is not working for me too. I run the shell from the master machine, and I do have SPARK_MASTER_IP set up in conf/spark-env.sh like this: export SPARK_MASTER_IP='10.2.1.5' But if I omit "--master spark://10.2.1.5:7077" then the console starts but I can't see it in the UI at http://10.2.1.5:8080. But when I go to http://10.2.1.5:4040 (the application UI) I see that the app is using only master as a worker. My question is: are those just the bugs in the documentation? That there is no --cores option and that SPARK_MASTER_IP is not used anymore when I run the Spark shell from the master? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/issues-with-bin-spark-shell-for-standalone-mode-tp9107.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
