I am using the script in sbin to set it up (spark/sbin/start-all.sh). It works fine. The problem is how to configure more than one worker per node (the default is one worker only). The documentation for 1.6.1 suggested SPARK_WORKER_INSTANCES as the way to do it but the latest documentation has no option for it. The warning does not appear when I start the cluster but rather when I submit an application. I see no way to define SPARK_WORKER_INSTANCES when starting the cluster but not define it when submitting an application. Assaf.
From: Ofer Eliassaf [mailto:ofer.elias...@gmail.com] Sent: Thursday, October 06, 2016 9:07 PM To: Mendelson, Assaf Cc: user@spark.apache.org Subject: Re: spark standalone with multiple workers gives a warning The slaves should connect to the master using the scripts in sbin... You can read about it here: http://spark.apache.org/docs/latest/spark-standalone.html On Thu, Oct 6, 2016 at 6:46 PM, Mendelson, Assaf <assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote: Hi, I have a spark standalone cluster. On it, I am using 3 workers per node. So I added SPARK_WORKER_INSTANCES set to 3 in spark-env.sh The problem is, that when I run spark-shell I get the following warning: WARN SparkConf: SPARK_WORKER_INSTANCES was detected (set to '3'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --num-executors to specify the number of executors - Or set SPARK_EXECUTOR_INSTANCES - spark.executor.instances to configure the number of instances in the spark config. So how would I start a cluster of 3? SPARK_WORKER_INSTANCES is the only way I see to start the standalone cluster and the only way I see to define it is in spark-env.sh. The spark submit option, SPARK_EXECUTOR_INSTANCES and spark.executor.instances are all related to submitting the job. Any ideas? Thanks Assaf -- Regards, Ofer Eliassaf
