Re: Changing number of workers for benchmarking purposes

I think "SPARK_WORKER_INSTANCES" is deprecated. This should work: "export SPARK_EXECUTOR_INSTANCES=2" -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Changing-number-of-workers-for-benchmarking-purposes-tp2606p26491.html Sent from t

Re: Changing number of workers for benchmarking purposes

umber-of-workers-for-benchmarking-purposes-tp2606p26488.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: u

Re: Changing number of workers for benchmarking purposes

This will create 3 worker processes on every slave machine. - Kalpit -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Changing-number-of-workers-for-benchmarking-purposes-tp2606p4169.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Changing number of workers for benchmarking purposes

How about hacking your way around it. Start with max workers & keep killing them off after each run. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Thu, Mar 13, 2014 at 2:00 AM, Pierre Borckmans < pierre.borckm...@realim

Re: Changing number of workers for benchmarking purposes

Thanks Patrick. I could try that. But the idea was to be able to write a fully automated benchmark, varying the dataset size, the number of workers, the memory, … without having to stop/start the cluster each time. I was thinking something like SparkConf.set(“spark.max_number_workers”, n) wou

Re: Changing number of workers for benchmarking purposes

One related question. Is there any way to automatically determine the optimal # of workers in yarn based on the data size, and available resources without explicitly specifying it when the job is lunched? Thanks. Sincerely, DB Tsai Machine Learning Engineer Alpine Data Labs -

Re: Changing number of workers for benchmarking purposes

Hey Pierre, Currently modifying the "slaves" file is the best way to do this because in general we expect that users will want to launch workers on any slave. I think you could hack something together pretty easily to allow this. For instance if you modify the line in slaves.sh from this: for

Changing number of workers for benchmarking purposes

Hi there! I was performing some tests for benchmarking purposes, among other things to observe the evolution of the performances versus the number of workers. In that context, I was wondering if there is any easy way to choose the number of workers to be used in standalone mode, without having