On Sun, Feb 8, 2015 at 10:26 PM, java8964 <java8...@hotmail.com> wrote: > standalone one box environment, if I want to use all 48G memory allocated to > worker for my application, I should ask 48G memory for the executor in the > spark shell, right? Because 48G is too big for a JVM heap in normal case, I > can and should consider to start multi workers in one box, to lower the > executor memory, but still use all 48G memory.
Yes. > In the spark document, about the "-- cores" parameter, the default is all > available cores, so it means using all available cores in all workers, even > in the cluster environment? If so, in default case, if one client submit a > huge job, it will use all the available cores from the cluster for all the > tasks it generates? Have a look at how cores work in standalone mode: http://spark.apache.org/docs/latest/job-scheduling.html > One thing is still not clear is in the given example I have, if 10 tasks (1 > per partition) will execute, but there is one executor per application, in > this case, I have the following 2 questions, assuming that the worker memory > is set to 48G, and executor memory is set to 4G, and I use one spark-shell > to connect to the master to submit my application: > > 1) How many executor will be created on this box (Or even in the cluster it > it is running in the cluster)? I don't see any spark configuration related > to set number of executor in spark shell. If it is more than one, how this > number is calculated? Again from http://spark.apache.org/docs/latest/job-scheduling.html for standalone mode the default should be 1 executor per worker, but you can change that. > 2) Do you mean that one partition (or one task for it) will be run by one > executor? Is that one executor will run the task sequentially, but job > concurrency comes from that multi executors could run synchronous, right? A partition maps to a task, which is computed serially. Tasks are executed in parallel in an executor, which can execute many tasks at once. No, parallelism does not (only) come from running many executors. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
