I think I understand what you're saying, but I think whether you're
"over-provisioning" or not depends on the nature of your workload, your
system's resources, and how Spark determines how to spawn task threads
inside executor processes.
As I concluded in the post, if you're doing CPU-bound work,
IMHO, you are making mistake.
spark manages tasks and cores internally. when you open new threads inside
executor - meaning you "over-provisioning" executor(e.g. tasks on other
cores will be preempted)
On 26 January 2016 at 07:59, Elango Cheran wrote:
> Hi everyone,
> I've gone through the eff
