suppose you have 1 job that do some transformation, suppose you have X
cores in your cluster and you are willing to give all of them to your job
suppose you have no shuffles(to keep it simple)
set number of partitions of your input data to be 3X or 2X, thus you'll get
2/3 tasks per each core
On 3
The tuning documentations tells us to have 2-3 tasks per CPU core
> In general, we recommend 2-3 tasks per CPU core in your cluster.
I’m wondering how you’d actually accomplish this.
Setting spark.task.cpus to a fraction like 0.5 or 0.3 does not work.
Perhaps I’m misunderstanding, any advice i
