[
https://issues.apache.org/jira/browse/SPARK-57878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-57878:
------------------------------
Description:
The scheduler hard-coded a single placement policy (random shuffle), only
overridable through a protected test hook. Different workloads want different
placement:
- binpack concentrates tasks so idle executors can be reclaimed by dynamic
allocation / decommissioning — useful for cost efficiency on elastic clusters.
- balance spreads tasks to maximize parallelism and avoid hot-spotting a single
executor.
Making the policy pluggable and config-driven lets operators pick per workload
without code changes, while keeping the existing shuffle behavior as the
default.
> Add task assignment strategy to support binpack/balance task scheduling
> -----------------------------------------------------------------------
>
> Key: SPARK-57878
> URL: https://issues.apache.org/jira/browse/SPARK-57878
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.3.0
> Reporter: XiDuo You
> Priority: Major
>
> The scheduler hard-coded a single placement policy (random shuffle), only
> overridable through a protected test hook. Different workloads want different
> placement:
> - binpack concentrates tasks so idle executors can be reclaimed by dynamic
> allocation / decommissioning — useful for cost efficiency on elastic clusters.
> - balance spreads tasks to maximize parallelism and avoid hot-spotting a
> single executor.
> Making the policy pluggable and config-driven lets operators pick per
> workload without code changes, while keeping the existing shuffle behavior as
> the default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
