[
https://issues.apache.org/jira/browse/SPARK-58879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved SPARK-58879.
------------------------------
Fix Version/s: 4.3.0
5.0.0
Resolution: Fixed
Issue resolved by pull request 58131
[https://github.com/apache/spark/pull/58131]
> Graceful dynamic allocation can decommission executors with newly assigned
> tasks
> --------------------------------------------------------------------------------
>
> Key: SPARK-58879
> URL: https://issues.apache.org/jira/browse/SPARK-58879
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
> Affects Versions: 3.5.0, 4.0.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0, 5.0.0
>
>
> h3. Problem
> SPARK-9552 protected ordinary dynamic-allocation scale-down from removing an
> executor that became busy after it was identified as idle. The
> graceful-decommission path introduced by SPARK-31198 does not perform the
> equivalent check. When both {{spark.dynamicAllocation.enabled}} and
> {{spark.decommission.enabled}} are enabled, an idle-timeout candidate can
> therefore be accepted for decommissioning even after the scheduler has
> assigned it new work.
> h3. Example interleaving
> # The allocation manager identifies executor E as an idle-timeout candidate.
> # A scheduler offer assigns a task to E. {{TaskSchedulerImpl}} records the
> assignment before the backend sends {{LaunchTask}}.
> # The allocation manager requests graceful decommissioning of E.
> {{CoarseGrainedSchedulerBackend.decommissionExecutors}} checks that E is
> active, but does not recheck {{scheduler.isExecutorBusy(E)}}.
> # E is marked for decommissioning even though it now has assigned work. The
> task can reach or continue on E after decommissioning starts.
> This can cause unnecessary task retries. With storage decommissioning
> enabled, a task that tries to save a non-broadcast block after BlockManager
> decommissioning begins can fail. This is a core scheduler race; it does not
> depend on Spark Connect or a particular Kubernetes pod allocator.
> h3. Expected behavior and proposed fix
> Ordinary idle scale-down should recheck the scheduler's current task
> assignments atomically with making an executor unavailable for further
> offers. Add an idle-only graceful-decommission entry point and use it from
> the core allocation manager. In the coarse-grained backend, hold the existing
> scheduler/backend locks while filtering active, idle executors and invoking
> the existing decommission path. If assignment wins, scale-down must reject
> that executor; if decommissioning wins, later offers must exclude it.
> Preserve the existing unconditional path for explicit and
> infrastructure-initiated drains.
> h3. Public source and related work
> The issue is present in released Spark 3.5.0 and 4.0.0 and in current master
> at {{75ac74bafec1f489224765041a19bec6bad36a9d}}.
> * [Graceful admission checks active state
> only|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L559-L575]
> * [Non-forced kill already checks whether the executor is
> busy|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L969-L974]
> * [Core allocation manager's graceful
> path|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L583-L590]
> * [BlockManager write
> guard|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L395-L400]
> Related: SPARK-9552 and SPARK-31198.
> A deterministic regression test should pause after a real scheduler has
> assigned a task but before {{LaunchTask}} is delivered, then attempt idle
> decommissioning. It should also exercise the reverse ordering and confirm
> that explicit decommissioning of a busy executor remains supported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
