Chao Sun created SPARK-59444:
--------------------------------
Summary: Retain execution-memory task registration while
allocations wait
Key: SPARK-59444
URL: https://issues.apache.org/jira/browse/SPARK-59444
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Chao Sun
h3. Problem
ExecutionMemoryPool can remove a task from memoryForTask while an acquireMemory
call for the same task is waiting. The waiter later indexes the missing entry
and throws NoSuchElementException. This is an execution-memory
accounting/lifecycle bug, independent of any external execution engine.
h3. Minimal interleaving
Use a 1,000-byte execution pool:
# Task B acquires 900 bytes, then task A acquires the remaining 100 bytes.
# Another consumer belonging to A requests 300 bytes. It waits because A cannot
reach the minimum fair share.
# The first consumer of A releases its last 100 bytes. releaseMemory removes A
from memoryForTask, even though the other acquisition is still waiting.
# The waiter wakes and reads memoryForTask(A), which now throws
NoSuchElementException.
Expected: releasing bytes must not unregister a task with an outstanding
waiting acquisition. Once B releases 300 bytes, A should acquire 300 bytes
normally.
The interleaving is reproducible for both ON_HEAP and OFF_HEAP using Spark
MemoryConsumer and TaskMemoryManager. It requires concurrent memory operations
within one task; ordinary single-threaded allocations do not trigger it. This
report does not claim an end-to-end SQL or Python query reproducer.
h3. Proposed fix
Track acquisitions only after they actually wait, under the existing
memory-manager monitor. Retain a zero-byte task entry while any waiter remains,
then remove it when the final waiter exits without a reservation. Clean up the
waiter count on success, interruption, and exceptions. Count each task once for
fairness, even when it has multiple waiting acquisitions. Keep the existing
allocation limits, lock ordering, and storage-reclamation callbacks unchanged.
Releasing all currently charged bytes is not cancellation of outstanding
acquisitions; document that distinction.
h3. Regression coverage
Cover last-byte release and releaseAllMemoryForTask, two simultaneous waiters,
partial release, interruption, an exception after waiting, and two public
MemoryConsumer instances sharing one TaskMemoryManager. Run each scenario in
both memory modes.
The same vulnerable map-removal pattern is present in the inspected Apache
Spark master and 3.5/4.0/4.1 branch sources. The proposed patch targets master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
