L. C. Hsieh created SPARK-60134:
-----------------------------------
Summary: Remove memory consumers from TaskMemoryManager once they
no longer hold memory
Key: SPARK-60134
URL: https://issues.apache.org/jira/browse/SPARK-60134
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
TaskMemoryManager adds a consumer to its consumers set on every
acquireExecutionMemory call, but nothing removes it before
cleanUpAllAllocatedMemory() at the end of the task. A consumer that has
released all its memory (or has been closed) therefore stays reachable until
the task ends, together with everything it references, and both spill loops
(acquireExecutionMemory and spillConsumersForPageAllocation) keep scanning it
on every spill.
This was raised in the review of SPARK-60112
(https://github.com/apache/spark/pull/59331#discussion_r4236874042). After
SPARK-60112 tracks consumers by identity, equal-but-distinct consumers that
used to be collapsed into a single entry are each retained this way, although
the same already holds for any other distinct consumer.
Proposal: drop a consumer from the set once it no longer holds memory (for
example when its usage reaches zero, or when it is closed), instead of only
ever adding to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
