andygrove opened a new pull request, #5933:
URL: https://github.com/apache/datafusion-comet/pull/5933
## Which issue does this PR close?
Relates to #4576. It does not close it: this documents the memory model as it
exists today, including the gaps that issue is about, rather than changing
any
behavior.
## Rationale for this change
Comet's memory model is spread across four places that are individually
reasonable and collectively hard to hold in your head: the JVM config path
that
computes the budget, the native memory-pool decorators, the Arrow FFI
ownership
rules, and the container's cgroup. None of it was written down in one place.
The practical cost shows up when triaging an out-of-memory report. Answering
"which budget did this exceed, and who was supposed to be counting?"
currently
means reading `CometExecIterator.getMemoryConfig`,
`parse_memory_pool_config`,
the pool implementations, the FFI export paths, and Spark's
`ResourceProfile`.
It also makes it hard to reason about proposed fixes, because the baseline
they
would improve on is not written down.
I started this while reviewing #4582 and found that most of what I needed to
explain was existing behavior, not the prototype. The prototype is a separate
question; this page is useful either way.
## What changes are included in this PR?
A single new contributor-guide page, `memory_management.md`, registered under
Project Architecture. No behavior changes.
Sections:
- **Who allocates what.** Enabling Comet adds several distinct consumers, not
one, and they are not accounted by the same party. Notably
`CometArrowAllocator`
is a process-wide `RootAllocator(Long.MaxValue)` that is unbounded and
invisible
to both Spark's `TaskMemoryManager` and Comet's native pool; the JVM
shuffle
allocator switches between a Spark `MemoryConsumer` (off-heap) and a
self-bounded `UnsafeMemoryAllocator` (on-heap); and on-heap mode sizes the
native pool and the JVM shuffle pool from the same
`spark.comet.memoryOverhead`,
so it can occupy roughly twice that figure.
- **Where the budget comes from**, per Spark memory mode, and which of
`memory_limit` / `memory_limit_per_task` each pool type is sized from.
- **The pool stack**, the unified pools' JNI delegation to Spark, and the
task-shared pool's RAII lifetime including the acquire/drop race.
- **Crossing the FFI boundary.** Zero-copy transfer means the allocator that
produced a batch and the runtime that decides when it dies can be on
opposite
sides. JVM-to-native batches are charged to nothing while native pins them;
native-to-JVM batches stay charged until the JVM closes them.
- **The accounting gap**, and why `spark.comet.exec.memoryPool.fraction`
exists.
- **What the container sees.** The Kubernetes pod limit is
`executor.memory + memoryOverhead + offHeap.size + pyspark`, so
`spark.memory.offHeap.size` is inside the pod limit rather than headroom
on top
of it.
- **Open problems** and a triage checklist.
## How are these changes tested?
Documentation only; there is no behavior to test. Every claim was checked
against
the code on `main` or against the referenced Spark source rather than
written from
memory, and `npx prettier --check` passes.
Marked as draft because I would like a sanity check on the accuracy of the
"Who allocates what" and "Crossing the FFI boundary" sections in particular
before this becomes the reference people cite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
