Omega359 commented on issue #23447:
URL: https://github.com/apache/datafusion/issues/23447#issuecomment-4939705107
I pointed codex at this pr and it thinks the issue is related to
FairSpillPool. It ran the reproducer above among other things and came up with
this:
> Likely cause: FairSpillPool fair-share accounting interacts badly with
high partition fan-out. The 10-partition plan registers many spillable
consumers before the query has really made progress: grouped aggregates, two
sorters per SMJ partition, and several RepartitionExec reservations.
FairSpillPool::try_grow divides available spillable memory by num_spill, so
once ~60 spillable consumers are registered, each active consumer only gets a
small per-consumer allowance. That makes RepartitionExec, ExternalSorter, and
later grouped aggregate spill paths repeatedly hit memory pressure instead of
progressing.
Key code points:
- dev/datafusion/datafusion/execution/src/memory_pool/pool.rs:236:
FairSpillPool::try_grow caps each spillable reservation at (pool_size -
unspillable) / num_spill.
- dev/datafusion/datafusion/physical-plan/src/repartition/mod.rs:452:
RepartitionExec registers one spillable reservation per output partition.
- dev/datafusion/datafusion/physical-plan/src/sorts/sort.rs:255: each
ExternalSorter registers a spillable sorter reservation and an unspillable
merge reservation.
-
dev/datafusion/datafusion/physical-plan/src/aggregates/grouped_hash_stream.rs:515:
grouped hash aggregate marks its reservation spillable when spilling is
enabled.
Evidence from runs using a temporary repro crate outside the repo:
- FairSpillPool, 150M: 4, 6, 7 partitions completed; 8, 9, 10 partitions
timed out.
- FairSpillPool, 10 partitions: 180M timed out; 220M, 300M, 600M completed.
- GreedyMemoryPool, 10 partitions, same 150M: completed quickly.
- Logging wrapper showed roughly 60 spillable consumers registered in the
10-partition case, with repeated try_grow failures first around RepartitionExec
/ ExternalSorter, then grouped aggregate.
```
Results:
pool partitions memory sort reservation result
fair 8 150M default timed out at 90s
greedy 8 150M default completed
fair 10 150M default timed out at 90s
greedy 10 150M default completed
greedy 12 150M default fast OOM error
greedy 16 150M default fast OOM error
greedy 20 150M default fast OOM error
greedy 12 220M default completed
greedy 20 300M default completed
greedy 20 150M 1 MiB completed
fair 12 220M default completed
fair 20 150M 1 MiB timed out at 90s
The greedy failures at 12+ partitions with 150M were not stalls. They
failed quickly with:
Not enough memory to continue external sort ...
Failed to allocate additional 10.0 MB for ExternalSorterMerge[...]
So the behavior is pretty distinct:
- GreedyMemoryPool either completes or fast-fails with a clear OOM-style
error.
- FairSpillPool is the one producing the timeout/stall pattern under the
same 8/10 partition, 150M setup.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
