yashdotrv commented on PR #27129:
URL: https://github.com/apache/datafusion/pull/27129#issuecomment-6795635505
@jayzhan211 Thanks again for the review and detailed pointers! I have
updated the branch with the requested changes:
1. **Per-Row Charging Threshold**: Replaced the fixed 64 KiB threshold with
`MAX_IN_FLIGHT_STATE_BYTES_PER_ROW = 1024` and `MIN_CHARGED_STATE_BATCH_BYTES =
64 * 1024`. Normal bounded batches remain exempt, avoiding unnecessary
spills/refusals on queries with many aggregates.
2. **Soft Limit Early Break**: Kept the early break for soft group limit
before `try_resize` in `FinalHashAggregateStream`.
3. **Spill Logic Cleanup**: Extracted the duplicate spill-and-resize logic
in `FinalHashAggregateStream` into a shared `spill_table` helper.
4. **Ungrouped Aggregate Guard**: Guarded the calculation of incoming state
batch memory in `AggregateStream` strictly to `AggregateInputMode::Partial`.
5. **Call Sequence Tests Restored**: Restored
`aggregate_memory_call_sequence.rs` to match `main` without edits; all 8
snapshot tests pass.
6. **Test Coverage**:
- Updated `final_aggregate_reserves_incoming_state_batch` and
`final_hash_reserves_incoming_state_batch` to use 64-row sliced batches
retaining multi-megabyte underlying buffers.
- Added `clustered_final_refuses_large_state_batch`.
- Added `final_hash_spills_to_admit_large_state_batch`.
- Added `clustered_final_spills_to_admit_large_state_batch` to test the
spill retry path under `GroupClusteringMode::Partial(vec![0])`.
7. **PR Description**: Clarified that #26705 remains open for the queued
partial states in `CoalescePartitionsExec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
