parthchandra opened a new issue, #7274:
URL: https://github.com/apache/datafusion-comet/issues/7274
Part of #7267.
## Problem
`BoundedWindowAggExec` (built in
`native/core/src/execution/planner/window.rs:135`)
keeps the rows it can't emit yet. For each new input batch it runs
`concat_batches(input_schema, [input_buffer, &record_batch])` (DataFusion
55.2.0,
`bounded_window_agg_exec.rs:706`). Each window partition's state is grown
the same
way in `PartitionBatchState::extend`
(`datafusion-expr-55.2.0/src/window_state.rs:299-302`).
Both re-concatenate the whole buffer on every batch.
The buffer grows only while output can't be released yet:
- RANGE frames with many tied ORDER BY values. The default frame with ORDER
BY on a
low-cardinality column buffers the whole tie group.
- Large `N FOLLOWING` offsets.
- Long `lag` / `lead` offsets.
If that buffer holds more than 2 GiB in a string/binary column, the concat
fails
with `OffsetOverflowError`.
Not affected: UNBOUNDED FOLLOWING frames (they go to `WindowAggExec`,
sub-issue 3),
UNBOUNDED PRECEDING aggregates (they trim the buffer as they go,
`physical-expr window/aggregate.rs:166-169`), and a missing PARTITION BY on
its own.
Value functions such as `first_value` / `nth_value` with UNBOUNDED PRECEDING
are not
checked yet.
## Options
1. **Upstream:** keep the buffer as a list of batches, not one concatenated
batch.
This also removes the repeated copy on every input batch.
2. **Comet:** same pass-through projection idea as sub-issue 3. Only the
window's
keys and inputs go through the operator; large payload columns are joined
back
afterwards.
## Tests
- SQL-file test: `sum(x) OVER (ORDER BY const_col)` (one big tie group) with
a binary
pass-through column, compared with Spark.
- SQL-file test: `lag(x, N)` with a large N over a binary column.
## Notes
- Shares a design with sub-issue 3. Probably best solved together.
- Risk is low to medium: it needs a long tie group or a large offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
