d0ra-1h3-3xpl0ra opened a new pull request, #26857: URL: https://github.com/apache/datafusion/pull/26857
## Which issue does this PR close? - Closes #26381. ## Rationale for this change `take` and `interleave` share every data buffer of a view column, so a few retained rows keep their input batch's whole string buffer, and the store is charged for it. With this PR, the issue's repro passes at `--memory-limit 3M`, down from 463M. ## What changes are included in this PR? For each `Utf8View` / `BinaryView` column, the store tracks the bytes it is charged (a `RecordBatchMemoryTracker` per column, counting a shared buffer once) and the bytes it references (views plus data). On gather or compaction, a column's data is copied out with spill's `gc_array` (now `pub(crate)`) only if the column would otherwise be charged over twice what it references. Copying isn't always a saving: Parquet shares a dictionary page's buffer across batches, which the store charges once. Simpler rules regressed when I measured them (base 985bbbf60): always copying took dictionary `rn <= 45` from 20M to 41M and `rn <= 90` from 39M to 81M, and a per-array 2× check took 1 KB dictionary values at `rn <= 45` from 24M to 240M. ## What is the testing strategy for this PR? Unit tests cover the insert path (`ROW_NUMBER`, `RANK`), compaction both ways, a shared column next to a unique one, and 16-byte shared strings. The randomized bookkeeping test also runs with a `Utf8View` payload and recounts each column after every batch. Seven mutations each fail a test. Smallest passing `--memory-limit` (bisected from 1M) and wall time, release `datafusion-cli`, `main` → this PR: | shape | memory | time (ms) | |---|---|---| | issue repro (`rn <= 1`) | 463M → 3M | 245 → 219 | | same, every row admitted | 463M → 3M | 278 → 262 | | 1 KB dictionary + plain payload, `rn <= 1` | 467M → 8M | 245 → 222 | | same, `rn <= 45` | 488M → 80M | 267 → 242 | | issue's `rn <= 10`, 100K partitions | 208M → 208M | 362 → 356 | Thirteen dictionary-only shapes (16 B–1 KB values, both orders, `rn <= 1/45/90`, 100K partitions) keep their memory. The all-admitted ones are about 3% (6 ms per 4.1M rows) slower, and the rest are within run-to-run spread. The cost grows with the number of view columns: with two dictionary columns and one unique column, all admitted, it is 3.4% slower in wall time and uses 8% more CPU, while memory goes 500M → 94M. I can post the full table. Limits: - A run sharing a buffer can have its first entries copied (at most about that buffer's size, right after an insert), and evictions can leave one row pinning a shared buffer until a row-triggered compaction. - `gc_array` skips buffers under 10 KB. - Nested and dictionary-of-view columns are not compacted, the same as on `main`. - Per-column trackers and the per-entry cache (~60–70 B per entry, plus a map slot per buffer) aren't in the reservation. - Plain `TopK` and `PartitionedTopKDenseRank` are unchanged. ## Are there any user-facing changes? No API changes. Window top-N queries over view-typed columns need less memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
