NoahKusaba opened a new issue, #2521: URL: https://github.com/apache/datafusion-ballista/issues/2521
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** After each interleave, the sort-shuffle writer calls `compact_view_columns`, which runs `gc()` on every `Utf8View` / `BinaryView` column ([`partitioned_batch_iterator.rs:27-50`](https://github.com/apache/datafusion-ballista/blob/187e21fc/ballista/core/src/execution_plans/sort_shuffle/partitioned_batch_iterator.rs#L27-L50)). `gc()` copies every referenced byte into fresh buffers, even when the column is already dense, or has no data buffers at all because every value is 12 bytes or fewer and stored inline. The compaction exists for a good reason: `interleave_record_batch` keeps every source data buffer alive, so a sparse output would otherwise serialize far more bytes than it uses. But for columns that are already dense the copy is wasted work. **Describe the solution you'd like** Only compact a view column when its data buffers are more than twice the bytes its views reference. Arrow's `BatchCoalescer` uses the same threshold before copying strings ([`byte_view.rs#L376-L379`](https://github.com/apache/arrow-rs/blob/782e5a685501a9db6cc8e9a3b7cbff894940c47a/arrow-select/src/coalesce/byte_view.rs#L376-L379), arrow 59.2.0). **Describe alternatives you've considered** Keeping unconditional compaction. It gives the smallest files, but pays a full string copy on every output batch, including ones with nothing to reclaim. **Additional context** Found in the performance review tracked in #2520 (item C13). The tradeoff: a column that skips compaction can carry up to 2× its referenced string bytes into the shuffle file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
