peterxcli opened a new issue, #5793: URL: https://github.com/apache/datafusion-comet/issues/5793
### What is the problem the feature request solves? `shuffle_bench` builds its input with `SessionContext::read_parquet` under a default `SessionConfig`, and DataFusion's `schema_force_view_types` defaults to true, so string and binary columns reach `ShuffleWriterExec` as `Utf8View` and `BinaryView`. Comet never hands the shuffle writer those types: the serde maps Spark `String` to `Utf8`, and `planner.rs` casts UDF results back from `Utf8View` to `Utf8` with the comment that Comet does not yet support view types. The bench is measuring the writer on a data shape production does not produce. That would be a minor fidelity gap except that view arrays are the case the writer handles worst. `BufBatchWriter` configures its `BatchCoalescer` with `with_biggest_coalesce_batch_size(Some(batch_size - 1))`, so any produced batch of at least `batch_size` rows bypasses the coalescer and is serialized verbatim. An interleaved view array keeps the backing data buffers of every input batch it drew rows from, and IPC writes all of them in full, so the block carries far more bytes than its own rows. Measured on a 5 column input with 2 string columns, 4M rows, 200 partitions, no compression, comparing the output file against the writer's own `data size` metric: | batch size | output file | data size | ratio | | --- | --- | --- | --- | | 2048 | 33.17 GiB | 236.95 MiB | ~143x | | 4096 | 14.86 GiB | 242.07 MiB | ~63x | | 8192 | 7.62 GiB | 242.33 MiB | ~32x | | 16384 | 4.00 GiB | 243.96 MiB | ~17x | | 20480 | 294.92 MiB | 245.45 MiB | ~1.2x | The cliff at 20480 is where a partition's 20000 rows stop exceeding the threshold and go through the coalescer, which compacts the views. The same configuration on a numeric-only input writes 62.81 MiB for 62.00 MiB of data, and lowering `--max-buffer-bytes` shrinks the ratio in step with the buffered volume each block interleaves from. The practical effect is that any string-heavy `shuffle_bench` number is dominated by amplification real Comet does not have, with encode time and write time inflated in proportion. #5198 points at this bench for measuring its items 1 and 6. ### Describe the potential solution Set `datafusion.execution.parquet.schema_force_view_types` to false in the bench's `SessionConfig` so the input matches what Comet's planner produces. Two smaller things are worth handling alongside it. The header line prints a schema read separately through `ParquetRecordBatchReaderBuilder`, which reports the file's `Utf8` rather than the `Utf8View` that actually executes, so the mismatch is invisible in the output. And the amplification itself is worth remembering if Comet ever does take on view types, since neither the local writer nor the RSS writer's `compact_array` compacts them today. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
