andygrove opened a new issue, #6919: URL: https://github.com/apache/datafusion-comet/issues/6919
### What is the problem the feature request solves? For byte-array columns, parquet-rs folds every value into the page's minimum and maximum (`compute_min_max` in `arrow_writer/byte_array.rs`) with two `Ord` comparisons, each a `memcmp` call. In `CometIcebergEndToEndBenchmark` (#6910) that is about 5.6% of all CPU for nested types, whose string columns are short and repeat often (map keys, tags). It is separate from dictionary encoding, which hashes the same values again. ### Describe the potential solution Upstream in arrow-rs: - When the dictionary encoder is active, compute page statistics over the distinct dictionary entries the page uses rather than over every value. - Compare short values without a `memcmp` call, for example by an 8-byte prefix first. ### Additional context parquet 59.3.0 (the version Comet uses): `encode` and `compute_min_max` in `parquet/src/arrow/arrow_writer/byte_array.rs`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
