andygrove opened a new issue, #6914: URL: https://github.com/apache/datafusion-comet/issues/6914
### What is the problem the feature request solves? Before a partition's first rows reach the native Iceberg writer, `DictionaryChooser` replays parquet-mr's first-page dictionary decision for every column, so that native files use the encodings iceberg-java's would. It holds up to `write.parquet.page-row-limit` rows (20,000 by default) per partition and, for each column, inserts every value's bytes into a `HashSet<&[u8]>` through a per-value `value_bytes` closure, growing the set as it goes. The Parquet writer then dictionary-encodes the same values again. In `CometIcebergEndToEndBenchmark` (#6910), the pre-pass is 1.7% of all CPU for nested types and 2.4% for primitives, 4% to 8% of the writer. The cost is per partition, so a fanout write to many partitions pays it once for each. ### Describe the potential solution - Hash fixed-width values directly rather than as byte slices, and avoid the per-value dynamic dispatch through `value_bytes`. - Pre-size the set, which removes the rehashing visible in the profile. - Check whether the decision can be read from parquet-rs's own dictionary encoder state instead of a separate pass. ### Additional context `DictionaryChooser::choose` and `FirstPage::add` in `native/core/src/execution/operators/iceberg_dictionary.rs`; `PartitionFeed::release` in `native/core/src/execution/operators/iceberg_write.rs`. Measured on the fully native configuration of `CometIcebergEndToEndBenchmark`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
