Rich-T-kid opened a new pull request, #27500: URL: https://github.com/apache/datafusion/pull/27500
## Which issue does this PR close? - Closes #24113. - Part of #24111. ## Rationale for this change `enable_rle_to_dictionary` is an unbounded opt-in: if a top-level string or binary Parquet column has a dictionary page, DataFusion can infer and scan it as a dictionary. NDV alone is not a safe promotion signal because a column chunk can start with dictionary-encoded data pages and later fall back to plain encoding. That transition can force the Arrow reader to expand and rebuild dictionaries. Parquet page-encoding statistics let us exclude those mixed chunks, after which an NDV cap can provide the bounded user control requested in #24113. ## What changes are included in this PR? This adds `datafusion.execution.parquet.dictionary_read_max_distinct_values`. Setting the cap enables guarded dictionary reads without also setting `enable_rle_to_dictionary`. A top-level string or binary column is eligible only when every row group has NDV statistics, every data page is dictionary encoded, and the sum of row-group NDVs (a conservative per-file upper bound) is at most the configured cap. Missing statistics and mixed page encodings fail closed. Schema inference keeps a column as a dictionary only when every file containing it is eligible. The file opener rechecks each actual footer, so files added after schema inference and files under additional listing paths decode as plain values when they are ineligible, then use normal schema adaptation if the table already published a dictionary type. The option is wired through protobuf serialization, generated configuration documentation, and the 57.0 upgrade guide. `None` preserves the existing behavior. ## What is the testing strategy for this PR? Focused unit and SQL logic tests cover cap-only activation, missing NDV, exact and above-cap values, zero, multiple row groups, mixed files, dictionary-to-plain fallback, embedded Arrow dictionary schemas, per-file opener rechecks, legacy behavior, and protobuf round trips. Local benchmarks used SF1 Parquet data, 12 partitions, an 8,192-row batch size, a 32 GiB memory limit, and two-pair ABBA ordering. They ran on the pre-rebase candidate (`c23c7106c`), whose patch is identical to this PR head. Cap testing compared the same candidate binary with the cap disabled and set to 5. A cap-only type preflight on the exact benchmark files inferred TPC-H `region.r_name` and TPC-DS `customer_demographics.cd_gender` as `Dictionary(Int32, Utf8)`, confirming that the treatment exercised dictionary reads. Default-off testing compared the candidate with its exact build base (`375bf976`) while the cap remained unset and the legacy boolean remained false. All 5,604 measured iterations succeeded with row-count parity. | Comparison | Coverage | Aggregate mean delta (pair 1 / pair 2; combined) | Confirmed regression above 5% | | --- | --- | --- | --- | | cap=5 vs disabled | TPC-H 1-22, 7 iterations | -2.85% / -2.67%; -2.76% | none | | cap=5 vs disabled | TPC-DS 1-99, 3 iterations plus focused 25-50 iteration reruns | -2.39% / +3.22%; +0.39% | none after focused confirmation | | candidate default-off vs exact base | 11 previously flagged queries, 50-100 iterations | +0.61% / +7.17%; +3.90% | none in both independent pairs | The initial low-iteration TPC-DS Q1, Q22, and Q34 signals were rerun. Q22 and Q34 cleared. Q1's first focused median signal did not reproduce in a fresh reverse-order 50-iteration matrix: its first cap leg was an isolated slow run, while the closing pair was +2.3% median and +1.2% mean. No query's min, median, or mean regression above 5% survived both independent confirmation pairs. Validation also includes: - `cargo test -p datafusion-datasource-parquet test_rle_` - `cargo test -p datafusion-datasource-parquet dictionary_promotion_requires_dictionary_only_data_pages` - `cargo test -p datafusion-proto dictionary_read_options_round_trip_through_codec` - `cargo test --profile=ci --test sqllogictests -- parquet_rle_to_dictionary.slt` - `cargo test --profile=ci --test sqllogictests -- information_schema.slt` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? Yes. `ParquetOptions` gains the public field `dictionary_read_max_distinct_values: Option<usize>`, and the generated protobuf `ParquetOptions` gains the optional `dictionary_read_max_distinct_values_opt` field. Downstream exhaustive struct literals must add the new field or use `..Default::default()`. The protobuf wire format remains backward compatible, and the default runtime behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
